Nov 03 01:14:54.845843 ip-10-0-140-255 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 01:14:54.845856 ip-10-0-140-255 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 01:14:54.845865 ip-10-0-140-255 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 01:14:54.846198 ip-10-0-140-255 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 01:15:04.857426 ip-10-0-140-255 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 01:15:04.857446 ip-10-0-140-255 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 205207359dd04dd49a18fb622990c068 --
Nov 03 01:17:23.025627 ip-10-0-140-255 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 01:17:23.427922 ip-10-0-140-255 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 01:17:23.427922 ip-10-0-140-255 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 01:17:23.427922 ip-10-0-140-255 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 01:17:23.427922 ip-10-0-140-255 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 01:17:23.433020 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.432942    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:17:23.433020 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433020    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433025    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433028    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433031    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433034    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433037    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433040    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433043    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433047    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433051    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433055    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433058    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433060    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433063    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433066    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433068    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433071    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433074    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433077    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:17:23.433122 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433080    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433083    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433085    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433088    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433096    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433099    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433102    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433104    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433107    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433110    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433113    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433116    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433119    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433121    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433124    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433127    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433130    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433133    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433135    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433139    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:17:23.433573 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433141    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433144    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433147    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433150    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433152    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433155    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433157    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433160    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433162    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433165    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433168    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433170    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433173    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433175    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433178    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433180    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433183    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433186    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433188    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433191    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:17:23.434038 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433194    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433197    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433199    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433202    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433205    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433208    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433211    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433213    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433216    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433219    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433221    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433226    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433230    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433233    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433235    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433239    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433241    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433244    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433246    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:17:23.434550 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433249    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433251    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433254    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433257    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433260    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433262    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433265    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433267    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433270    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433273    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433276    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433279    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433282    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433284    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433287    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433290    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433293    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433296    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433299    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433301    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:17:23.434997 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433304    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433307    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433309    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433312    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433315    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433318    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.433321    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434106    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434114    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434117    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434120    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434123    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434126    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434129    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434131    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434134    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434137    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434140    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434143    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434145    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:17:23.435471 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434149    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434152    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434154    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434158    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434161    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434164    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434166    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434169    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434172    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434174    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434177    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434179    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434182    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434185    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434188    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434190    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434193    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434195    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434198    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434201    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:17:23.435945 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434203    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434206    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434208    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434211    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434213    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434216    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434218    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434221    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434223    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434226    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434228    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434231    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434234    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434236    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434239    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434242    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434245    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434248    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434250    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434253    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:17:23.436429 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434256    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434258    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434261    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434263    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434267    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434270    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434272    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434275    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434277    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434280    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434282    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434285    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434287    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434289    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434292    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434294    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434297    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434299    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434302    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:17:23.436936 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434304    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434308    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434312    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434318    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434321    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434324    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434326    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434329    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434333    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434335    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434338    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434341    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434343    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434361    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434363    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434366    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434369    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434371    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434374    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:17:23.437398 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434376    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434379    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434382    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434384    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434387    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434391    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434394    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434397    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434399    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434403    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434406    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434408    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434411    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434413    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434416    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434881    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434888    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434890    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:17:23.437857 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434893    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434896    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434899    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434901    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434904    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434907    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434910    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434912    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434915    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434918    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434921    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434923    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434926    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434929    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434931    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434934    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434937    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434940    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434942    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434945    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:17:23.438281 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434947    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434950    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434953    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434955    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434958    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434960    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434963    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434965    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434968    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434970    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434974    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434977    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434980    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434982    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434985    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434988    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434990    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434993    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434996    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.434999    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:17:23.438756 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435001    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435004    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435008    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435012    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435014    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435017    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435020    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435022    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435025    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435027    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435030    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435032    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435035    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435038    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435041    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435045    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435048    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435051    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435053    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:17:23.439217 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435056    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435058    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435061    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435064    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435067    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435070    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435072    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435075    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435078    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435080    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435083    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435085    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435089    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435092    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435094    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435097    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435100    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435102    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435105    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435107    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:17:23.439690 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435110    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435112    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435115    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435117    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435120    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435123    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435126    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435128    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435131    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435133    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435136    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435138    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435141    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435143    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435146    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435149    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435152    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435155    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435159    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435162    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:17:23.440158 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435165    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435167    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435170    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.435172    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436185    2575 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436195    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436202    2575 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436207    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436212    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436215    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436220    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436224    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436227    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436230    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436234    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436237    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436241    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436244    2575 flags.go:64] FLAG: --cgroup-root=""
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436247    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436250    2575 flags.go:64] FLAG: --client-ca-file=""
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436253    2575 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436256    2575 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436260    2575 flags.go:64] FLAG: --cluster-domain=""
Nov 03 01:17:23.440639 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436263    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436266    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436269    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436273    2575 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436277    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436280    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436283    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436288    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436291    2575 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436294    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436297    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436301    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436303    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436308    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436311    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436314    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436317    2575 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436320    2575 flags.go:64] FLAG: --enable-server="true"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436323    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436328    2575 flags.go:64] FLAG: --event-burst="100"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436331    2575 flags.go:64] FLAG: --event-qps="50"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436334    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436337    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436340    2575 flags.go:64] FLAG: --eviction-hard=""
Nov 03 01:17:23.441157 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436356    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436360    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436363    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436366    2575 flags.go:64] FLAG: --eviction-soft=""
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436369    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436372    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436375    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436379    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436382    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436385    2575 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436388    2575 flags.go:64] FLAG: --feature-gates=""
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436395    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436398    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436401    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436404    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436409    2575 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436412    2575 flags.go:64] FLAG: --help="false"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436415    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436418    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436421    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436424    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436427    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436431    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 01:17:23.441802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436434    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436438    2575 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436441    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436444    2575 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436447    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436450    2575 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436454    2575 flags.go:64] FLAG: --kube-reserved=""
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436457    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436460    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436463    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436466    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436469    2575 flags.go:64] FLAG: --lock-file=""
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436472    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436475    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436478    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436483    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436487    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436490    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436493    2575 flags.go:64] FLAG: --logging-format="text"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436496    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436500    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436503    2575 flags.go:64] FLAG: --manifest-url=""
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436505    2575 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436510    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436514    2575 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 01:17:23.442334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436517    2575 flags.go:64] FLAG: --max-pods="110"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436521    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436524    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436527    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436529    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436533    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436535    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436538    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436545    2575 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436549    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436552    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436555    2575 flags.go:64] FLAG: --pod-cidr=""
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436558    2575 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436561    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436564    2575 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436567    2575 flags.go:64] FLAG: --port="10250"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436570    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436573    2575 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-051d82144733e4e2c"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436576    2575 flags.go:64] FLAG: --qos-reserved=""
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436579    2575 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436582    2575 flags.go:64] FLAG: --register-node="true"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436584    2575 flags.go:64] FLAG: --register-with-taints=""
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436588    2575 flags.go:64] FLAG: --registry-burst="10"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436592    2575 flags.go:64] FLAG: --registry-qps="5"
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436595    2575 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436598    2575 flags.go:64] FLAG: --reserved-memory=""
Nov 03 01:17:23.442933 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436602    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436605    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436608    2575 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436611    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436614    2575 flags.go:64] FLAG: --runonce="false"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436617    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436620    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436623    2575 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436625    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436628    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436631    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436634    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436638    2575 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436641    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436643    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436646    2575 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436650    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436653    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436656    2575 flags.go:64] FLAG: --system-cgroups=""
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436659    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436665    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436667    2575 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436670    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436675    2575 flags.go:64] FLAG: --tls-min-version=""
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436677    2575 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 01:17:23.443564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436680    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436683    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436686    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436689    2575 flags.go:64] FLAG: --v="2"
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436694    2575 flags.go:64] FLAG: --version="false"
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436698    2575 flags.go:64] FLAG: --vmodule=""
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436702    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.436706    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436837    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436840    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436843    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436846    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436849    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436851    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436854    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436856    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436859    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436861    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436864    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436866    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436869    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436872    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:17:23.444145 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436874    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436877    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436880    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436882    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436885    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436888    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436890    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436893    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436895    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436898    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436900    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436902    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436905    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436908    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436911    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436914    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436917    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436919    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436921    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436924    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:17:23.444677 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436927    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436929    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436932    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436934    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436937    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436940    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436942    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436945    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436948    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436950    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436953    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436955    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436958    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436960    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436963    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436966    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436968    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436971    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436973    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:17:23.445140 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436975    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436978    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436982    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436985    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436988    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436991    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436993    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.436997    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437000    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437003    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437005    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437008    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437011    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437013    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437016    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437019    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437021    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437024    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437026    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:17:23.445694 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437029    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437031    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437034    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437036    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437039    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437042    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437044    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437047    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437049    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437052    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437055    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437058    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437060    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437062    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437066    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437069    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437073    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437076    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437078    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:17:23.446195 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437081    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437084    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437088    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437091    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437093    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437096    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437099    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437101    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437104    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437107    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437109    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437111    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437114    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437117    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437119    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.437145    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 01:17:23.446686 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437180    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437185    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437187    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437190    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437193    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437196    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437198    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437201    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437204    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437208    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437212    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437216    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437219    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437221    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437224    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437226    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437229    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437232    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437235    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:17:23.447092 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437239    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437241    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437244    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437247    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437250    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437252    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437255    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437258    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437260    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437263    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437265    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437268    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437270    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437273    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437276    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437278    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437281    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437283    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437286    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437288    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:17:23.447709 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437291    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437293    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437296    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437299    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437302    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437305    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437307    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437310    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437312    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437315    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437318    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437321    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437325    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437327    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437330    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437332    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437336    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437339    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437341    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437355    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:17:23.448310 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437358    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437361    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437363    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437366    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437369    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437372    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437374    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437377    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437380    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437382    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437385    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437387    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437390    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437393    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437395    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437398    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437401    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437404    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437406    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:17:23.448791 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437409    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437412    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437415    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437419    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437422    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437426    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437430    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437433    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437436    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437439    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437442    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437444    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437447    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437449    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437452    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437454    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437457    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437460    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437462    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:17:23.449241 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437465    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437467    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437470    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437473    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437475    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437477    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437481    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437484    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.437486    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.444427    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.444554    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.445382    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.445388    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445448    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445454    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445457    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445460    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445464    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445468    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:17:23.449824 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445471    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445474    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445477    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445479    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445482    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445485    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445489    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445491    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445494    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445497    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445499    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445502    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445505    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445507    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445510    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445513    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445515    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445518    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445520    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445523    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:17:23.450268 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445526    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445529    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445531    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445534    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445537    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445540    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445543    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445546    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445548    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445551    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445553    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445556    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445559    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445561    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445564    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445567    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445569    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445572    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445575    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:17:23.450746 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445577    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445580    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445583    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445586    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445588    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445591    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445593    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445596    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445600    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445604    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445607    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445610    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445613    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445616    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445619    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445621    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445624    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445627    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445629    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:17:23.451219 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445632    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445634    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445637    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445640    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445642    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445645    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445647    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445650    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445652    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445655    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445657    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445660    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445662    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445665    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445669    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445671    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445674    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445677    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445679    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:17:23.451859 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445682    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445684    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445687    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445690    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445693    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445695    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445698    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445701    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445704    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445707    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445710    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445713    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445715    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445718    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445720    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445723    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445726    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445728    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445731    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445733    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:17:23.452341 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445736    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445739    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.445742    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.445768    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447906    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447920    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447924    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447927    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447930    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447933    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447936    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447938    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447941    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447944    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447947    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447949    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:17:23.452856 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447953    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447955    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447958    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447961    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447963    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447966    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447969    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447971    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447974    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447976    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447979    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447982    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447985    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447988    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447990    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447993    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.447996    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448003    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448007    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448009    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:17:23.453234 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448012    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448015    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448017    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448020    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448022    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448025    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448027    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448030    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448032    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448035    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448037    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448040    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448044    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448046    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448049    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448051    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448054    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448056    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448059    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448061    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:17:23.453729 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448064    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448066    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448069    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448071    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448074    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448076    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448079    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448082    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448084    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448087    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448090    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448092    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448095    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448097    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448100    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448102    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448105    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448107    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448110    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448112    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:17:23.454182 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448115    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448117    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448120    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448123    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448125    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448128    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448132    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448136    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448139    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448142    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448145    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448148    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448151    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448154    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448157    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448159    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448162    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448164    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448167    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:17:23.454736 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448169    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448172    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448174    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448177    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448179    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448182    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448185    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448188    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448190    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448194    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448198    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448201    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448204    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448206    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448209    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.448235    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 01:17:23.455198 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448272    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448276    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448279    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448282    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448285    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448287    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448290    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448293    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448296    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448299    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448301    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448304    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448307    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448309    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448312    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448315    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448319    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448323    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448326    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:17:23.455586 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448329    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448331    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448334    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448336    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448338    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448341    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448343    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448360    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448364    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448368    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448373    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448377    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448382    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448385    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448387    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448390    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448393    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448395    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448398    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448400    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:17:23.456104 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448403    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448405    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448408    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448411    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448414    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448416    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448419    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448422    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448425    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448427    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448430    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448432    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448434    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448437    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448439    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448442    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448444    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448447    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448449    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448452    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:17:23.456592 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448454    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448457    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448459    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448462    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448465    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448467    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448470    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448472    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448475    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448477    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448480    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448482    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448485    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448487    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448489    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448492    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448495    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448497    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448500    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448502    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:17:23.457056 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448505    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448507    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448510    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448512    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448515    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448517    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448520    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448525    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448528    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448530    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448534    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448537    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448539    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448542    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448544    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448547    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448549    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448552    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448554    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:17:23.457584 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448557    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448559    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448563    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448567    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448570    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448574    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448576    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:23.448579    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.448638    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.451742    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.452683    2575 server.go:1014] "Starting client certificate rotation"
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.452772    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.452807    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.454744    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 01:17:23.458025 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.455807    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 01:17:23.466702 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.466687    2575 log.go:25] "Validated CRI v1 runtime API"
Nov 03 01:17:23.469678 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.469664    2575 log.go:25] "Validated CRI v1 image API"
Nov 03 01:17:23.471318 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.471291    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 01:17:23.473866 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.473845    2575 fs.go:135] Filesystem UUIDs: map[4be91eae-1d97-4253-8730-451bdbf64a85:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7c3803db-a80b-4872-834d-74c902bb660c:/dev/nvme0n1p3]
Nov 03 01:17:23.473938 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.473864    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 01:17:23.478809 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.478694    2575 manager.go:217] Machine: {Timestamp:2026-11-03 01:17:23.47820392 +0000 UTC m=+0.335192506 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099717 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bd29b167c3ad55ed43a9ee71b9523 SystemUUID:ec2bd29b-167c-3ad5-5ed4-3a9ee71b9523 BootID:20520735-9dd0-4dd4-9a18-fb622990c068 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f8:80:d4:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f8:80:d4:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:fb:d9:7c:d2:46 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 01:17:23.478809 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.478797    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 01:17:23.478972 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.478896    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 01:17:23.482492 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.482469    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 01:17:23.482668 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.482493    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-255.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 01:17:23.482746 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.482671    2575 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 01:17:23.482746 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.482684    2575 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 01:17:23.482746 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.482716    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 01:17:23.482746 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.482740    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 01:17:23.483964 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.483951    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 01:17:23.488427 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.488413    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 01:17:23.488709 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.488696    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 01:17:23.489483 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.489472    2575 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 01:17:23.490764 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.489789    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 01:17:23.490764 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.489839    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 01:17:23.490764 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.489867    2575 kubelet.go:405] "Adding apiserver pod source"
Nov 03 01:17:23.490764 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.489895    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 01:17:23.490764 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.490580    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 01:17:23.493300 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.493284    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 01:17:23.499192 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.499176    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 01:17:23.500104 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500091    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 01:17:23.500172 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500120    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 01:17:23.500269 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500259    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 01:17:23.500300 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500276    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 01:17:23.500300 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500283    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 01:17:23.500300 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500289    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 01:17:23.500300 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500294    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 01:17:23.500300 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500300    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 01:17:23.500956 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500948    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 01:17:23.500956 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500956    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 01:17:23.501013 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500964    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 01:17:23.501013 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500970    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 01:17:23.501013 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500979    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 01:17:23.501013 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.500987    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 01:17:23.502508 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.502498    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 01:17:23.502508 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.502508    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 01:17:23.506308 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.506295    2575 server.go:1263] "Started kubelet"
Nov 03 01:17:23.506439 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.506410    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 01:17:23.506479 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.506401    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 01:17:23.506506 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.506490    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 01:17:23.507224 ip-10-0-140-255 systemd[1]: Started Kubernetes Kubelet.
Nov 03 01:17:23.508210 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.508193    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 01:17:23.509478 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.509460    2575 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 01:17:23.513155 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.513139    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 01:17:23.515996 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.515972    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 01:17:23.516672 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.516660    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 01:17:23.516807 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.516747    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-255.us-west-2.compute.internal\" not found"
Nov 03 01:17:23.517441 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.517423    2575 factory.go:55] Registering systemd factory
Nov 03 01:17:23.517441 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.517439    2575 factory.go:223] Registration of the systemd container factory successfully
Nov 03 01:17:23.517562 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.517459    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 01:17:23.517562 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.517462    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 01:17:23.517562 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.517495    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 01:17:23.517703 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.517593    2575 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 01:17:23.517703 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.517626    2575 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 01:17:23.517703 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.517686    2575 factory.go:153] Registering CRI-O factory
Nov 03 01:17:23.517703 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.517698    2575 factory.go:223] Registration of the crio container factory successfully
Nov 03 01:17:23.517889 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.517753    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 01:17:23.517889 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.517766    2575 factory.go:103] Registering Raw factory
Nov 03 01:17:23.517889 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.517776    2575 manager.go:1199] Started watching for new ooms in manager
Nov 03 01:17:23.518954 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.518931    2575 manager.go:319] Starting recovery of all containers
Nov 03 01:17:23.531253 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.531235    2575 manager.go:324] Recovery completed
Nov 03 01:17:23.535525 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.535513    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:17:23.537240 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.537217    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:17:23.537337 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.537257    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:17:23.537337 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.537272    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:17:23.537800 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.537787    2575 cpu_manager.go:226] "Starting" policy="none"
Nov 03 01:17:23.537800 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.537799    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 01:17:23.537930 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.537818    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 01:17:23.537930 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.537801    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:17:23.538654 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.538639    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:17:23.538711 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.538666    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:17:23.538711 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.538676    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:17:23.540032 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.540020    2575 policy_none.go:50] "Start"
Nov 03 01:17:23.540099 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.540037    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 01:17:23.540099 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.540050    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 01:17:23.542098 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.542086    2575 policy_none.go:44] "Start"
Nov 03 01:17:23.576925 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.576895    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 01:17:23.579242 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.579223    2575 manager.go:341] "Starting Device Plugin manager"
Nov 03 01:17:23.579334 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.579271    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 01:17:23.579334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.579284    2575 server.go:87] "Starting device plugin registration server"
Nov 03 01:17:23.579520 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.579506    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 01:17:23.579577 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.579519    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 01:17:23.579633 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.579608    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 01:17:23.579711 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.579696    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 01:17:23.579711 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.579710    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 01:17:23.580283 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.580262    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 01:17:23.580448 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.580317    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-255.us-west-2.compute.internal\" not found"
Nov 03 01:17:23.599680 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.599659    2575 container_manager_linux.go:682] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2609 into the system container \"/system.slice\": <nil>"
Nov 03 01:17:23.648773 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.648742    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-255.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 01:17:23.649784 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.648733    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-255.us-west-2.compute.internal.18e46420c4f2ad0a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-255.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-255.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:17:23.50626945 +0000 UTC m=+0.363258037,LastTimestamp:2026-11-03 01:17:23.50626945 +0000 UTC m=+0.363258037,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-255.us-west-2.compute.internal,}"
Nov 03 01:17:23.680305 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.680209    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:17:23.681665 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.681649    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:17:23.681731 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.681677    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:17:23.681731 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.681692    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:17:23.681731 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.681724    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.705185 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.695027    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-255.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 01:17:23.705185 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.703623    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 01:17:23.705185 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.704887    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 01:17:23.705185 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.704907    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 01:17:23.705185 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.704929    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 01:17:23.705185 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.704966    2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 01:17:23.720864 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.720848    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 01:17:23.740764 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.740741    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjxld"
Nov 03 01:17:23.753629 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.753613    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jjxld"
Nov 03 01:17:23.799916 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.799854    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-255.us-west-2.compute.internal.18e46420c6cb4183  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-255.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-255.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-255.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:17:23.537240451 +0000 UTC m=+0.394229045,LastTimestamp:2026-11-03 01:17:23.537240451 +0000 UTC m=+0.394229045,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-255.us-west-2.compute.internal,}"
Nov 03 01:17:23.800724 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.800701    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-255.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 01:17:23.800797 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.800705    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 01:17:23.800797 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.800707    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 01:17:23.805987 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.805964    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"]
Nov 03 01:17:23.806031 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.806022    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:17:23.806736 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.806722    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:17:23.806805 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.806751    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:17:23.806805 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.806761    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:17:23.808949 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.808937    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:17:23.809095 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.809081    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.809095 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.809112    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:17:23.809628 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.809606    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:17:23.809689 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.809636    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:17:23.809689 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.809646    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:17:23.809751 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.809606    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:17:23.809751 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.809711    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:17:23.809751 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.809721    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:17:23.811778 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.811761    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.811834 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.811784    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:17:23.812528 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.812503    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:17:23.812528 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.812523    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:17:23.812662 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.812533    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:17:23.819993 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.819959    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd3c5a92e769172fde5847bb75e6fd4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"bd3c5a92e769172fde5847bb75e6fd4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.820078 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.820011    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd3c5a92e769172fde5847bb75e6fd4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"bd3c5a92e769172fde5847bb75e6fd4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.820078 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.820068    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0eb27a029b0b668ca0885e8f9e1efac2-config\") pod \"kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"0eb27a029b0b668ca0885e8f9e1efac2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.843100 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.843078    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-255.us-west-2.compute.internal\" not found" node="ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.845387 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.845371    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-255.us-west-2.compute.internal\" not found" node="ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.859769 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.859754    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:17:23.860295 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.860279    2575 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-140-255.us-west-2.compute.internal\" not found" node="ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.920754 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.920731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd3c5a92e769172fde5847bb75e6fd4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"bd3c5a92e769172fde5847bb75e6fd4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.920855 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.920758    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd3c5a92e769172fde5847bb75e6fd4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"bd3c5a92e769172fde5847bb75e6fd4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.920855 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.920776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0eb27a029b0b668ca0885e8f9e1efac2-config\") pod \"kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"0eb27a029b0b668ca0885e8f9e1efac2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.920855 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.920836    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0eb27a029b0b668ca0885e8f9e1efac2-config\") pod \"kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"0eb27a029b0b668ca0885e8f9e1efac2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.920977 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.920859    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd3c5a92e769172fde5847bb75e6fd4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"bd3c5a92e769172fde5847bb75e6fd4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.920977 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:23.920836    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd3c5a92e769172fde5847bb75e6fd4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"bd3c5a92e769172fde5847bb75e6fd4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:23.948565 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:23.948517    2575 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-255.us-west-2.compute.internal\" not found" node="ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:24.060541 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.060521    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:17:24.061455 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.061433    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:17:24.061552 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.061463    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:17:24.061552 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.061473    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:17:24.061552 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.061506    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:24.116312 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.116292    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:24.116312 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:24.116314    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-255.us-west-2.compute.internal\": node \"ip-10-0-140-255.us-west-2.compute.internal\" not found"
Nov 03 01:17:24.147601 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.147586    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:24.148694 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.148680    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:24.228215 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:24.228169    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-255.us-west-2.compute.internal\" not found"
Nov 03 01:17:24.328377 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:24.328341    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-255.us-west-2.compute.internal\" not found"
Nov 03 01:17:24.428835 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:24.428808    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-255.us-west-2.compute.internal\" not found"
Nov 03 01:17:24.453072 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.453046    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 01:17:24.453179 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.453160    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 01:17:24.453320 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:24.453237    2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aece31dc849e6402abdece8830c7a683-e1048f7a7a1c1f18.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-140-255.us-west-2.compute.internal.18e46420c6cb4183\": read tcp 10.0.140.255:58414->13.184.28.100:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-140-255.us-west-2.compute.internal.18e46420c6cb4183  default   5208 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-255.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-255.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-255.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 01:17:23 +0000 UTC,LastTimestamp:2026-11-03 01:17:23.806736458 +0000 UTC m=+0.663725044,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-255.us-west-2.compute.internal,}"
Nov 03 01:17:24.529703 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:24.529656    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-255.us-west-2.compute.internal\" not found"
Nov 03 01:17:24.630355 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:24.630334    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-255.us-west-2.compute.internal\" not found"
Nov 03 01:17:24.677688 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.677663    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:17:24.678831 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.678816    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:17:24.678918 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.678851    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:17:24.678918 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.678868    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:17:24.730441 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:24.730417    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-255.us-west-2.compute.internal\" not found"
Nov 03 01:17:24.759576 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.759525    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 01:12:24 +0000 UTC" deadline="2028-04-04 04:59:16.484507577 +0000 UTC"
Nov 03 01:17:24.759576 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.759569    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12435h41m51.724940905s"
Nov 03 01:17:24.816910 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.816867    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:17:24.816994 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.816930    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:24.924961 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.924935    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:17:24.925703 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.925688    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 01:17:24.926857 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.926841    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:17:24.990305 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:24.990283    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 01:17:25.250404 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.250384    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:17:25.447027 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:25.446871    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3c5a92e769172fde5847bb75e6fd4d.slice/crio-78fe8905288581090afb36532f05784b20ebca0c055ed6c8e1074a9d6ff519be WatchSource:0}: Error finding container 78fe8905288581090afb36532f05784b20ebca0c055ed6c8e1074a9d6ff519be: Status 404 returned error can't find the container with id 78fe8905288581090afb36532f05784b20ebca0c055ed6c8e1074a9d6ff519be
Nov 03 01:17:25.452577 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:25.452552    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb27a029b0b668ca0885e8f9e1efac2.slice/crio-fe46741258363e7b02e84970875d6fe7881a3c9c79580df4429238fde64e8890 WatchSource:0}: Error finding container fe46741258363e7b02e84970875d6fe7881a3c9c79580df4429238fde64e8890: Status 404 returned error can't find the container with id fe46741258363e7b02e84970875d6fe7881a3c9c79580df4429238fde64e8890
Nov 03 01:17:25.454952 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.454937    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:17:25.494023 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.494009    2575 apiserver.go:52] "Watching apiserver"
Nov 03 01:17:25.516531 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.516489    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 01:17:25.606983 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.606960    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 01:17:25.607298 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.607282    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal","openshift-dns/node-resolver-287jt","openshift-image-registry/node-ca-8flnl","openshift-multus/multus-additional-cni-plugins-qv8q5","kube-system/global-pull-secret-syncer-b4htm","kube-system/konnectivity-agent-g4jmf","openshift-ovn-kubernetes/ovnkube-node-hr6kr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj","openshift-cluster-node-tuning-operator/tuned-vqm5v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal","openshift-multus/multus-46tlx","openshift-multus/network-metrics-daemon-vxv8x","openshift-network-diagnostics/network-check-target-ttbvc","openshift-network-operator/iptables-alerter-7xkjx"]
Nov 03 01:17:25.611160 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.611143    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-287jt"
Nov 03 01:17:25.613381 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.613360    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:17:25.616508 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.616489    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.618102 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.618084    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:25.618209 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:25.618186    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:25.620415 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.620393    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:17:25.622880 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.622861    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.625236 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.625221    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.627559 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.627544    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.630502 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.630484    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.630602 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.630558    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f57a8017-198c-42a9-9fb4-481348d8568c-serviceca\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:17:25.630602 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.630583    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-system-cni-dir\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.630781 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.630609    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-kubelet-config\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:25.630781 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.630676    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2rxv\" (UniqueName: \"kubernetes.io/projected/125c3968-dca4-4163-b473-b5cc7ac5a23f-kube-api-access-r2rxv\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:17:25.630781 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.630703    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f57a8017-198c-42a9-9fb4-481348d8568c-host\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:17:25.630781 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.630730    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-cnibin\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.631015 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.630819    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-cni-binary-copy\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.631015 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.630870    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-dbus\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:25.631015 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.630932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/125c3968-dca4-4163-b473-b5cc7ac5a23f-hosts-file\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:17:25.631015 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.630971    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.631015 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.631003    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.631178 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.631053    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqwfz\" (UniqueName: \"kubernetes.io/projected/2c57dcf4-5aa3-4992-b357-47b388f3db90-kube-api-access-jqwfz\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.631178 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.631083    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:25.631178 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.631107    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/125c3968-dca4-4163-b473-b5cc7ac5a23f-tmp-dir\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:17:25.631178 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.631144    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4gt7\" (UniqueName: \"kubernetes.io/projected/f57a8017-198c-42a9-9fb4-481348d8568c-kube-api-access-x4gt7\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:17:25.631178 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.631174    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.633447 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.633423    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:25.633514 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:25.633495    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:25.636794 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.636773    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:25.636884 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:25.636841    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:25.639041 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.639024    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:17:25.668254 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.668238    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 01:17:25.668457 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.668438    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 01:17:25.668532 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.668451    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r4kvh\""
Nov 03 01:17:25.670557 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.670536    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 01:17:25.670557 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.670545    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 01:17:25.670703 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.670596    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 01:17:25.670703 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.670607    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-th4bh\""
Nov 03 01:17:25.673117 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.673099    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 01:17:25.673188 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.673137    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 01:17:25.711044 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.711006    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal" event={"ID":"0eb27a029b0b668ca0885e8f9e1efac2","Type":"ContainerStarted","Data":"fe46741258363e7b02e84970875d6fe7881a3c9c79580df4429238fde64e8890"}
Nov 03 01:17:25.712144 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.712124    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal" event={"ID":"bd3c5a92e769172fde5847bb75e6fd4d","Type":"ContainerStarted","Data":"78fe8905288581090afb36532f05784b20ebca0c055ed6c8e1074a9d6ff519be"}
Nov 03 01:17:25.713619 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.713597    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 01:17:25.713619 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.713596    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 01:17:25.713757 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.713664    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c92gz\""
Nov 03 01:17:25.713810 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.713785    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 01:17:25.713866 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.713859    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 01:17:25.713918 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.713857    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nkj8v\""
Nov 03 01:17:25.713918 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.713872    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 01:17:25.713918 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.713875    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 01:17:25.714128 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.714113    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 01:17:25.714179 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.714127    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 01:17:25.714179 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.714140    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 01:17:25.714338 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.714326    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tcxvw\""
Nov 03 01:17:25.714435 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.714424    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 01:17:25.719016 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.718998    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 01:17:25.722691 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.722678    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bdhgr\""
Nov 03 01:17:25.722778 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.722764    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2fc25\""
Nov 03 01:17:25.723179 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.723167    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 01:17:25.728241 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.728226    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 01:17:25.728578 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.728561    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 01:17:25.728652 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.728566    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 01:17:25.728652 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.728647    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 01:17:25.730055 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.730040    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-9tz7d\""
Nov 03 01:17:25.732401 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732385    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.732476 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732411    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-cni-multus\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.732476 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732430    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwx2t\" (UniqueName: \"kubernetes.io/projected/3d824491-3dcc-4c32-8378-7c8459b34086-kube-api-access-mwx2t\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:25.732476 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732449    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-ovn\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.732476 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732472    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-cni-bin\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.732674 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732512    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.732674 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732566    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-ovnkube-config\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.732674 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732606    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-system-cni-dir\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.732674 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732615    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.732674 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-slash\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.732906 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732684    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-system-cni-dir\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.732906 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732716    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-etc-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.732906 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732734    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-modprobe-d\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.732906 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732782    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-sys\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.732906 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732818    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af477ca4-5745-4345-8e68-21ac9c32221f-etc-tuned\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.732906 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732848    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-627v6\" (UniqueName: \"kubernetes.io/projected/af477ca4-5745-4345-8e68-21ac9c32221f-kube-api-access-627v6\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.732906 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732877    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz2vv\" (UniqueName: \"kubernetes.io/projected/32d4bb3a-8abf-408a-8168-4581172f47f1-kube-api-access-wz2vv\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:17:25.733184 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732917    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-sys-fs\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.733184 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-cni-binary-copy\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.733184 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.732988    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-device-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.733184 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733003    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-run-netns\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.733184 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733093    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-var-lib-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.733184 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733112    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e263af32-db0f-4984-8e90-83347cdd15f7-ovn-node-metrics-cert\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.733184 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733127    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-systemd-units\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.733184 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733146    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqwfz\" (UniqueName: \"kubernetes.io/projected/2c57dcf4-5aa3-4992-b357-47b388f3db90-kube-api-access-jqwfz\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.733184 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733167    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-socket-dir-parent\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.733596 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733220    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-k8s-cni-cncf-io\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.733596 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733281    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-hostroot\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.733596 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733312    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4hm8\" (UniqueName: \"kubernetes.io/projected/2f7f681b-d410-40e9-a4d0-487230d8dece-kube-api-access-s4hm8\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.733596 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733340    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a4694-e358-4491-b4e1-c05bd477f19a-konnectivity-ca\") pod \"konnectivity-agent-g4jmf\" (UID: \"ee4a4694-e358-4491-b4e1-c05bd477f19a\") " pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:17:25.733596 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733383    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.733596 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733434    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-system-cni-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.733596 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733461    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-log-socket\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.733596 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733485    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-run\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.733596 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733488    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-cni-binary-copy\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.733596 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733537    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-965xj\" (UniqueName: \"kubernetes.io/projected/3a492c60-d3a8-417b-b45f-254e82f2ae19-kube-api-access-965xj\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.733596 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f7f681b-d410-40e9-a4d0-487230d8dece-cni-binary-copy\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.733596 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-daemon-config\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.733596 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-dbus\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733634    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-registration-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733658    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-cnibin\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733692    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-dbus\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733739    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733756    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysctl-d\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733813    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysctl-conf\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gt7\" (UniqueName: \"kubernetes.io/projected/f57a8017-198c-42a9-9fb4-481348d8568c-kube-api-access-x4gt7\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733910    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-multus-certs\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733962    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp42j\" (UniqueName: \"kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j\") pod \"network-check-target-ttbvc\" (UID: \"5a8d8001-c1b0-4489-9e25-fb8b67048d7d\") " pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.733990    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-kubelet\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734017    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-systemd\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734048    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysconfig\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734097    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-host\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.734121 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af477ca4-5745-4345-8e68-21ac9c32221f-tmp\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734180    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f57a8017-198c-42a9-9fb4-481348d8568c-serviceca\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734193    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734210    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-cni-bin\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734236    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-ovnkube-script-lib\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734275    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4nvm\" (UniqueName: \"kubernetes.io/projected/e263af32-db0f-4984-8e90-83347cdd15f7-kube-api-access-x4nvm\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734293    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-lib-modules\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32d4bb3a-8abf-408a-8168-4581172f47f1-host-slash\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2rxv\" (UniqueName: \"kubernetes.io/projected/125c3968-dca4-4163-b473-b5cc7ac5a23f-kube-api-access-r2rxv\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-cnibin\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734408    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-etc-selinux\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734433    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-conf-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734465    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734491    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-cni-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/125c3968-dca4-4163-b473-b5cc7ac5a23f-hosts-file\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734526    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-cnibin\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734557    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.734796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734566    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/125c3968-dca4-4163-b473-b5cc7ac5a23f-hosts-file\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734589    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f57a8017-198c-42a9-9fb4-481348d8568c-serviceca\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734598    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734625    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-etc-kubernetes\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734652    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee4a4694-e358-4491-b4e1-c05bd477f19a-agent-certs\") pod \"konnectivity-agent-g4jmf\" (UID: \"ee4a4694-e358-4491-b4e1-c05bd477f19a\") " pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-var-lib-kubelet\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:25.734701    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734709    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-kubelet\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:25.734770    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret podName:4a5d501e-fa40-467b-a4fa-9fc52f9c45fa nodeName:}" failed. No retries permitted until 2026-11-03 01:17:26.234753684 +0000 UTC m=+3.091742283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret") pod "global-pull-secret-syncer-b4htm" (UID: "4a5d501e-fa40-467b-a4fa-9fc52f9c45fa") : object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734785    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734807    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-systemd\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-kubelet-config\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734878    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-node-log\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734885    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-kubelet-config\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734894    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-env-overrides\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f57a8017-198c-42a9-9fb4-481348d8568c-host\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734951    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-socket-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.735317 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.734967    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-netns\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.735964 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.735002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-cni-netd\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.735964 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.735023    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f57a8017-198c-42a9-9fb4-481348d8568c-host\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:17:25.735964 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.735047    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-kubernetes\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.735964 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.735064    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32d4bb3a-8abf-408a-8168-4581172f47f1-iptables-alerter-script\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:17:25.735964 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.735098    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/125c3968-dca4-4163-b473-b5cc7ac5a23f-tmp-dir\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:17:25.735964 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.735860    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/125c3968-dca4-4163-b473-b5cc7ac5a23f-tmp-dir\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:17:25.735964 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.735913    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.768766 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.768705    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:17:25.768766 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.768732    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t8fl2\""
Nov 03 01:17:25.768766 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.768743    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 01:17:25.769061 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.768772    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 01:17:25.769061 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.768780    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:17:25.769061 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.768783    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:17:25.769061 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.768785    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 01:17:25.769061 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.768812    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 01:17:25.769061 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.768865    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m8klw"
Nov 03 01:17:25.797590 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.797569    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 01:17:25.801390 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.801369    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4gt7\" (UniqueName: \"kubernetes.io/projected/f57a8017-198c-42a9-9fb4-481348d8568c-kube-api-access-x4gt7\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:17:25.801497 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.801463    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqwfz\" (UniqueName: \"kubernetes.io/projected/2c57dcf4-5aa3-4992-b357-47b388f3db90-kube-api-access-jqwfz\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.801497 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.801463    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2rxv\" (UniqueName: \"kubernetes.io/projected/125c3968-dca4-4163-b473-b5cc7ac5a23f-kube-api-access-r2rxv\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:17:25.835947 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.835924    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a4694-e358-4491-b4e1-c05bd477f19a-konnectivity-ca\") pod \"konnectivity-agent-g4jmf\" (UID: \"ee4a4694-e358-4491-b4e1-c05bd477f19a\") " pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:17:25.836038 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.835955    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.836038 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.835971    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-system-cni-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.836038 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.835987    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-log-socket\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.836162 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836043    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.836162 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836074    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-run\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.836162 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836078    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-system-cni-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.836162 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836093    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-965xj\" (UniqueName: \"kubernetes.io/projected/3a492c60-d3a8-417b-b45f-254e82f2ae19-kube-api-access-965xj\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.836162 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836145    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-run\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.836162 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836149    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-log-socket\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836199    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f7f681b-d410-40e9-a4d0-487230d8dece-cni-binary-copy\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836229    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-daemon-config\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-registration-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836316    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-cnibin\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836377    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-registration-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836387    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysctl-d\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysctl-conf\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836423    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-cnibin\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836443    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-multus-certs\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836472    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp42j\" (UniqueName: \"kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j\") pod \"network-check-target-ttbvc\" (UID: \"5a8d8001-c1b0-4489-9e25-fb8b67048d7d\") " pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836479    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-multus-certs\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836445    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836443    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a4694-e358-4491-b4e1-c05bd477f19a-konnectivity-ca\") pod \"konnectivity-agent-g4jmf\" (UID: \"ee4a4694-e358-4491-b4e1-c05bd477f19a\") " pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:17:25.836512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836512    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-kubelet\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836513    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysctl-d\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-systemd\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836531    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysctl-conf\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836550    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-kubelet\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836563    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysconfig\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836592    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysconfig\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836593    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-systemd\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836595    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-host\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836627    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af477ca4-5745-4345-8e68-21ac9c32221f-tmp\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836634    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-host\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836655    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-cni-bin\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836679    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-ovnkube-script-lib\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836708    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-cni-bin\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836712    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f7f681b-d410-40e9-a4d0-487230d8dece-cni-binary-copy\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4nvm\" (UniqueName: \"kubernetes.io/projected/e263af32-db0f-4984-8e90-83347cdd15f7-kube-api-access-x4nvm\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836757    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-lib-modules\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32d4bb3a-8abf-408a-8168-4581172f47f1-host-slash\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:17:25.837295 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836797    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-etc-selinux\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-conf-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836838    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-cni-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836868    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-lib-modules\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836924    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-conf-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836929    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836951    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-etc-selinux\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-etc-kubernetes\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836980    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32d4bb3a-8abf-408a-8168-4581172f47f1-host-slash\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836987    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-etc-kubernetes\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.836988    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee4a4694-e358-4491-b4e1-c05bd477f19a-agent-certs\") pod \"konnectivity-agent-g4jmf\" (UID: \"ee4a4694-e358-4491-b4e1-c05bd477f19a\") " pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837007    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-cni-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-var-lib-kubelet\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837053    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-kubelet\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837074    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-daemon-config\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837120    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-systemd\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.838285 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837121    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-var-lib-kubelet\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837143    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-kubelet\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-node-log\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837188    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-env-overrides\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837197    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-node-log\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:25.837209    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837214    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-socket-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-netns\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837251    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-systemd\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:25.837259    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs podName:3d824491-3dcc-4c32-8378-7c8459b34086 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:26.337242524 +0000 UTC m=+3.194231118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs") pod "network-metrics-daemon-vxv8x" (UID: "3d824491-3dcc-4c32-8378-7c8459b34086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837274    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-netns\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-cni-netd\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837313    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-kubernetes\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837313    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-ovnkube-script-lib\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837338    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32d4bb3a-8abf-408a-8168-4581172f47f1-iptables-alerter-script\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837367    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-cni-netd\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837376    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-socket-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.838897 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-cni-multus\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837389    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-kubernetes\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837426    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-cni-multus\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837459    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwx2t\" (UniqueName: \"kubernetes.io/projected/3d824491-3dcc-4c32-8378-7c8459b34086-kube-api-access-mwx2t\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-ovn\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837502    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-cni-bin\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837540    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-ovnkube-config\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837565    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-slash\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837573    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-cni-bin\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-etc-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837615    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-ovn\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837616    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-modprobe-d\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837649    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-sys\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837674    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af477ca4-5745-4345-8e68-21ac9c32221f-etc-tuned\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837694    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-modprobe-d\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-627v6\" (UniqueName: \"kubernetes.io/projected/af477ca4-5745-4345-8e68-21ac9c32221f-kube-api-access-627v6\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.839398 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837728    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz2vv\" (UniqueName: \"kubernetes.io/projected/32d4bb3a-8abf-408a-8168-4581172f47f1-kube-api-access-wz2vv\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837735    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-slash\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837758    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-sys-fs\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837785    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-device-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837811    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-run-netns\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837818    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-env-overrides\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837837    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-var-lib-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837820    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837864    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e263af32-db0f-4984-8e90-83347cdd15f7-ovn-node-metrics-cert\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-systemd-units\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837927    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-socket-dir-parent\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837952    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-k8s-cni-cncf-io\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-hostroot\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.838007    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4hm8\" (UniqueName: \"kubernetes.io/projected/2f7f681b-d410-40e9-a4d0-487230d8dece-kube-api-access-s4hm8\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.838081    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32d4bb3a-8abf-408a-8168-4581172f47f1-iptables-alerter-script\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.838151    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-ovnkube-config\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.838166    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-var-lib-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.839912 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.838199    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-sys\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.840412 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.837786    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-etc-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.840412 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.838233    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-socket-dir-parent\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.840412 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.838236    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-k8s-cni-cncf-io\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.840412 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.838247    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-sys-fs\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.840412 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.838284    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-hostroot\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.840412 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.838287    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-device-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.840412 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.838323    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-run-netns\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.840412 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.838344    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-systemd-units\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.840412 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.839020    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af477ca4-5745-4345-8e68-21ac9c32221f-tmp\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.840412 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.839654    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee4a4694-e358-4491-b4e1-c05bd477f19a-agent-certs\") pod \"konnectivity-agent-g4jmf\" (UID: \"ee4a4694-e358-4491-b4e1-c05bd477f19a\") " pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:17:25.840412 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.839999    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af477ca4-5745-4345-8e68-21ac9c32221f-etc-tuned\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.840412 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.840409    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e263af32-db0f-4984-8e90-83347cdd15f7-ovn-node-metrics-cert\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.900293 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:25.900270    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 01:17:25.900293 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:25.900292    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 01:17:25.900428 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:25.900301    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dp42j for pod openshift-network-diagnostics/network-check-target-ttbvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:25.900428 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:25.900371    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j podName:5a8d8001-c1b0-4489-9e25-fb8b67048d7d nodeName:}" failed. No retries permitted until 2026-11-03 01:17:26.400335968 +0000 UTC m=+3.257324558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dp42j" (UniqueName: "kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j") pod "network-check-target-ttbvc" (UID: "5a8d8001-c1b0-4489-9e25-fb8b67048d7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:25.901623 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.901601    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-965xj\" (UniqueName: \"kubernetes.io/projected/3a492c60-d3a8-417b-b45f-254e82f2ae19-kube-api-access-965xj\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.902650 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.902629    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4nvm\" (UniqueName: \"kubernetes.io/projected/e263af32-db0f-4984-8e90-83347cdd15f7-kube-api-access-x4nvm\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.903450 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.903417    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwx2t\" (UniqueName: \"kubernetes.io/projected/3d824491-3dcc-4c32-8378-7c8459b34086-kube-api-access-mwx2t\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:25.905625 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.905605    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4hm8\" (UniqueName: \"kubernetes.io/projected/2f7f681b-d410-40e9-a4d0-487230d8dece-kube-api-access-s4hm8\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.905699 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.905679    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz2vv\" (UniqueName: \"kubernetes.io/projected/32d4bb3a-8abf-408a-8168-4581172f47f1-kube-api-access-wz2vv\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:17:25.906003 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.905988    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-627v6\" (UniqueName: \"kubernetes.io/projected/af477ca4-5745-4345-8e68-21ac9c32221f-kube-api-access-627v6\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.924229 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.924214    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-287jt"
Nov 03 01:17:25.930490 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.930476    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:17:25.934565 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:25.934546    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125c3968_dca4_4163_b473_b5cc7ac5a23f.slice/crio-85ece4a25e76922c64587102669c48816d4b7c3f1a4dc7929269a6296bd3e586 WatchSource:0}: Error finding container 85ece4a25e76922c64587102669c48816d4b7c3f1a4dc7929269a6296bd3e586: Status 404 returned error can't find the container with id 85ece4a25e76922c64587102669c48816d4b7c3f1a4dc7929269a6296bd3e586
Nov 03 01:17:25.938947 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:25.938925    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57a8017_198c_42a9_9fb4_481348d8568c.slice/crio-af02d4102d3d30d1c8d8279100a03aa5439e262ea74992ca242c0f330dcb4f3d WatchSource:0}: Error finding container af02d4102d3d30d1c8d8279100a03aa5439e262ea74992ca242c0f330dcb4f3d: Status 404 returned error can't find the container with id af02d4102d3d30d1c8d8279100a03aa5439e262ea74992ca242c0f330dcb4f3d
Nov 03 01:17:25.949059 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.949042    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:17:25.953869 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.953852    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:17:25.958557 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:25.958534    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4a4694_e358_4491_b4e1_c05bd477f19a.slice/crio-8d47f5ae5cf9b860ed0482d0634fb887e4bbaff1cd439d70bb2b1ccd7cb3b0ff WatchSource:0}: Error finding container 8d47f5ae5cf9b860ed0482d0634fb887e4bbaff1cd439d70bb2b1ccd7cb3b0ff: Status 404 returned error can't find the container with id 8d47f5ae5cf9b860ed0482d0634fb887e4bbaff1cd439d70bb2b1ccd7cb3b0ff
Nov 03 01:17:25.962270 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:25.962252    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c57dcf4_5aa3_4992_b357_47b388f3db90.slice/crio-802890bbb04d593748e26313e577cb888c7d3021afd8886c63fb74d4a641ff18 WatchSource:0}: Error finding container 802890bbb04d593748e26313e577cb888c7d3021afd8886c63fb74d4a641ff18: Status 404 returned error can't find the container with id 802890bbb04d593748e26313e577cb888c7d3021afd8886c63fb74d4a641ff18
Nov 03 01:17:25.964066 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.964044    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:17:25.968880 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.968864    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:17:25.971119 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.971098    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-46tlx"
Nov 03 01:17:25.972568 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:25.972538    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf477ca4_5745_4345_8e68_21ac9c32221f.slice/crio-a16d3db883c744089c57d4dac0f355a2f506bca4718fdaefe798069bec60c267 WatchSource:0}: Error finding container a16d3db883c744089c57d4dac0f355a2f506bca4718fdaefe798069bec60c267: Status 404 returned error can't find the container with id a16d3db883c744089c57d4dac0f355a2f506bca4718fdaefe798069bec60c267
Nov 03 01:17:25.976985 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.976968    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:17:25.977101 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:25.977038    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a492c60_d3a8_417b_b45f_254e82f2ae19.slice/crio-3dc67cea4c5f3357064a3154d04a1e9e6df73c9d41b25a0e72c2d4faa0b9ef7e WatchSource:0}: Error finding container 3dc67cea4c5f3357064a3154d04a1e9e6df73c9d41b25a0e72c2d4faa0b9ef7e: Status 404 returned error can't find the container with id 3dc67cea4c5f3357064a3154d04a1e9e6df73c9d41b25a0e72c2d4faa0b9ef7e
Nov 03 01:17:25.980776 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:25.980754    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7f681b_d410_40e9_a4d0_487230d8dece.slice/crio-43a30ba52f58f20dddbae3241d362c4a7e8a1e3aeb090b9cbd6fe066a476c494 WatchSource:0}: Error finding container 43a30ba52f58f20dddbae3241d362c4a7e8a1e3aeb090b9cbd6fe066a476c494: Status 404 returned error can't find the container with id 43a30ba52f58f20dddbae3241d362c4a7e8a1e3aeb090b9cbd6fe066a476c494
Nov 03 01:17:25.981482 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:25.981465    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:25.986902 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:25.986884    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d4bb3a_8abf_408a_8168_4581172f47f1.slice/crio-ccd809ce8e6cd82cf0867fe0496a4e5e03174f8099bdbf41a43f338214d4cf7e WatchSource:0}: Error finding container ccd809ce8e6cd82cf0867fe0496a4e5e03174f8099bdbf41a43f338214d4cf7e: Status 404 returned error can't find the container with id ccd809ce8e6cd82cf0867fe0496a4e5e03174f8099bdbf41a43f338214d4cf7e
Nov 03 01:17:25.991155 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:17:25.991138    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode263af32_db0f_4984_8e90_83347cdd15f7.slice/crio-f5fa2ad137caf5bd6425053cc5496d81379b1fb1e5e7ad5046731d516d42c366 WatchSource:0}: Error finding container f5fa2ad137caf5bd6425053cc5496d81379b1fb1e5e7ad5046731d516d42c366: Status 404 returned error can't find the container with id f5fa2ad137caf5bd6425053cc5496d81379b1fb1e5e7ad5046731d516d42c366
Nov 03 01:17:26.240569 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.240534    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:26.240725 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:26.240664    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:26.240725 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:26.240723    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret podName:4a5d501e-fa40-467b-a4fa-9fc52f9c45fa nodeName:}" failed. No retries permitted until 2026-11-03 01:17:27.240708252 +0000 UTC m=+4.097696830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret") pod "global-pull-secret-syncer-b4htm" (UID: "4a5d501e-fa40-467b-a4fa-9fc52f9c45fa") : object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:26.340883 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.340853    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:26.341020 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:26.340968    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:26.341020 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:26.341015    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs podName:3d824491-3dcc-4c32-8378-7c8459b34086 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:27.341001597 +0000 UTC m=+4.197990173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs") pod "network-metrics-daemon-vxv8x" (UID: "3d824491-3dcc-4c32-8378-7c8459b34086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:26.441272 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.441242    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp42j\" (UniqueName: \"kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j\") pod \"network-check-target-ttbvc\" (UID: \"5a8d8001-c1b0-4489-9e25-fb8b67048d7d\") " pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:26.441409 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:26.441392    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 01:17:26.441409 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:26.441406    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 01:17:26.441501 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:26.441415    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dp42j for pod openshift-network-diagnostics/network-check-target-ttbvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:26.441501 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:26.441462    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j podName:5a8d8001-c1b0-4489-9e25-fb8b67048d7d nodeName:}" failed. No retries permitted until 2026-11-03 01:17:27.441447531 +0000 UTC m=+4.298436107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dp42j" (UniqueName: "kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j") pod "network-check-target-ttbvc" (UID: "5a8d8001-c1b0-4489-9e25-fb8b67048d7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:26.706040 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.706009    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:26.706599 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.706014    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:26.706599 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:26.706117    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:26.706599 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:26.706193    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:26.715759 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.715727    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qv8q5" event={"ID":"2c57dcf4-5aa3-4992-b357-47b388f3db90","Type":"ContainerStarted","Data":"802890bbb04d593748e26313e577cb888c7d3021afd8886c63fb74d4a641ff18"}
Nov 03 01:17:26.716931 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.716906    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g4jmf" event={"ID":"ee4a4694-e358-4491-b4e1-c05bd477f19a","Type":"ContainerStarted","Data":"8d47f5ae5cf9b860ed0482d0634fb887e4bbaff1cd439d70bb2b1ccd7cb3b0ff"}
Nov 03 01:17:26.718134 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.718109    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-287jt" event={"ID":"125c3968-dca4-4163-b473-b5cc7ac5a23f","Type":"ContainerStarted","Data":"85ece4a25e76922c64587102669c48816d4b7c3f1a4dc7929269a6296bd3e586"}
Nov 03 01:17:26.719445 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.719392    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-46tlx" event={"ID":"2f7f681b-d410-40e9-a4d0-487230d8dece","Type":"ContainerStarted","Data":"43a30ba52f58f20dddbae3241d362c4a7e8a1e3aeb090b9cbd6fe066a476c494"}
Nov 03 01:17:26.720841 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.720758    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj" event={"ID":"3a492c60-d3a8-417b-b45f-254e82f2ae19","Type":"ContainerStarted","Data":"3dc67cea4c5f3357064a3154d04a1e9e6df73c9d41b25a0e72c2d4faa0b9ef7e"}
Nov 03 01:17:26.722100 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.722074    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vqm5v" event={"ID":"af477ca4-5745-4345-8e68-21ac9c32221f","Type":"ContainerStarted","Data":"a16d3db883c744089c57d4dac0f355a2f506bca4718fdaefe798069bec60c267"}
Nov 03 01:17:26.723334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.723291    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8flnl" event={"ID":"f57a8017-198c-42a9-9fb4-481348d8568c","Type":"ContainerStarted","Data":"af02d4102d3d30d1c8d8279100a03aa5439e262ea74992ca242c0f330dcb4f3d"}
Nov 03 01:17:26.724702 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.724679    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr" event={"ID":"e263af32-db0f-4984-8e90-83347cdd15f7","Type":"ContainerStarted","Data":"f5fa2ad137caf5bd6425053cc5496d81379b1fb1e5e7ad5046731d516d42c366"}
Nov 03 01:17:26.725879 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.725850    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7xkjx" event={"ID":"32d4bb3a-8abf-408a-8168-4581172f47f1","Type":"ContainerStarted","Data":"ccd809ce8e6cd82cf0867fe0496a4e5e03174f8099bdbf41a43f338214d4cf7e"}
Nov 03 01:17:26.773191 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.773125    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:12:26 +0000 UTC" deadline="2028-08-17 23:00:41.45782004 +0000 UTC"
Nov 03 01:17:26.773191 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:26.773192    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15693h43m14.68463298s"
Nov 03 01:17:27.246139 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:27.246101    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:27.246352 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:27.246280    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:27.246352 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:27.246344    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret podName:4a5d501e-fa40-467b-a4fa-9fc52f9c45fa nodeName:}" failed. No retries permitted until 2026-11-03 01:17:29.246326703 +0000 UTC m=+6.103315279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret") pod "global-pull-secret-syncer-b4htm" (UID: "4a5d501e-fa40-467b-a4fa-9fc52f9c45fa") : object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:27.346985 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:27.346895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:27.347162 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:27.347035    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:27.347162 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:27.347108    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs podName:3d824491-3dcc-4c32-8378-7c8459b34086 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:29.347078201 +0000 UTC m=+6.204066782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs") pod "network-metrics-daemon-vxv8x" (UID: "3d824491-3dcc-4c32-8378-7c8459b34086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:27.447855 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:27.447810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp42j\" (UniqueName: \"kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j\") pod \"network-check-target-ttbvc\" (UID: \"5a8d8001-c1b0-4489-9e25-fb8b67048d7d\") " pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:27.460816 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:27.454213    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 01:17:27.460816 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:27.454247    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 01:17:27.460816 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:27.454272    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dp42j for pod openshift-network-diagnostics/network-check-target-ttbvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:27.460816 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:27.454350    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j podName:5a8d8001-c1b0-4489-9e25-fb8b67048d7d nodeName:}" failed. No retries permitted until 2026-11-03 01:17:29.454321367 +0000 UTC m=+6.311309951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dp42j" (UniqueName: "kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j") pod "network-check-target-ttbvc" (UID: "5a8d8001-c1b0-4489-9e25-fb8b67048d7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:27.705400 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:27.705310    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:27.705582 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:27.705457    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:27.774165 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:27.774076    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:12:26 +0000 UTC" deadline="2028-05-01 01:54:42.033740833 +0000 UTC"
Nov 03 01:17:27.774165 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:27.774136    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13080h37m14.259609194s"
Nov 03 01:17:28.706298 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:28.706221    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:28.706498 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:28.706362    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:28.706498 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:28.706461    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:28.706680 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:28.706555    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:29.267613 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:29.267324    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:29.267613 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:29.267508    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:29.267613 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:29.267576    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret podName:4a5d501e-fa40-467b-a4fa-9fc52f9c45fa nodeName:}" failed. No retries permitted until 2026-11-03 01:17:33.267557458 +0000 UTC m=+10.124546040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret") pod "global-pull-secret-syncer-b4htm" (UID: "4a5d501e-fa40-467b-a4fa-9fc52f9c45fa") : object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:29.368952 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:29.368684    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:29.368952 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:29.368839    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:29.368952 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:29.368912    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs podName:3d824491-3dcc-4c32-8378-7c8459b34086 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:33.368889481 +0000 UTC m=+10.225878057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs") pod "network-metrics-daemon-vxv8x" (UID: "3d824491-3dcc-4c32-8378-7c8459b34086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:29.470744 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:29.470678    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp42j\" (UniqueName: \"kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j\") pod \"network-check-target-ttbvc\" (UID: \"5a8d8001-c1b0-4489-9e25-fb8b67048d7d\") " pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:29.470943 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:29.470923    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 01:17:29.471006 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:29.470945    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 01:17:29.471006 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:29.470960    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dp42j for pod openshift-network-diagnostics/network-check-target-ttbvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:29.471113 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:29.471022    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j podName:5a8d8001-c1b0-4489-9e25-fb8b67048d7d nodeName:}" failed. No retries permitted until 2026-11-03 01:17:33.471002869 +0000 UTC m=+10.327991451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dp42j" (UniqueName: "kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j") pod "network-check-target-ttbvc" (UID: "5a8d8001-c1b0-4489-9e25-fb8b67048d7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:29.708541 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:29.706592    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:29.708541 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:29.706752    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:30.706077 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:30.705838    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:30.706606 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:30.706215    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:30.706606 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:30.706288    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:30.706606 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:30.706388    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:30.744003 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:30.742993    2575 generic.go:356] "Generic (PLEG): container finished" podID="bd3c5a92e769172fde5847bb75e6fd4d" containerID="0bab3edfc0f815357e2d56c2a6bacd1b4a446e9d135e35231ae74b4f35a623ec" exitCode=0
Nov 03 01:17:30.744003 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:30.743428    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal" event={"ID":"bd3c5a92e769172fde5847bb75e6fd4d","Type":"ContainerDied","Data":"0bab3edfc0f815357e2d56c2a6bacd1b4a446e9d135e35231ae74b4f35a623ec"}
Nov 03 01:17:30.747716 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:30.747071    2575 generic.go:356] "Generic (PLEG): container finished" podID="2c57dcf4-5aa3-4992-b357-47b388f3db90" containerID="84284ba03e65fc56b0b2da0ca8b11dc86f03666733ab714fb1543bd473b7019f" exitCode=0
Nov 03 01:17:30.747716 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:30.747115    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qv8q5" event={"ID":"2c57dcf4-5aa3-4992-b357-47b388f3db90","Type":"ContainerDied","Data":"84284ba03e65fc56b0b2da0ca8b11dc86f03666733ab714fb1543bd473b7019f"}
Nov 03 01:17:31.708482 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:31.708452    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:31.708954 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:31.708587    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:31.754502 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:31.754462    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal" event={"ID":"0eb27a029b0b668ca0885e8f9e1efac2","Type":"ContainerStarted","Data":"8b9657b6479991cce2f4f877eab66e4c1f9dd243cebbe76b19f493d26246f6c4"}
Nov 03 01:17:31.760675 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:31.760608    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal" event={"ID":"bd3c5a92e769172fde5847bb75e6fd4d","Type":"ContainerStarted","Data":"07afb357b6e0480360e728d76adc242e764fa276ec056dd07a2d57e4afd7b6c8"}
Nov 03 01:17:31.767471 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:31.767446    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g4jmf" event={"ID":"ee4a4694-e358-4491-b4e1-c05bd477f19a","Type":"ContainerStarted","Data":"75485bcf85eb9000613356beb608fe6b3da246253204c7943582376258e8c696"}
Nov 03 01:17:31.881712 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:31.881637    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal" podStartSLOduration=6.881616197 podStartE2EDuration="6.881616197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:31.880928314 +0000 UTC m=+8.737916916" watchObservedRunningTime="2026-11-03 01:17:31.881616197 +0000 UTC m=+8.738604803"
Nov 03 01:17:32.006586 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:32.006530    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g4jmf" podStartSLOduration=1.455087137 podStartE2EDuration="7.006509052s" totalImagesPullingTime="5.551421915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:32.005598698 +0000 UTC m=+8.862587300" watchObservedRunningTime="2026-11-03 01:17:32.006509052 +0000 UTC m=+8.863497652"
Nov 03 01:17:32.706394 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:32.705643    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:32.706394 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:32.705811    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:32.706394 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:32.705909    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:32.706394 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:32.706006    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:32.777022 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:32.775946    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8flnl" event={"ID":"f57a8017-198c-42a9-9fb4-481348d8568c","Type":"ContainerStarted","Data":"fab80bb22a5a247270bb6b6fc030d8116fc4c52b44b40ee65675130e1d91e20c"}
Nov 03 01:17:32.901400 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:32.901091    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal" podStartSLOduration=7.9010765240000005 podStartE2EDuration="7.901076524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:32.132805273 +0000 UTC m=+8.989793876" watchObservedRunningTime="2026-11-03 01:17:32.901076524 +0000 UTC m=+9.758065124"
Nov 03 01:17:33.304054 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:33.304008    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:33.304238 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:33.304175    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:33.304238 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:33.304238    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret podName:4a5d501e-fa40-467b-a4fa-9fc52f9c45fa nodeName:}" failed. No retries permitted until 2026-11-03 01:17:41.304220185 +0000 UTC m=+18.161208780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret") pod "global-pull-secret-syncer-b4htm" (UID: "4a5d501e-fa40-467b-a4fa-9fc52f9c45fa") : object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:33.404598 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:33.404526    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:33.404764 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:33.404680    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:33.404764 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:33.404741    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs podName:3d824491-3dcc-4c32-8378-7c8459b34086 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:41.404723435 +0000 UTC m=+18.261712011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs") pod "network-metrics-daemon-vxv8x" (UID: "3d824491-3dcc-4c32-8378-7c8459b34086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:33.505377 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:33.505291    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp42j\" (UniqueName: \"kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j\") pod \"network-check-target-ttbvc\" (UID: \"5a8d8001-c1b0-4489-9e25-fb8b67048d7d\") " pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:33.505590 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:33.505510    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 01:17:33.505590 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:33.505532    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 01:17:33.505590 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:33.505545    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dp42j for pod openshift-network-diagnostics/network-check-target-ttbvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:33.505771 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:33.505610    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j podName:5a8d8001-c1b0-4489-9e25-fb8b67048d7d nodeName:}" failed. No retries permitted until 2026-11-03 01:17:41.505593129 +0000 UTC m=+18.362581706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dp42j" (UniqueName: "kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j") pod "network-check-target-ttbvc" (UID: "5a8d8001-c1b0-4489-9e25-fb8b67048d7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:33.715587 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:33.715555    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:33.715887 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:33.715866    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:33.779420 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:33.779266    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj" event={"ID":"3a492c60-d3a8-417b-b45f-254e82f2ae19","Type":"ContainerStarted","Data":"bca017d5e513be298806e945c75117cc0600dcbe5b9bdb3f392b03dedc256392"}
Nov 03 01:17:34.705521 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:34.705484    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:34.705766 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:34.705543    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:34.705766 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:34.705649    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:34.705766 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:34.705733    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:35.706679 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:35.706519    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:35.706679 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:35.706629    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:35.787645 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:35.787592    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-287jt" event={"ID":"125c3968-dca4-4163-b473-b5cc7ac5a23f","Type":"ContainerStarted","Data":"ebd866e6da8c598f18d909a4ebac5952f621dad321b91db323d7b0434605954d"}
Nov 03 01:17:35.789861 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:35.789830    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-46tlx" event={"ID":"2f7f681b-d410-40e9-a4d0-487230d8dece","Type":"ContainerStarted","Data":"0ed9131762c4dd89424fce41cdb16358a74a9fda207c817bc29ff4898d769bd0"}
Nov 03 01:17:35.916680 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:35.916629    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-287jt" podStartSLOduration=1.76594743 podStartE2EDuration="10.916614643s" totalImagesPullingTime="9.150667213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:35.916031634 +0000 UTC m=+12.773020236" watchObservedRunningTime="2026-11-03 01:17:35.916614643 +0000 UTC m=+12.773603224"
Nov 03 01:17:35.916854 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:35.916771    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8flnl" podStartSLOduration=4.94706279 podStartE2EDuration="10.916763169s" totalImagesPullingTime="5.969700379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:32.901383159 +0000 UTC m=+9.758371759" watchObservedRunningTime="2026-11-03 01:17:35.916763169 +0000 UTC m=+12.773751767"
Nov 03 01:17:35.997098 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:35.997054    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 01:17:36.044584 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:36.044533    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-46tlx" podStartSLOduration=2.315805708 podStartE2EDuration="11.044516607s" totalImagesPullingTime="8.728710899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:36.043834992 +0000 UTC m=+12.900823592" watchObservedRunningTime="2026-11-03 01:17:36.044516607 +0000 UTC m=+12.901505206"
Nov 03 01:17:36.594745 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:36.594631    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T01:17:35.997075333Z","UUID":"f235fbca-5cf3-4c28-a36c-e124e2df8b25","Handler":null,"Name":"","Endpoint":""}
Nov 03 01:17:36.596471 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:36.596446    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 01:17:36.596594 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:36.596480    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 01:17:36.630417 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:36.630384    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:17:36.631070 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:36.631050    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:17:36.705734 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:36.705695    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:36.705734 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:36.705719    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:36.705935 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:36.705823    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:36.705935 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:36.705895    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:36.793130 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:36.793093    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj" event={"ID":"3a492c60-d3a8-417b-b45f-254e82f2ae19","Type":"ContainerStarted","Data":"b7beeb1e74d0094a41952964d21513b4b4c9413d753b09bfb7c43ba7debabd8c"}
Nov 03 01:17:36.794404 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:36.794382    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7xkjx" event={"ID":"32d4bb3a-8abf-408a-8168-4581172f47f1","Type":"ContainerStarted","Data":"725cc9f6d343525b23c6cca71f447ff2aa94de09938449503c522956d84ab071"}
Nov 03 01:17:36.795970 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:36.795949    2575 generic.go:356] "Generic (PLEG): container finished" podID="2c57dcf4-5aa3-4992-b357-47b388f3db90" containerID="a302200e3662c2c7b13cb7b62b70a0de5dcb1dc4fd80d18fb81302807db923e2" exitCode=0
Nov 03 01:17:36.796067 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:36.796016    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qv8q5" event={"ID":"2c57dcf4-5aa3-4992-b357-47b388f3db90","Type":"ContainerDied","Data":"a302200e3662c2c7b13cb7b62b70a0de5dcb1dc4fd80d18fb81302807db923e2"}
Nov 03 01:17:36.912792 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:36.912583    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7xkjx" podStartSLOduration=2.637004219 podStartE2EDuration="11.912566486s" totalImagesPullingTime="9.275562267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:36.91254717 +0000 UTC m=+13.769535766" watchObservedRunningTime="2026-11-03 01:17:36.912566486 +0000 UTC m=+13.769555083"
Nov 03 01:17:37.705735 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:37.705692    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:37.705923 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:37.705802    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:37.799414 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:37.799381    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vqm5v" event={"ID":"af477ca4-5745-4345-8e68-21ac9c32221f","Type":"ContainerStarted","Data":"b4f29a50465564f12ba03fa5bb9b666ae00f4edfb8391eff30571a580f9d7e5e"}
Nov 03 01:17:37.918073 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:37.918017    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vqm5v" podStartSLOduration=1.9833810939999998 podStartE2EDuration="12.918001703s" totalImagesPullingTime="10.934620609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:37.917645554 +0000 UTC m=+14.774634151" watchObservedRunningTime="2026-11-03 01:17:37.918001703 +0000 UTC m=+14.774990299"
Nov 03 01:17:38.705232 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:38.705197    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:38.705335 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:38.705263    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:38.705417 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:38.705395    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:38.705520 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:38.705479    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:38.804219 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:38.804187    2575 generic.go:356] "Generic (PLEG): container finished" podID="2c57dcf4-5aa3-4992-b357-47b388f3db90" containerID="2225b0e970fda624a9e6c39a207bdc3947ef3e9d572abe0ee930fce0dc18e6fb" exitCode=0
Nov 03 01:17:38.804883 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:38.804270    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qv8q5" event={"ID":"2c57dcf4-5aa3-4992-b357-47b388f3db90","Type":"ContainerDied","Data":"2225b0e970fda624a9e6c39a207bdc3947ef3e9d572abe0ee930fce0dc18e6fb"}
Nov 03 01:17:38.806385 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:38.806364    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj" event={"ID":"3a492c60-d3a8-417b-b45f-254e82f2ae19","Type":"ContainerStarted","Data":"acb06a19a091d0faeecfc2ee9f897d122c8abe3175f5d87728e3787c3049fb8f"}
Nov 03 01:17:39.134542 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:39.134490    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj" podStartSLOduration=2.640316794 podStartE2EDuration="14.134476752s" totalImagesPullingTime="11.494159958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:39.133874861 +0000 UTC m=+15.990863457" watchObservedRunningTime="2026-11-03 01:17:39.134476752 +0000 UTC m=+15.991465348"
Nov 03 01:17:39.706085 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:39.706055    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:39.706256 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:39.706179    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:40.705580 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:40.705494    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:40.706232 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:40.705494    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:40.706232 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:40.705625    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:40.706232 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:40.705714    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:40.812603 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:40.812568    2575 generic.go:356] "Generic (PLEG): container finished" podID="2c57dcf4-5aa3-4992-b357-47b388f3db90" containerID="58861f089f8441c24593827d53971956c1d0570b7a85ac5fda92968ad7dcf7dc" exitCode=0
Nov 03 01:17:40.812603 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:40.812607    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qv8q5" event={"ID":"2c57dcf4-5aa3-4992-b357-47b388f3db90","Type":"ContainerDied","Data":"58861f089f8441c24593827d53971956c1d0570b7a85ac5fda92968ad7dcf7dc"}
Nov 03 01:17:41.377299 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:41.377267    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:41.377476 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:41.377390    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:41.377476 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:41.377438    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret podName:4a5d501e-fa40-467b-a4fa-9fc52f9c45fa nodeName:}" failed. No retries permitted until 2026-11-03 01:17:57.377425724 +0000 UTC m=+34.234414298 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret") pod "global-pull-secret-syncer-b4htm" (UID: "4a5d501e-fa40-467b-a4fa-9fc52f9c45fa") : object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:41.479757 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:41.479726    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:41.479886 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:41.479845    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:41.479919 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:41.479901    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs podName:3d824491-3dcc-4c32-8378-7c8459b34086 nodeName:}" failed. No retries permitted until 2026-11-03 01:17:57.479884947 +0000 UTC m=+34.336873535 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs") pod "network-metrics-daemon-vxv8x" (UID: "3d824491-3dcc-4c32-8378-7c8459b34086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:41.580462 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:41.580434    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp42j\" (UniqueName: \"kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j\") pod \"network-check-target-ttbvc\" (UID: \"5a8d8001-c1b0-4489-9e25-fb8b67048d7d\") " pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:41.580606 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:41.580587    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 01:17:41.580644 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:41.580613    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 01:17:41.580644 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:41.580624    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dp42j for pod openshift-network-diagnostics/network-check-target-ttbvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:41.580707 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:41.580668    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j podName:5a8d8001-c1b0-4489-9e25-fb8b67048d7d nodeName:}" failed. No retries permitted until 2026-11-03 01:17:57.580654307 +0000 UTC m=+34.437642898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dp42j" (UniqueName: "kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j") pod "network-check-target-ttbvc" (UID: "5a8d8001-c1b0-4489-9e25-fb8b67048d7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:41.705902 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:41.705878    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:41.706274 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:41.705978    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:42.705727 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:42.705527    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:42.705921 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:42.705828    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:42.706309 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:42.705527    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:42.706309 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:42.706261    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:43.706516 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:43.706476    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:43.706978 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:43.706600    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:43.822427 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:43.822393    2575 generic.go:356] "Generic (PLEG): container finished" podID="2c57dcf4-5aa3-4992-b357-47b388f3db90" containerID="875bb96059a4fc0adaf42f12c6172991544deca6ba35a0c0e75d75f391d71088" exitCode=0
Nov 03 01:17:43.822585 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:43.822446    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qv8q5" event={"ID":"2c57dcf4-5aa3-4992-b357-47b388f3db90","Type":"ContainerDied","Data":"875bb96059a4fc0adaf42f12c6172991544deca6ba35a0c0e75d75f391d71088"}
Nov 03 01:17:44.705366 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:44.705320    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:44.705549 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:44.705372    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:44.705549 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:44.705467    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:44.705651 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:44.705561    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:44.829141 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:44.829108    2575 generic.go:356] "Generic (PLEG): container finished" podID="2c57dcf4-5aa3-4992-b357-47b388f3db90" containerID="2ee77a8535902aae0d33bd7822cda2d7b539c999517479538319fbf49fd26aa1" exitCode=0
Nov 03 01:17:44.829600 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:44.829148    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qv8q5" event={"ID":"2c57dcf4-5aa3-4992-b357-47b388f3db90","Type":"ContainerDied","Data":"2ee77a8535902aae0d33bd7822cda2d7b539c999517479538319fbf49fd26aa1"}
Nov 03 01:17:45.706085 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:45.706046    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:45.706251 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:45.706177    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:45.834777 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:45.834738    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qv8q5" event={"ID":"2c57dcf4-5aa3-4992-b357-47b388f3db90","Type":"ContainerStarted","Data":"4c9d6284b11ff1cd48c59b86a8e91a243ddf23acb7e8199c00c93b136af07e63"}
Nov 03 01:17:46.059557 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:46.059467    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qv8q5" podStartSLOduration=6.919825651 podStartE2EDuration="21.059452778s" totalImagesPullingTime="14.139627127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:46.008909765 +0000 UTC m=+22.865898360" watchObservedRunningTime="2026-11-03 01:17:46.059452778 +0000 UTC m=+22.916441373"
Nov 03 01:17:46.087811 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:46.087776    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-287jt_125c3968-dca4-4163-b473-b5cc7ac5a23f/dns-node-resolver/0.log"
Nov 03 01:17:46.705270 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:46.705236    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:46.705472 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:46.705238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:46.705472 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:46.705373    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:46.705556 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:46.705469    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:46.911169 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:46.911144    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8flnl_f57a8017-198c-42a9-9fb4-481348d8568c/node-ca/0.log"
Nov 03 01:17:47.705717 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:47.705523    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:47.705875 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:47.705795    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:47.841834 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:47.841801    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr" event={"ID":"e263af32-db0f-4984-8e90-83347cdd15f7","Type":"ContainerStarted","Data":"9321b70edd510f93a5d3722bf4a0f9fc938c41cae5a018643f3387477ac6b40f"}
Nov 03 01:17:47.841983 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:47.841870    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr" event={"ID":"e263af32-db0f-4984-8e90-83347cdd15f7","Type":"ContainerStarted","Data":"707277c268cb0e980eddee414d62bbabb2d1e1adb1f5da3244f042769c3ed3cf"}
Nov 03 01:17:47.841983 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:47.841902    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr" event={"ID":"e263af32-db0f-4984-8e90-83347cdd15f7","Type":"ContainerStarted","Data":"2708441eb13f2ee2e75b399c9c89908e0c88cf6bec74ef244980e87484ec77d4"}
Nov 03 01:17:47.841983 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:47.841931    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr" event={"ID":"e263af32-db0f-4984-8e90-83347cdd15f7","Type":"ContainerStarted","Data":"d53cc5b85872ccb2b0cbb390370b2664ec43a76568be5972ee5f2bc890c59e98"}
Nov 03 01:17:47.841983 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:47.841960    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr" event={"ID":"e263af32-db0f-4984-8e90-83347cdd15f7","Type":"ContainerStarted","Data":"73ccbc7862f5bd141b398af31c99fe51e5ab592b083bc2fbaa6ccdeaeca361ff"}
Nov 03 01:17:47.842124 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:47.841985    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr" event={"ID":"e263af32-db0f-4984-8e90-83347cdd15f7","Type":"ContainerStarted","Data":"ba5ee99275bc17f13c3da5d6454d378e39b7119acbc1d0d19fe06e6f0baf2805"}
Nov 03 01:17:48.705225 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:48.705181    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:48.705225 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:48.705230    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:48.705667 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:48.705322    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:48.705667 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:48.705443    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:49.705210 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:49.705157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:49.705369 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:49.705304    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:49.741396 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:49.741358    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:17:49.741575 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:49.741485    2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 01:17:49.741890 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:49.741876    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:17:49.848412 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:49.848374    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr" event={"ID":"e263af32-db0f-4984-8e90-83347cdd15f7","Type":"ContainerStarted","Data":"68a531878549cdea1339bc1c66f847cc7013feed4da70acd5d991ad687be98b0"}
Nov 03 01:17:50.705689 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:50.705632    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:50.705689 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:50.705693    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:50.706129 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:50.705778    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:50.706129 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:50.705894    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:51.705661 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:51.705623    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:51.705875 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:51.705736    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:51.855503 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:51.855466    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr" event={"ID":"e263af32-db0f-4984-8e90-83347cdd15f7","Type":"ContainerStarted","Data":"b3b2d5011687b4561c31f798a5ed9e61d7ac27d2518a33608942846f99ddeef4"}
Nov 03 01:17:51.855845 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:51.855814    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:51.870308 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:51.870281    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:52.136923 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:52.136878    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr" podStartSLOduration=6.286408623 podStartE2EDuration="27.136864805s" totalImagesPullingTime="20.850456182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:52.111148568 +0000 UTC m=+28.968137164" watchObservedRunningTime="2026-11-03 01:17:52.136864805 +0000 UTC m=+28.993853400"
Nov 03 01:17:52.705834 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:52.705803    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:52.706014 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:52.705803    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:52.706014 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:52.705922    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:52.706014 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:52.705990    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:52.857861 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:52.857826    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:52.858027 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:52.857912    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:52.872234 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:52.872211    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:17:53.706149 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:53.706109    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:53.706701 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:53.706215    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:54.705566 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:54.705525    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:54.705748 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:54.705531    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:54.705748 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:54.705641    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:54.705748 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:54.705724    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:55.706009 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:55.705981    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:55.706434 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:55.706090    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:56.705271 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:56.705235    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:56.705510 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:56.705280    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:56.705510 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:56.705377    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:56.705510 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:56.705460    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:57.394796 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:57.394764    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:57.395166 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:57.394869    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:57.395166 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:57.394915    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret podName:4a5d501e-fa40-467b-a4fa-9fc52f9c45fa nodeName:}" failed. No retries permitted until 2026-11-03 01:18:29.394902729 +0000 UTC m=+66.251891304 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret") pod "global-pull-secret-syncer-b4htm" (UID: "4a5d501e-fa40-467b-a4fa-9fc52f9c45fa") : object "kube-system"/"original-pull-secret" not registered
Nov 03 01:17:57.495364 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:57.495326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:57.495489 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:57.495432    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:57.495489 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:57.495476    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs podName:3d824491-3dcc-4c32-8378-7c8459b34086 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:29.495464902 +0000 UTC m=+66.352453479 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs") pod "network-metrics-daemon-vxv8x" (UID: "3d824491-3dcc-4c32-8378-7c8459b34086") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 01:17:57.596147 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:57.596122    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp42j\" (UniqueName: \"kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j\") pod \"network-check-target-ttbvc\" (UID: \"5a8d8001-c1b0-4489-9e25-fb8b67048d7d\") " pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:57.596257 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:57.596237    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 01:17:57.596257 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:57.596248    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 01:17:57.596257 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:57.596256    2575 projected.go:196] Error preparing data for projected volume kube-api-access-dp42j for pod openshift-network-diagnostics/network-check-target-ttbvc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:57.596386 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:57.596292    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j podName:5a8d8001-c1b0-4489-9e25-fb8b67048d7d nodeName:}" failed. No retries permitted until 2026-11-03 01:18:29.596281595 +0000 UTC m=+66.453270169 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dp42j" (UniqueName: "kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j") pod "network-check-target-ttbvc" (UID: "5a8d8001-c1b0-4489-9e25-fb8b67048d7d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 01:17:57.706044 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:57.706018    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:57.706163 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:57.706136    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:17:58.705692 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:58.705660    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:17:58.706056 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:58.705660    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:17:58.706056 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:58.705764    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:17:58.706056 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:58.705838    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:17:59.705759 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:17:59.705723    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:17:59.706187 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:17:59.705839    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:18:00.705238 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:00.705208    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:00.705450 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:00.705208    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:00.705450 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:00.705327    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:18:00.705450 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:00.705398    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:18:01.705860 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:01.705827    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:01.706258 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:01.705964    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:18:02.705156 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:02.705122    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:02.705334 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:02.705229    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:18:02.705334 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:02.705282    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:02.705428 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:02.705398    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:18:03.706857 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:03.706830    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:03.707231 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:03.706906    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:18:04.706052 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:04.706022    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:04.706052 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:04.706049    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:04.706233 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:04.706126    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:18:04.706276 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:04.706236    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:18:05.705468 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:05.705439    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:05.705806 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:05.705579    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:18:06.099825 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:06.099791    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b4htm"]
Nov 03 01:18:06.099975 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:06.099913    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:06.100033 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:06.100021    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:18:06.102682 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:06.102658    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ttbvc"]
Nov 03 01:18:06.102802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:06.102752    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:06.102876 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:06.102854    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:18:06.103125 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:06.103107    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vxv8x"]
Nov 03 01:18:06.103191 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:06.103182    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:06.103264 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:06.103251    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:18:07.705864 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:07.705830    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:07.706415 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:07.705932    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:18:07.706415 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:07.706015    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:07.706415 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:07.706015    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:07.706415 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:07.706139    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:18:07.706415 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:07.706215    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:18:09.706281 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:09.706241    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:09.706782 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:09.706237    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:09.706782 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:09.706366    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttbvc" podUID="5a8d8001-c1b0-4489-9e25-fb8b67048d7d"
Nov 03 01:18:09.706782 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:09.706253    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:09.706782 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:09.706481    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b4htm" podUID="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa"
Nov 03 01:18:09.706782 ip-10-0-140-255 kubenswrapper[2575]: E1103 01:18:09.706582    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxv8x" podUID="3d824491-3dcc-4c32-8378-7c8459b34086"
Nov 03 01:18:10.886669 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:10.886644    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeReady"
Nov 03 01:18:10.887041 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:10.886756    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 01:18:11.004847 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.004820    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bds75"]
Nov 03 01:18:11.006991 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.006974    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.054763 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.054742    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5pxx9"]
Nov 03 01:18:11.057502 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.057488    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.065462 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.065447    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 01:18:11.068162 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.068138    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwk4g\""
Nov 03 01:18:11.068276 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.068259    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 01:18:11.086994 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.086976    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rvpjl"]
Nov 03 01:18:11.089561 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.089547    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rvpjl"
Nov 03 01:18:11.097609 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.097590    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da04dce2-a300-44f6-ab49-5fb77c6c8a19-cert\") pod \"ingress-canary-rvpjl\" (UID: \"da04dce2-a300-44f6-ab49-5fb77c6c8a19\") " pod="openshift-ingress-canary/ingress-canary-rvpjl"
Nov 03 01:18:11.097688 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.097617    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.097688 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.097638    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8pqf\" (UniqueName: \"kubernetes.io/projected/da04dce2-a300-44f6-ab49-5fb77c6c8a19-kube-api-access-s8pqf\") pod \"ingress-canary-rvpjl\" (UID: \"da04dce2-a300-44f6-ab49-5fb77c6c8a19\") " pod="openshift-ingress-canary/ingress-canary-rvpjl"
Nov 03 01:18:11.097688 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.097652    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/571bdc94-d1ed-4ede-87fd-65af23ce24e9-config-volume\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.097797 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.097703    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/571bdc94-d1ed-4ede-87fd-65af23ce24e9-tmp-dir\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.097851 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.097836    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-crio-socket\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.097907 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.097894    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmkv6\" (UniqueName: \"kubernetes.io/projected/571bdc94-d1ed-4ede-87fd-65af23ce24e9-kube-api-access-zmkv6\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.097947 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.097917    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-data-volume\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.097980 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.097948    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmdcp\" (UniqueName: \"kubernetes.io/projected/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-kube-api-access-wmdcp\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.098011 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.097974    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571bdc94-d1ed-4ede-87fd-65af23ce24e9-metrics-tls\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.098011 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.098006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.105887 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.105870    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bds75"]
Nov 03 01:18:11.114374 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.112924    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5pxx9"]
Nov 03 01:18:11.149131 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.149113    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rvpjl"]
Nov 03 01:18:11.166337 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.166320    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9f8wp\""
Nov 03 01:18:11.166429 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.166333    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 01:18:11.166429 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.166320    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 01:18:11.166429 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.166385    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r69hg\""
Nov 03 01:18:11.168792 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.168774    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 01:18:11.168842 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.168778    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 01:18:11.168842 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.168785    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 01:18:11.169001 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.168990    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 01:18:11.169035 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.169010    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 01:18:11.198475 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.198455    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.198547 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.198490    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da04dce2-a300-44f6-ab49-5fb77c6c8a19-cert\") pod \"ingress-canary-rvpjl\" (UID: \"da04dce2-a300-44f6-ab49-5fb77c6c8a19\") " pod="openshift-ingress-canary/ingress-canary-rvpjl"
Nov 03 01:18:11.198547 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.198511    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.198547 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.198531    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8pqf\" (UniqueName: \"kubernetes.io/projected/da04dce2-a300-44f6-ab49-5fb77c6c8a19-kube-api-access-s8pqf\") pod \"ingress-canary-rvpjl\" (UID: \"da04dce2-a300-44f6-ab49-5fb77c6c8a19\") " pod="openshift-ingress-canary/ingress-canary-rvpjl"
Nov 03 01:18:11.198684 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.198546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/571bdc94-d1ed-4ede-87fd-65af23ce24e9-config-volume\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.198684 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.198563    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/571bdc94-d1ed-4ede-87fd-65af23ce24e9-tmp-dir\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.198684 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.198593    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-crio-socket\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.198684 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.198661    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmkv6\" (UniqueName: \"kubernetes.io/projected/571bdc94-d1ed-4ede-87fd-65af23ce24e9-kube-api-access-zmkv6\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.198826 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.198696    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-data-volume\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.198826 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.198746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdcp\" (UniqueName: \"kubernetes.io/projected/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-kube-api-access-wmdcp\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.198826 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.198777    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-crio-socket\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.198826 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.198819    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571bdc94-d1ed-4ede-87fd-65af23ce24e9-metrics-tls\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.199218 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.199191    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/571bdc94-d1ed-4ede-87fd-65af23ce24e9-tmp-dir\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.199442 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.199423    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/571bdc94-d1ed-4ede-87fd-65af23ce24e9-config-volume\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.199512 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.199443    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.199659 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.199638    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-data-volume\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.202270 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.202243    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.202343 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.202270    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571bdc94-d1ed-4ede-87fd-65af23ce24e9-metrics-tls\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.202457 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.202442    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da04dce2-a300-44f6-ab49-5fb77c6c8a19-cert\") pod \"ingress-canary-rvpjl\" (UID: \"da04dce2-a300-44f6-ab49-5fb77c6c8a19\") " pod="openshift-ingress-canary/ingress-canary-rvpjl"
Nov 03 01:18:11.259273 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.259253    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8pqf\" (UniqueName: \"kubernetes.io/projected/da04dce2-a300-44f6-ab49-5fb77c6c8a19-kube-api-access-s8pqf\") pod \"ingress-canary-rvpjl\" (UID: \"da04dce2-a300-44f6-ab49-5fb77c6c8a19\") " pod="openshift-ingress-canary/ingress-canary-rvpjl"
Nov 03 01:18:11.259378 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.259364    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdcp\" (UniqueName: \"kubernetes.io/projected/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-kube-api-access-wmdcp\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.259515 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.259497    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmkv6\" (UniqueName: \"kubernetes.io/projected/571bdc94-d1ed-4ede-87fd-65af23ce24e9-kube-api-access-zmkv6\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.319029 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.319011    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:11.370627 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.370583    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:11.399973 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.399454    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rvpjl"
Nov 03 01:18:11.700646 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.700568    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bds75"]
Nov 03 01:18:11.705264 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.705234    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:11.705400 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.705282    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:11.705400 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.705299    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:11.705510 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:18:11.705419    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571bdc94_d1ed_4ede_87fd_65af23ce24e9.slice/crio-ef0f59abb1c1d15c9ef032f36c068cd2b1f91f73d54c6b7e197180ac6a5f12c6 WatchSource:0}: Error finding container ef0f59abb1c1d15c9ef032f36c068cd2b1f91f73d54c6b7e197180ac6a5f12c6: Status 404 returned error can't find the container with id ef0f59abb1c1d15c9ef032f36c068cd2b1f91f73d54c6b7e197180ac6a5f12c6
Nov 03 01:18:11.752684 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.752649    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5pxx9"]
Nov 03 01:18:11.757259 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:18:11.757236    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63489afa_8dc5_48b4_a0b6_2c40bdd8ff79.slice/crio-505cac9e6dc55bc5f598dfb85f7b1048f7a448fdb3ece07be3bc44096df40b8c WatchSource:0}: Error finding container 505cac9e6dc55bc5f598dfb85f7b1048f7a448fdb3ece07be3bc44096df40b8c: Status 404 returned error can't find the container with id 505cac9e6dc55bc5f598dfb85f7b1048f7a448fdb3ece07be3bc44096df40b8c
Nov 03 01:18:11.758502 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.758465    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 01:18:11.758815 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.758787    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mmkcs\""
Nov 03 01:18:11.758906 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.758894    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 01:18:11.758961 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.758897    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 01:18:11.758961 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.758946    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sd2v\""
Nov 03 01:18:11.758961 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.758956    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fqzf6\""
Nov 03 01:18:11.759090 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.759016    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 01:18:11.803731 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.803710    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rvpjl"]
Nov 03 01:18:11.807833 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:18:11.807813    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda04dce2_a300_44f6_ab49_5fb77c6c8a19.slice/crio-53a809541a9c2f2c9dfb47ffbe524e464932d733927bfde7895be99685c0fe65 WatchSource:0}: Error finding container 53a809541a9c2f2c9dfb47ffbe524e464932d733927bfde7895be99685c0fe65: Status 404 returned error can't find the container with id 53a809541a9c2f2c9dfb47ffbe524e464932d733927bfde7895be99685c0fe65
Nov 03 01:18:11.897166 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.897134    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rvpjl" event={"ID":"da04dce2-a300-44f6-ab49-5fb77c6c8a19","Type":"ContainerStarted","Data":"53a809541a9c2f2c9dfb47ffbe524e464932d733927bfde7895be99685c0fe65"}
Nov 03 01:18:11.898543 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.898521    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5pxx9" event={"ID":"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79","Type":"ContainerStarted","Data":"8096ae6448c152b06ba4d8658ab3fb97062c94d40d4ecf6e1df2d5236e5f582d"}
Nov 03 01:18:11.898606 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.898571    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5pxx9" event={"ID":"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79","Type":"ContainerStarted","Data":"505cac9e6dc55bc5f598dfb85f7b1048f7a448fdb3ece07be3bc44096df40b8c"}
Nov 03 01:18:11.899454 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:11.899436    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bds75" event={"ID":"571bdc94-d1ed-4ede-87fd-65af23ce24e9","Type":"ContainerStarted","Data":"ef0f59abb1c1d15c9ef032f36c068cd2b1f91f73d54c6b7e197180ac6a5f12c6"}
Nov 03 01:18:12.392405 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.392375    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-65599784d8-h59hz"]
Nov 03 01:18:12.395315 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.395298    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.407675 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.407656    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.407767 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.407681    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.407767 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.407708    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.407767 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.407755    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.407863 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.407780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6wp\" (UniqueName: \"kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.407895 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.407886    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.407948 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.407935    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-tmp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.450162 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.450136    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 01:18:12.450162 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.450141    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 01:18:12.450452 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.450418    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 01:18:12.450452 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.450455    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 01:18:12.450660 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.450471    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fc5zz\""
Nov 03 01:18:12.450660 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.450460    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 01:18:12.450660 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.450496    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 01:18:12.450660 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.450460    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 01:18:12.451520 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.451500    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65599784d8-h59hz"]
Nov 03 01:18:12.509075 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.509054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-tmp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.509217 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.509110    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.509217 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.509134    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.509322 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.509257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.509322 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.509306    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.509449 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.509331    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6wp\" (UniqueName: \"kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.509449 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.509393    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.509774 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.509753    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-tmp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.510078 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.510054    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.510153 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.510079    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.510505 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.510484    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.511601 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.511583    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.511779 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.511760    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.571104 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.571069    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6wp\" (UniqueName: \"kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:12.706319 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:12.706286    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:13.078101 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:13.078025    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65599784d8-h59hz"]
Nov 03 01:18:13.904936 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:13.904899    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65599784d8-h59hz" event={"ID":"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4","Type":"ContainerStarted","Data":"378fbd1c82140a4a08b20ef9e4bae339b4e720a2852ece2091adb3c0b7f2a9f7"}
Nov 03 01:18:13.907089 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:13.907055    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5pxx9" event={"ID":"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79","Type":"ContainerStarted","Data":"7a847fd80562e3914d79bcaf904eab4757d7c1bd083122af21135c8373f95da3"}
Nov 03 01:18:14.913552 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:14.913451    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bds75" event={"ID":"571bdc94-d1ed-4ede-87fd-65af23ce24e9","Type":"ContainerStarted","Data":"4cc03443f83b6e4df4f4fbc39fa2ef676efc3c97da79c48e12c4b6a08be34194"}
Nov 03 01:18:14.913552 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:14.913514    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bds75" event={"ID":"571bdc94-d1ed-4ede-87fd-65af23ce24e9","Type":"ContainerStarted","Data":"f972666909483e789dc013e266b4817093c327ed44f53011e235106d56c14e89"}
Nov 03 01:18:14.914130 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:14.914103    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:15.032989 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.032939    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-bds75" podStartSLOduration=2.138720672 podStartE2EDuration="5.032920074s" totalImagesPullingTime="2.894199402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:15.032120147 +0000 UTC m=+51.889108743" watchObservedRunningTime="2026-11-03 01:18:15.032920074 +0000 UTC m=+51.889908664"
Nov 03 01:18:15.519015 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.518979    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m24b6"]
Nov 03 01:18:15.522497 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.522472    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.531931 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.531912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86vpg\" (UniqueName: \"kubernetes.io/projected/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-kube-api-access-86vpg\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.532050 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.531950    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-root\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.532050 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.531974    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-textfile\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.532050 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.531990    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-metrics-client-ca\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.532050 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.532006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-accelerators-collector-config\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.532050 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.532024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-sys\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.532050 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.532043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-wtmp\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.532367 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.532058    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.532367 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.532090    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-tls\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.576622 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.576544    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w7g4c\""
Nov 03 01:18:15.576622 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.576598    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 01:18:15.576622 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.576602    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 01:18:15.632967 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.632930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-tls\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.633115 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.633000    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86vpg\" (UniqueName: \"kubernetes.io/projected/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-kube-api-access-86vpg\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.633115 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.633039    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-root\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.633115 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.633053    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-textfile\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.633115 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.633070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-metrics-client-ca\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.633115 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.633088    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-accelerators-collector-config\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.633331 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.633118    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-sys\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.633331 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.633140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-wtmp\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.633331 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.633154    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.633805 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.633784    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-textfile\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.634491 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.634268    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-sys\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.634491 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.634423    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-root\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.634491 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.634453    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-wtmp\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.636326 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.636302    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.649709 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.649682    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-tls\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.673031 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.673004    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 01:18:15.673189 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.673173    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 01:18:15.673256 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.673213    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 01:18:15.673296 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.673268    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 01:18:15.674973 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.674948    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-metrics-client-ca\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.675074 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.675013    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-accelerators-collector-config\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.695905 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.695875    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86vpg\" (UniqueName: \"kubernetes.io/projected/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-kube-api-access-86vpg\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.842852 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.842771    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:15.856293 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:18:15.856250    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d8a03d_d25a_448a_955f_9b7ea3f4e07b.slice/crio-beac3c04fe0d07c4c840497823bbf4f74df7dcc3467bf87f339a69b70bb73ddf WatchSource:0}: Error finding container beac3c04fe0d07c4c840497823bbf4f74df7dcc3467bf87f339a69b70bb73ddf: Status 404 returned error can't find the container with id beac3c04fe0d07c4c840497823bbf4f74df7dcc3467bf87f339a69b70bb73ddf
Nov 03 01:18:15.916915 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:15.916871    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m24b6" event={"ID":"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b","Type":"ContainerStarted","Data":"beac3c04fe0d07c4c840497823bbf4f74df7dcc3467bf87f339a69b70bb73ddf"}
Nov 03 01:18:16.920680 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:16.920541    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rvpjl" event={"ID":"da04dce2-a300-44f6-ab49-5fb77c6c8a19","Type":"ContainerStarted","Data":"b6ec8f791a591132cf24d444d63a3e4a9a45ad42d62a31f7fda40ec803358f91"}
Nov 03 01:18:16.922904 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:16.922878    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5pxx9" event={"ID":"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79","Type":"ContainerStarted","Data":"fd47c161e515c0e6fc530c44f386b540bc47a35a543e3c7a4a260a29b7e92a56"}
Nov 03 01:18:17.037969 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.037922    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rvpjl" podStartSLOduration=1.9110726009999999 podStartE2EDuration="6.037909106s" totalImagesPullingTime="4.126836505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:17.037609536 +0000 UTC m=+53.894598132" watchObservedRunningTime="2026-11-03 01:18:17.037909106 +0000 UTC m=+53.894897702"
Nov 03 01:18:17.156292 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.156230    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5pxx9" podStartSLOduration=3.081002592 podStartE2EDuration="7.156212368s" totalImagesPullingTime="4.075209776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:18:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:17.156055334 +0000 UTC m=+54.013043923" watchObservedRunningTime="2026-11-03 01:18:17.156212368 +0000 UTC m=+54.013200962"
Nov 03 01:18:17.881788 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.881762    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 01:18:17.885055 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.885039    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.927620 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.927541    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65599784d8-h59hz" event={"ID":"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4","Type":"ContainerStarted","Data":"a3ac9b969ec40464f79a0fdfb528b0f1a1a0ee24a5109cd75a22aeb83cc3a7f6"}
Nov 03 01:18:17.929556 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.929526    2575 generic.go:356] "Generic (PLEG): container finished" podID="b7d8a03d-d25a-448a-955f-9b7ea3f4e07b" containerID="e52d33eba4e2367878a39bf15e5f9650f46ecd5413210a0ba324eb2c3639e168" exitCode=0
Nov 03 01:18:17.929684 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.929609    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m24b6" event={"ID":"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b","Type":"ContainerDied","Data":"e52d33eba4e2367878a39bf15e5f9650f46ecd5413210a0ba324eb2c3639e168"}
Nov 03 01:18:17.933025 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.933002    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 01:18:17.938049 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.938028    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 01:18:17.938142 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.938068    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 01:18:17.950635 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.950611    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-web-config\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.950729 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.950672    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.950729 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.950702    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-config-out\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.950951 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.950929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.951840 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.951166    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.951840 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.951410    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.951840 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.951494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.951840 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.951571    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.951840 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.951602    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.951840 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.951668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.951840 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.951718    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.951840 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.951767    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.951840 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.951786    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfvkd\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-kube-api-access-wfvkd\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:17.982119 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.982090    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 01:18:17.982254 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.982090    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 01:18:17.982254 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.982153    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 01:18:17.982254 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.982167    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 01:18:17.982254 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:17.982153    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tnwmc\""
Nov 03 01:18:18.039427 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.039400    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 01:18:18.039427 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.039422    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 01:18:18.039929 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.039909    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 01:18:18.052437 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.052417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.052523 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.052456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.052523 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.052481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.052523 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.052499    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfvkd\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-kube-api-access-wfvkd\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.052707 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.052522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-web-config\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.052759 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.052725    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.052809 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.052766    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-config-out\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.052862 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.052826    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.052913 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.052878    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.052967 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.052915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.053020 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.052982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.053073 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.053029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.053073 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.053063    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.053887 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.053855    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.055832 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.055336    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-config-out\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.055832 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.055458    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.055832 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.055525    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-web-config\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.055832 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.055576    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.055832 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.055799    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.056113 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.055930    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.056113 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.055971    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.056340 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.056313    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.056706 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.056686    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.057840 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.057819    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.057937 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.057885    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.113656 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.113623    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfvkd\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-kube-api-access-wfvkd\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.204327 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.204225    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-65599784d8-h59hz" podStartSLOduration=2.425806539 podStartE2EDuration="6.204212109s" totalImagesPullingTime="3.77840557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:18.203607874 +0000 UTC m=+55.060596470" watchObservedRunningTime="2026-11-03 01:18:18.204212109 +0000 UTC m=+55.061200703"
Nov 03 01:18:18.209908 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.209885    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:18.663727 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.663699    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 01:18:18.667615 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:18:18.667591    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda804a1ea_8545_4932_a98b_fba23d40688b.slice/crio-96657bb1ef573dd77f20f4de64ba4b4c5112a364f0daf3424978f71b7ff36c61 WatchSource:0}: Error finding container 96657bb1ef573dd77f20f4de64ba4b4c5112a364f0daf3424978f71b7ff36c61: Status 404 returned error can't find the container with id 96657bb1ef573dd77f20f4de64ba4b4c5112a364f0daf3424978f71b7ff36c61
Nov 03 01:18:18.933899 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.933821    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerStarted","Data":"96657bb1ef573dd77f20f4de64ba4b4c5112a364f0daf3424978f71b7ff36c61"}
Nov 03 01:18:18.936023 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.935994    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m24b6" event={"ID":"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b","Type":"ContainerStarted","Data":"8e92be73d5b1f94f062274f4f1f0e2f38f940e193d75361277723a52ffb6a811"}
Nov 03 01:18:18.936142 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:18.936044    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m24b6" event={"ID":"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b","Type":"ContainerStarted","Data":"8b185a60584f8bcc438a8ad670e044c7973b1529ce11d454bd4ae50e461fdc01"}
Nov 03 01:18:19.064913 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:19.064018    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m24b6" podStartSLOduration=2.191661028 podStartE2EDuration="4.063993574s" totalImagesPullingTime="1.807234613s" totalInitContainerRuntime="65.097933ms" isStatefulPod=true podCreationTimestamp="2026-11-03 01:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:19.061864043 +0000 UTC m=+55.918852639" watchObservedRunningTime="2026-11-03 01:18:19.063993574 +0000 UTC m=+55.920982172"
Nov 03 01:18:20.569333 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.569273    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-67f9494f54-92n2g"]
Nov 03 01:18:20.572294 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.572276    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.627623 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.627602    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 01:18:20.628993 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.628972    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67f9494f54-92n2g"]
Nov 03 01:18:20.670893 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.670871    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-secret-metrics-server-client-certs\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.670990 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.670905    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df19de7c-3e77-48ce-b5e4-b4b56ab26017-metrics-server-audit-profiles\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.670990 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.670928    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df19de7c-3e77-48ce-b5e4-b4b56ab26017-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.670990 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.670974    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df19de7c-3e77-48ce-b5e4-b4b56ab26017-audit-log\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.671103 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.671082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-client-ca-bundle\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.671144 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.671113    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-secret-metrics-server-tls\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.671195 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.671151    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfhx7\" (UniqueName: \"kubernetes.io/projected/df19de7c-3e77-48ce-b5e4-b4b56ab26017-kube-api-access-pfhx7\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.678199 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.678180    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 01:18:20.678287 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.678200    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 01:18:20.699577 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.699559    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-94q6h1no3itrh\""
Nov 03 01:18:20.699662 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.699622    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-whk5z\""
Nov 03 01:18:20.699662 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.699639    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 01:18:20.771946 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.771915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-client-ca-bundle\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.771946 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.771948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-secret-metrics-server-tls\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.772217 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.772198    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfhx7\" (UniqueName: \"kubernetes.io/projected/df19de7c-3e77-48ce-b5e4-b4b56ab26017-kube-api-access-pfhx7\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.772258 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.772244    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-secret-metrics-server-client-certs\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.772289 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.772281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df19de7c-3e77-48ce-b5e4-b4b56ab26017-metrics-server-audit-profiles\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.772321 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.772309    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df19de7c-3e77-48ce-b5e4-b4b56ab26017-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.772386 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.772335    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df19de7c-3e77-48ce-b5e4-b4b56ab26017-audit-log\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.772778 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.772755    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df19de7c-3e77-48ce-b5e4-b4b56ab26017-audit-log\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.773161 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.773138    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df19de7c-3e77-48ce-b5e4-b4b56ab26017-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.773456 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.773436    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df19de7c-3e77-48ce-b5e4-b4b56ab26017-metrics-server-audit-profiles\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.774527 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.774508    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-secret-metrics-server-tls\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.774598 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.774525    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-client-ca-bundle\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.774661 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.774647    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-secret-metrics-server-client-certs\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.835397 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.835332    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfhx7\" (UniqueName: \"kubernetes.io/projected/df19de7c-3e77-48ce-b5e4-b4b56ab26017-kube-api-access-pfhx7\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.918521 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.918484    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:20.943120 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.943093    2575 generic.go:356] "Generic (PLEG): container finished" podID="a804a1ea-8545-4932-a98b-fba23d40688b" containerID="5d9c4f61324f1587b1cb3764bbedf414f734f3e32409c48ab3b04449dfe8a22c" exitCode=0
Nov 03 01:18:20.943230 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:20.943147    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerDied","Data":"5d9c4f61324f1587b1cb3764bbedf414f734f3e32409c48ab3b04449dfe8a22c"}
Nov 03 01:18:21.340299 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:21.340263    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67f9494f54-92n2g"]
Nov 03 01:18:21.344851 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:18:21.344826    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf19de7c_3e77_48ce_b5e4_b4b56ab26017.slice/crio-9c03a8882de743114118d57225ecbfd9b55b207162f7326b89957a2928e3a925 WatchSource:0}: Error finding container 9c03a8882de743114118d57225ecbfd9b55b207162f7326b89957a2928e3a925: Status 404 returned error can't find the container with id 9c03a8882de743114118d57225ecbfd9b55b207162f7326b89957a2928e3a925
Nov 03 01:18:21.947364 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:21.947313    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67f9494f54-92n2g" event={"ID":"df19de7c-3e77-48ce-b5e4-b4b56ab26017","Type":"ContainerStarted","Data":"9c03a8882de743114118d57225ecbfd9b55b207162f7326b89957a2928e3a925"}
Nov 03 01:18:22.707510 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:22.707472    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:22.707670 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:22.707561    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:22.713030 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:22.713007    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:22.953434 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:22.953412    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:23.962326 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:23.962294    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67f9494f54-92n2g" event={"ID":"df19de7c-3e77-48ce-b5e4-b4b56ab26017","Type":"ContainerStarted","Data":"651def415a2cf9cb8c54c7237778c945fb0e900840b4f669ba8b1f3eed06d090"}
Nov 03 01:18:24.109077 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.109018    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-67f9494f54-92n2g" podStartSLOduration=1.588591745 podStartE2EDuration="4.109000185s" totalImagesPullingTime="2.52040844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:24.108326245 +0000 UTC m=+60.965314843" watchObservedRunningTime="2026-11-03 01:18:24.109000185 +0000 UTC m=+60.965988783"
Nov 03 01:18:24.662467 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.662433    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 01:18:24.666191 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.666157    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.706587 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.706551    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.706587 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.706590    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.706774 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.706610    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.706774 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.706659    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.706774 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.706692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.706774 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.706765    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.706906 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.706785    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.706906 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.706802    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config-out\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.706906 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.706817    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.706906 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.706835    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.706906 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.706854    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84tzx\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-kube-api-access-84tzx\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.707054 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.706912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.707054 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.706984    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.707054 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.707009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.707054 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.707032    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.707164 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.707109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.707164 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.707133    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.707223 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.707181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.707223 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.707200    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.707281 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.707238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-web-config\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.722742 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.722706    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 01:18:24.722742 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.722721    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 01:18:24.722977 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.722809    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 01:18:24.722977 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.722819    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 01:18:24.722977 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.722867    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 01:18:24.722977 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.722839    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tp979\""
Nov 03 01:18:24.722977 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.722902    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fsir5cbmlb4a0\""
Nov 03 01:18:24.773555 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.773491    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 01:18:24.773555 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.773491    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 01:18:24.775048 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.775021    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 01:18:24.775287 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.775272    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 01:18:24.808114 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808089    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808231 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808127    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808231 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808154    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84tzx\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-kube-api-access-84tzx\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808231 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808182    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808231 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808221    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808505 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808244    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808505 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808505 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808505 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808435    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808505 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808471    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808505 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808500    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808801 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808535    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-web-config\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808801 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808626    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808801 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808656    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808801 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808683    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808801 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808801 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808734    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808801 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808769    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.808801 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808793    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.809213 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config-out\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.809213 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808850    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.809213 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.808919    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.809213 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.809090    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.809995 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.809679    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.810818 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.810558    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.810818 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.810709    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.811584 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.811530    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config-out\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.813015 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.811904    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.813015 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.811940    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.813015 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.812489    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.813176 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.813016    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.813296 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.813277    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.814084 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.814064    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.825107 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.825052    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 01:18:24.831953 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.831923    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.873392 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.873366    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:24.874807 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.874720    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 01:18:24.875408 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.875334    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 01:18:24.875408 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.875376    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 01:18:24.875408 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.875374    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 01:18:24.875408 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.875380    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 01:18:24.877903 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.877882    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84tzx\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-kube-api-access-84tzx\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.879427 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.879318    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.880802 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.880777    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.881415 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.881393    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.881550 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.881531    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-web-config\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.881748 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.881729    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:24.969429 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.969380    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerStarted","Data":"b99934dab8545eb6799b349e221c084fd820bc7bad4588312519565be1f66955"}
Nov 03 01:18:24.969797 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.969508    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerStarted","Data":"b2a51c8299dbebd40251d0f5c0887804f83031a0f2cd548b7fbcee729d8bc0d1"}
Nov 03 01:18:24.969797 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.969541    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerStarted","Data":"e62cfc33997f80e5b94efa81757d525e6d8659ef2fe2bf243892de12a3273b24"}
Nov 03 01:18:24.995517 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:24.995487    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:25.436764 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:25.436734    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 01:18:25.441652 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:18:25.441626    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8fe558_c1cf_4f0c_96d0_e930abeb857b.slice/crio-b423824438a1755dab77a92ade9560b77f48bdca244bbf5fd64c0af4154268e8 WatchSource:0}: Error finding container b423824438a1755dab77a92ade9560b77f48bdca244bbf5fd64c0af4154268e8: Status 404 returned error can't find the container with id b423824438a1755dab77a92ade9560b77f48bdca244bbf5fd64c0af4154268e8
Nov 03 01:18:25.925662 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:25.925634    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:25.974420 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:25.974388    2575 generic.go:356] "Generic (PLEG): container finished" podID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerID="b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4" exitCode=0
Nov 03 01:18:25.974794 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:25.974472    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerDied","Data":"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"}
Nov 03 01:18:25.974794 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:25.974544    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerStarted","Data":"b423824438a1755dab77a92ade9560b77f48bdca244bbf5fd64c0af4154268e8"}
Nov 03 01:18:25.977818 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:25.977799    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerStarted","Data":"547b58a4fd6c9f80ff3eed4793cab2dccb486153ecaa80af60d02579587c6ed5"}
Nov 03 01:18:25.977934 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:25.977878    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerStarted","Data":"5100f6594561a64c0236ecccf5ec997ddffd41a353045cbe99752753db474db4"}
Nov 03 01:18:26.984720 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:26.984688    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerStarted","Data":"ad308c4395e03f4f9176d4a62afa47ac4591f5acef95b857f19c93f7832cec87"}
Nov 03 01:18:27.165059 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:27.164993    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.009298991 podStartE2EDuration="10.16497957s" totalImagesPullingTime="7.142180739s" totalInitContainerRuntime="13.49984ms" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:27.164298263 +0000 UTC m=+64.021286859" watchObservedRunningTime="2026-11-03 01:18:27.16497957 +0000 UTC m=+64.021968165"
Nov 03 01:18:29.451872 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.451826    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:29.504158 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.504127    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 01:18:29.514564 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.514535    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:29.552831 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.552802    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:29.604930 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.604908    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 01:18:29.615988 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.615965    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:29.653899 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.653871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp42j\" (UniqueName: \"kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j\") pod \"network-check-target-ttbvc\" (UID: \"5a8d8001-c1b0-4489-9e25-fb8b67048d7d\") " pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:29.706039 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.706017    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 01:18:29.766416 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.766397    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 01:18:29.775240 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.775172    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fqzf6\""
Nov 03 01:18:29.777316 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.777294    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp42j\" (UniqueName: \"kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j\") pod \"network-check-target-ttbvc\" (UID: \"5a8d8001-c1b0-4489-9e25-fb8b67048d7d\") " pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:29.782583 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.782565    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sd2v\""
Nov 03 01:18:29.783612 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.783589    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:29.790522 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.790501    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:29.998622 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.998554    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerStarted","Data":"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"}
Nov 03 01:18:29.998622 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:29.998620    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerStarted","Data":"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"}
Nov 03 01:18:30.070731 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:30.070707    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mmkcs\""
Nov 03 01:18:30.079859 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:30.079831    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:30.171801 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:30.171775    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b4htm"]
Nov 03 01:18:30.175658 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:18:30.175633    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5d501e_fa40_467b_a4fa_9fc52f9c45fa.slice/crio-3928afa6d9ebb017886812fdf9721e4bf2f3ffca76f5d478378f2b2a9c4ba709 WatchSource:0}: Error finding container 3928afa6d9ebb017886812fdf9721e4bf2f3ffca76f5d478378f2b2a9c4ba709: Status 404 returned error can't find the container with id 3928afa6d9ebb017886812fdf9721e4bf2f3ffca76f5d478378f2b2a9c4ba709
Nov 03 01:18:30.223104 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:30.223071    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vxv8x"]
Nov 03 01:18:30.226898 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:18:30.226872    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d824491_3dcc_4c32_8378_7c8459b34086.slice/crio-f38450dda7be5a25d62612d25a520ea805998c5bfc99b696fa5c244732de00f9 WatchSource:0}: Error finding container f38450dda7be5a25d62612d25a520ea805998c5bfc99b696fa5c244732de00f9: Status 404 returned error can't find the container with id f38450dda7be5a25d62612d25a520ea805998c5bfc99b696fa5c244732de00f9
Nov 03 01:18:30.449488 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:30.449456    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ttbvc"]
Nov 03 01:18:30.453124 ip-10-0-140-255 kubenswrapper[2575]: W1103 01:18:30.453098    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8d8001_c1b0_4489_9e25_fb8b67048d7d.slice/crio-8205c515c9a9f8f445dc87459de92b0bbc943cddc6bbdafcf0e5447bfcd49876 WatchSource:0}: Error finding container 8205c515c9a9f8f445dc87459de92b0bbc943cddc6bbdafcf0e5447bfcd49876: Status 404 returned error can't find the container with id 8205c515c9a9f8f445dc87459de92b0bbc943cddc6bbdafcf0e5447bfcd49876
Nov 03 01:18:31.003702 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:31.003667    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ttbvc" event={"ID":"5a8d8001-c1b0-4489-9e25-fb8b67048d7d","Type":"ContainerStarted","Data":"8205c515c9a9f8f445dc87459de92b0bbc943cddc6bbdafcf0e5447bfcd49876"}
Nov 03 01:18:31.005134 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:31.005095    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxv8x" event={"ID":"3d824491-3dcc-4c32-8378-7c8459b34086","Type":"ContainerStarted","Data":"f38450dda7be5a25d62612d25a520ea805998c5bfc99b696fa5c244732de00f9"}
Nov 03 01:18:31.006697 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:31.006422    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b4htm" event={"ID":"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa","Type":"ContainerStarted","Data":"3928afa6d9ebb017886812fdf9721e4bf2f3ffca76f5d478378f2b2a9c4ba709"}
Nov 03 01:18:33.016779 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:33.016730    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxv8x" event={"ID":"3d824491-3dcc-4c32-8378-7c8459b34086","Type":"ContainerStarted","Data":"3fa9fb16d3506d8d716ff4ebe1231d002915428241381cedd87b6eb827e3af20"}
Nov 03 01:18:33.017199 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:33.016810    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxv8x" event={"ID":"3d824491-3dcc-4c32-8378-7c8459b34086","Type":"ContainerStarted","Data":"09c3f928077a3b17c1b998f868a09a13a2bc3e5976a7e6ae56fd612c4c0464ad"}
Nov 03 01:18:33.133169 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:33.133120    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vxv8x" podStartSLOduration=65.917864927 podStartE2EDuration="1m8.133107644s" totalImagesPullingTime="2.215242717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:33.132831628 +0000 UTC m=+69.989820226" watchObservedRunningTime="2026-11-03 01:18:33.133107644 +0000 UTC m=+69.990096231"
Nov 03 01:18:34.026441 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:34.026326    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerStarted","Data":"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"}
Nov 03 01:18:34.026866 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:34.026467    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerStarted","Data":"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"}
Nov 03 01:18:34.026866 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:34.026499    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerStarted","Data":"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"}
Nov 03 01:18:34.026866 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:34.026526    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerStarted","Data":"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"}
Nov 03 01:18:34.134642 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:34.134252    2575 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65599784d8-h59hz"]
Nov 03 01:18:34.212973 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:34.212899    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7921455059999998 podStartE2EDuration="10.212878812s" totalImagesPullingTime="7.403560392s" totalInitContainerRuntime="17.172914ms" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:34.212160759 +0000 UTC m=+71.069149367" watchObservedRunningTime="2026-11-03 01:18:34.212878812 +0000 UTC m=+71.069867409"
Nov 03 01:18:34.996541 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:34.996507    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:35.031838 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:35.031739    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ttbvc" event={"ID":"5a8d8001-c1b0-4489-9e25-fb8b67048d7d","Type":"ContainerStarted","Data":"3321a0a09f9b1f0aebc26a9e7afe8c3a63e587621c1e7022f10b0bcc07f1359b"}
Nov 03 01:18:35.032200 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:35.032150    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:35.147494 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:35.147436    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ttbvc" podStartSLOduration=65.818878807 podStartE2EDuration="1m10.147420573s" totalImagesPullingTime="4.328541766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:35.147109858 +0000 UTC m=+72.004098454" watchObservedRunningTime="2026-11-03 01:18:35.147420573 +0000 UTC m=+72.004409147"
Nov 03 01:18:40.491760 ip-10-0-140-255 kubenswrapper[2575]: I1103 01:18:40.491619    2575 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 01:18:40.491784 ip-10-0-140-255 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 01:18:40.512757 ip-10-0-140-255 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 01:18:40.512939 ip-10-0-140-255 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 01:18:40.513119 ip-10-0-140-255 systemd[1]: kubelet.service: Consumed 4.196s CPU time.
Nov 03 01:18:40.528069 ip-10-0-140-255 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 01:18:40.602641 ip-10-0-140-255 kubenswrapper[5508]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 01:18:40.602641 ip-10-0-140-255 kubenswrapper[5508]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 01:18:40.602641 ip-10-0-140-255 kubenswrapper[5508]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 01:18:40.602641 ip-10-0-140-255 kubenswrapper[5508]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 01:18:40.605704 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605614    5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:18:40.605704 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605700    5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:18:40.605704 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605705    5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:18:40.605704 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605708    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:18:40.605704 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605711    5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605714    5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605719    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605722    5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605724    5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605727    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605731    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605733    5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605736    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605740    5508 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605743    5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605746    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605748    5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605751    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605754    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605756    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605760    5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605763    5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605766    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:18:40.605885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605768    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605771    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605774    5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605777    5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605779    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605782    5508 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605785    5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605787    5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605790    5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605793    5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605795    5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605798    5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605801    5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605803    5508 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605806    5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605809    5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605815    5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605818    5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605821    5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605823    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:18:40.606379 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605826    5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605828    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605831    5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605833    5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605838    5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605843    5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605846    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605848    5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605851    5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605854    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605857    5508 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605859    5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605862    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605864    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605868    5508 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605870    5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605873    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605876    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605878    5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:18:40.606883 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605881    5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605883    5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605886    5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605889    5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605891    5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605894    5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605897    5508 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605900    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605903    5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605905    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605908    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605910    5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605913    5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605915    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605918    5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605922    5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605925    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605928    5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605930    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605933    5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:18:40.607410 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605936    5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605939    5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605942    5508 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605945    5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605947    5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605950    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605954    5508 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605957    5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605959    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605962    5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605965    5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605967    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605970    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605972    5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605975    5508 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605977    5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605980    5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605982    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605985    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605989    5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:18:40.607892 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605991    5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605994    5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605997    5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.605999    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606761    5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606769    5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606772    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606775    5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606778    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606781    5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606783    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606786    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606789    5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606792    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606796    5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606798    5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606802    5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606805    5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606808    5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:18:40.608394 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606811    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606813    5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606816    5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606819    5508 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606821    5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606824    5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606826    5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606829    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606832    5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606834    5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606837    5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606840    5508 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606843    5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606846    5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606849    5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606852    5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606854    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606857    5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606860    5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606862    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:18:40.608894 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606865    5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606867    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606870    5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606872    5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606875    5508 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606878    5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606881    5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606883    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606886    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606889    5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606892    5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606897    5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606901    5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606904    5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606909    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606914    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606918    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606923    5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606927    5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:18:40.609404 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606932    5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606936    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606939    5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606942    5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606944    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606947    5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606950    5508 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606953    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606956    5508 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606958    5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606961    5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606964    5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606966    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606969    5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606971    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606974    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606977    5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606979    5508 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606982    5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606985    5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:18:40.609965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606987    5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606990    5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606993    5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606996    5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.606999    5508 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607001    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607005    5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607007    5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607010    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607013    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607015    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607018    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607021    5508 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607023    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607026    5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607028    5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607031    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607033    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607036    5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607038    5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:18:40.610543 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607041    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607043    5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607046    5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607048    5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607051    5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607054    5508 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607057    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607059    5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607065    5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607068    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607071    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607074    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607597    5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607603    5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607607    5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607610    5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607613    5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607616    5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607620    5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607622    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:18:40.611032 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607625    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607628    5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607631    5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607633    5508 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607636    5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607639    5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607641    5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607644    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607647    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607649    5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607652    5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607655    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607657    5508 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607660    5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607662    5508 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607665    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607668    5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607670    5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607673    5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607675    5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:18:40.611566 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607678    5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607680    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607683    5508 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607685    5508 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607688    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607690    5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607693    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607695    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607698    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607701    5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607704    5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607707    5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607710    5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607713    5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607716    5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607719    5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607722    5508 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607724    5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607727    5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607730    5508 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:18:40.612049 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607732    5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607735    5508 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607737    5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607740    5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607743    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607747    5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607750    5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607754    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607758    5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607762    5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607765    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607767    5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607770    5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607772    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607775    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607777    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607780    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607782    5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607785    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:18:40.612542 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607787    5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607791    5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607793    5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607796    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607798    5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607801    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607804    5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607807    5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607810    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607812    5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607814    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607817    5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607820    5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607822    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607825    5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607827    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607830    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607832    5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607835    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:18:40.612995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607838    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607840    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607843    5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607845    5508 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607853    5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607855    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607858    5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607861    5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607863    5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607866    5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607869    5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607871    5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607874    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607876    5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607879    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607882    5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607885    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607887    5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607890    5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.607894    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:18:40.613482 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.607994    5508 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608014    5508 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608031    5508 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608037    5508 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608042    5508 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608045    5508 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608050    5508 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608054    5508 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608057    5508 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608060    5508 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608064    5508 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608067    5508 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608071    5508 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608074    5508 flags.go:64] FLAG: --cgroup-root=""
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608077    5508 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608080    5508 flags.go:64] FLAG: --client-ca-file=""
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608086    5508 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608089    5508 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608094    5508 flags.go:64] FLAG: --cluster-domain=""
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608098    5508 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608101    5508 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608105    5508 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608108    5508 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608112    5508 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 01:18:40.613962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608115    5508 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608118    5508 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608122    5508 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608127    5508 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608130    5508 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608133    5508 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608137    5508 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608140    5508 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608144    5508 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608148    5508 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608151    5508 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608154    5508 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608157    5508 flags.go:64] FLAG: --enable-server="true"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608160    5508 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608165    5508 flags.go:64] FLAG: --event-burst="100"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608168    5508 flags.go:64] FLAG: --event-qps="50"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608171    5508 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608174    5508 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608177    5508 flags.go:64] FLAG: --eviction-hard=""
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608181    5508 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608185    5508 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608188    5508 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608191    5508 flags.go:64] FLAG: --eviction-soft=""
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608194    5508 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608198    5508 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 01:18:40.614589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608201    5508 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608204    5508 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608207    5508 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608210    5508 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608213    5508 flags.go:64] FLAG: --feature-gates=""
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608217    5508 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608220    5508 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608224    5508 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608227    5508 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608230    5508 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608235    5508 flags.go:64] FLAG: --help="false"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608238    5508 flags.go:64] FLAG: --hostname-override="ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608242    5508 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608245    5508 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608248    5508 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608252    5508 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608255    5508 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608259    5508 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608262    5508 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608265    5508 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608268    5508 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608271    5508 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608274    5508 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608277    5508 flags.go:64] FLAG: --kube-reserved=""
Nov 03 01:18:40.615229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608280    5508 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608283    5508 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608286    5508 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608289    5508 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608292    5508 flags.go:64] FLAG: --lock-file=""
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608295    5508 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608298    5508 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608302    5508 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608308    5508 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608311    5508 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608315    5508 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608318    5508 flags.go:64] FLAG: --logging-format="text"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608321    5508 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608324    5508 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608327    5508 flags.go:64] FLAG: --manifest-url=""
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608330    5508 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608339    5508 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608342    5508 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608362    5508 flags.go:64] FLAG: --max-pods="110"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608367    5508 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608372    5508 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608378    5508 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608382    5508 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608385    5508 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608388    5508 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 01:18:40.615816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608392    5508 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608399    5508 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608402    5508 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608405    5508 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608409    5508 flags.go:64] FLAG: --pod-cidr=""
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608412    5508 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608415    5508 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608418    5508 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608421    5508 flags.go:64] FLAG: --port="10250"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608424    5508 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608427    5508 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-051d82144733e4e2c"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608430    5508 flags.go:64] FLAG: --qos-reserved=""
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608433    5508 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608437    5508 flags.go:64] FLAG: --register-node="true"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608441    5508 flags.go:64] FLAG: --register-with-taints=""
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608445    5508 flags.go:64] FLAG: --registry-burst="10"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608448    5508 flags.go:64] FLAG: --registry-qps="5"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608451    5508 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608454    5508 flags.go:64] FLAG: --reserved-memory=""
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608458    5508 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608461    5508 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608464    5508 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608467    5508 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608470    5508 flags.go:64] FLAG: --runonce="false"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608473    5508 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608478    5508 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608481    5508 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 01:18:40.616461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608484    5508 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608487    5508 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608491    5508 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608494    5508 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608498    5508 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608500    5508 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608503    5508 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608506    5508 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608509    5508 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608512    5508 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608515    5508 flags.go:64] FLAG: --system-cgroups=""
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608518    5508 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608523    5508 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608526    5508 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608529    5508 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608534    5508 flags.go:64] FLAG: --tls-min-version=""
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608537    5508 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608540    5508 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608543    5508 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608548    5508 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608551    5508 flags.go:64] FLAG: --v="2"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608555    5508 flags.go:64] FLAG: --version="false"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608560    5508 flags.go:64] FLAG: --vmodule=""
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608563    5508 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.608567    5508 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 01:18:40.617106 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608700    5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608704    5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608707    5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608710    5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608713    5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608719    5508 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608722    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608724    5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608727    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608730    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608733    5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608736    5508 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608739    5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608741    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608744    5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608747    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608749    5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608752    5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608755    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608757    5508 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:18:40.618067 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608760    5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608763    5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608765    5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608768    5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608770    5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608773    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608777    5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608780    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608782    5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608785    5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608787    5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608790    5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608792    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608795    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608797    5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608800    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608803    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608808    5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608810    5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608813    5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:18:40.618786 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608815    5508 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608818    5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608821    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608824    5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608826    5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608829    5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608831    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608834    5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608837    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608839    5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608842    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608844    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608847    5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608849    5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608852    5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608854    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608857    5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608860    5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608863    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608866    5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:18:40.619322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608869    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608871    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608874    5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608876    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608879    5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608883    5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608886    5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608889    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608892    5508 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608896    5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608899    5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608902    5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608905    5508 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608907    5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608910    5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608913    5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608917    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608919    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608922    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:18:40.619967 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608925    5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608927    5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608930    5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608932    5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608935    5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608938    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608940    5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608943    5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608945    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608948    5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608950    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608955    5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608957    5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608960    5508 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608963    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608965    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608968    5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608970    5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608973    5508 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:18:40.620565 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608975    5508 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608978    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608981    5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608985    5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608988    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608991    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608995    5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.608999    5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.609025    5508 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609062    5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609067    5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609070    5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609073    5508 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609076    5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609079    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:18:40.621102 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609082    5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609085    5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609087    5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609090    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609092    5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609095    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609098    5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609101    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609103    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609106    5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609110    5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609113    5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609115    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609118    5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609121    5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609123    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609126    5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609128    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609131    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609134    5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:18:40.621534 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609137    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609140    5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609144    5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609147    5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609151    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609154    5508 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609156    5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609159    5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609162    5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609166    5508 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609169    5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609171    5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609174    5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609177    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609179    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609182    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609184    5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609187    5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609190    5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:18:40.622048 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609192    5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609195    5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609197    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609201    5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609203    5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609206    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609209    5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609212    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609215    5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609217    5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609220    5508 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609222    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609232    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609236    5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609239    5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609242    5508 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609244    5508 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609247    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609250    5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609252    5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:18:40.622618 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609256    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609258    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609263    5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609271    5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609275    5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609279    5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609282    5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609284    5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609287    5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609290    5508 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609293    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609296    5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609298    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609301    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609303    5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609307    5508 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609310    5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609312    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609315    5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:18:40.623180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609318    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609320    5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609324    5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609326    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609329    5508 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609332    5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609335    5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609338    5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609341    5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609343    5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609362    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609366    5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609370    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609374    5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609378    5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609382    5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609386    5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609389    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609393    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609395    5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:18:40.623688 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609398    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.609400    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.613213    5508 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.613236    5508 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.613255    5508 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.613260    5508 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613317    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613323    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613326    5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613329    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613332    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613335    5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613338    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613340    5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613343    5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613358    5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613361    5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613364    5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613366    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:18:40.624171 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613369    5508 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613372    5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613375    5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613378    5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613380    5508 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613383    5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613385    5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613389    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613391    5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613395    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613398    5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613400    5508 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613403    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613406    5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613409    5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613411    5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613420    5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613423    5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613428    5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:18:40.624648 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613432    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613436    5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613439    5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613442    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613444    5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613447    5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613450    5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613452    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613455    5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613457    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613460    5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613463    5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613465    5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613468    5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613471    5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613473    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613476    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613479    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613481    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613484    5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:18:40.625103 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613487    5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613490    5508 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613494    5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613496    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613499    5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613502    5508 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613504    5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613507    5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613509    5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613512    5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613520    5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613523    5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613526    5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613530    5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613534    5508 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613537    5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613539    5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613542    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613544    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613547    5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:18:40.625692 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613549    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613552    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613554    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613557    5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613560    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613566    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613569    5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613571    5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613574    5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613577    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613580    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613582    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613585    5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613588    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613591    5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613593    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613596    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613599    5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613601    5508 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:18:40.626176 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613604    5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613607    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613609    5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613612    5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613620    5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613623    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613626    5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613629    5508 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613631    5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613634    5508 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613636    5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613639    5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613641    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613644    5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613647    5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.613674    5508 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 01:18:40.626656 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613852    5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613857    5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613860    5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613863    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613866    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613869    5508 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613871    5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613874    5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613877    5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613879    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613883    5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613887    5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613890    5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613893    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613895    5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613898    5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613901    5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613903    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613906    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:18:40.627086 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613908    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613911    5508 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613919    5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613922    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613925    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613927    5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613930    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613932    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613935    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613937    5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613939    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613942    5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613945    5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613948    5508 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613950    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613952    5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613955    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613957    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613960    5508 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613962    5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:18:40.627614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613965    5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613968    5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613970    5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613973    5508 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613976    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613979    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613981    5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613984    5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613986    5508 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613989    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613992    5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613994    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613996    5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.613999    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614001    5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614011    5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614014    5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614016    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614018    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614021    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:18:40.628084 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614024    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614027    5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614029    5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614032    5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614034    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614037    5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614039    5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614042    5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614044    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614047    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614050    5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614052    5508 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614055    5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614057    5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614060    5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614062    5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614065    5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614073    5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614076    5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614078    5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:18:40.628580 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614081    5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614083    5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614086    5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614088    5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614091    5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614093    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614096    5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614098    5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614107    5508 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614110    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614112    5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614115    5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614117    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614120    5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614122    5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614125    5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614128    5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614130    5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614132    5508 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:18:40.629054 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614135    5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614137    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614140    5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614144    5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614148    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614151    5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614154    5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614157    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.614181    5508 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614218    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614222    5508 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614225    5508 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614228    5508 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614230    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614233    5508 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 01:18:40.629513 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614235    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614238    5508 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614240    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614243    5508 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614245    5508 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614248    5508 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614250    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614258    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614261    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614263    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614266    5508 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614269    5508 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614271    5508 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614274    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614276    5508 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614279    5508 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614282    5508 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614285    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614287    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 01:18:40.629873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614290    5508 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614292    5508 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614295    5508 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614297    5508 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614300    5508 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614302    5508 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614305    5508 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614307    5508 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614310    5508 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614312    5508 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614314    5508 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614317    5508 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614319    5508 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614322    5508 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614324    5508 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614327    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614329    5508 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614332    5508 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614334    5508 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614337    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 01:18:40.630322 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614339    5508 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614359    5508 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614362    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614365    5508 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614367    5508 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614370    5508 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614372    5508 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614377    5508 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614379    5508 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614382    5508 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614384    5508 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614387    5508 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614391    5508 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614395    5508 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614397    5508 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614400    5508 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614403    5508 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614405    5508 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614408    5508 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 01:18:40.630826 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614410    5508 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614413    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614415    5508 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614418    5508 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614421    5508 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614423    5508 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614426    5508 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614428    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614432    5508 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614435    5508 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614439    5508 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614442    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614445    5508 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614448    5508 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614450    5508 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614459    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614461    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614464    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614467    5508 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 01:18:40.631367 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614469    5508 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614473    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614476    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614478    5508 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614481    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614483    5508 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614486    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614488    5508 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614491    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614494    5508 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614497    5508 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614499    5508 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614502    5508 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614504    5508 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614507    5508 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614509    5508 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614512    5508 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614514    5508 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614517    5508 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 01:18:40.631825 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614519    5508 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614522    5508 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614524    5508 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:18:40.614526    5508 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.614593    5508 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.615585    5508 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.615635    5508 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.615830    5508 server.go:1014] "Starting client certificate rotation"
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.616005    5508 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.616045    5508 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 01:12:24 +0000 UTC" deadline="2028-07-22 01:47:49.242060394 +0000 UTC"
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.616106    5508 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15048h29m8.625957141s"
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.616371    5508 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.619161    5508 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.620325    5508 log.go:25] "Validated CRI v1 runtime API"
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.622084    5508 log.go:25] "Validated CRI v1 image API"
Nov 03 01:18:40.632313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.622806    5508 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 01:18:40.632803 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.624831    5508 fs.go:135] Filesystem UUIDs: map[4be91eae-1d97-4253-8730-451bdbf64a85:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7c3803db-a80b-4872-834d-74c902bb660c:/dev/nvme0n1p3]
Nov 03 01:18:40.632870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.624846    5508 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/378fbd1c82140a4a08b20ef9e4bae339b4e720a2852ece2091adb3c0b7f2a9f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/378fbd1c82140a4a08b20ef9e4bae339b4e720a2852ece2091adb3c0b7f2a9f7/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3928afa6d9ebb017886812fdf9721e4bf2f3ffca76f5d478378f2b2a9c4ba709/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3928afa6d9ebb017886812fdf9721e4bf2f3ffca76f5d478378f2b2a9c4ba709/userdata/shm major:0 minor:369 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3dc67cea4c5f3357064a3154d04a1e9e6df73c9d41b25a0e72c2d4faa0b9ef7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3dc67cea4c5f3357064a3154d04a1e9e6df73c9d41b25a0e72c2d4faa0b9ef7e/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43a30ba52f58f20dddbae3241d362c4a7e8a1e3aeb090b9cbd6fe066a476c494/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43a30ba52f58f20dddbae3241d362c4a7e8a1e3aeb090b9cbd6fe066a476c494/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/505cac9e6dc55bc5f598dfb85f7b1048f7a448fdb3ece07be3bc44096df40b8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/505cac9e6dc55bc5f598dfb85f7b1048f7a448fdb3ece07be3bc44096df40b8c/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53a809541a9c2f2c9dfb47ffbe524e464932d733927bfde7895be99685c0fe65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53a809541a9c2f2c9dfb47ffbe524e464932d733927bfde7895be99685c0fe65/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78fe8905288581090afb36532f05784b20ebca0c055ed6c8e1074a9d6ff519be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78fe8905288581090afb36532f05784b20ebca0c055ed6c8e1074a9d6ff519be/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/802890bbb04d593748e26313e577cb888c7d3021afd8886c63fb74d4a641ff18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/802890bbb04d593748e26313e577cb888c7d3021afd8886c63fb74d4a641ff18/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8205c515c9a9f8f445dc87459de92b0bbc943cddc6bbdafcf0e5447bfcd49876/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8205c515c9a9f8f445dc87459de92b0bbc943cddc6bbdafcf0e5447bfcd49876/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85ece4a25e76922c64587102669c48816d4b7c3f1a4dc7929269a6296bd3e586/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85ece4a25e76922c64587102669c48816d4b7c3f1a4dc7929269a6296bd3e586/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d47f5ae5cf9b860ed0482d0634fb887e4bbaff1cd439d70bb2b1ccd7cb3b0ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d47f5ae5cf9b860ed0482d0634fb887e4bbaff1cd439d70bb2b1ccd7cb3b0ff/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96657bb1ef573dd77f20f4de64ba4b4c5112a364f0daf3424978f71b7ff36c61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96657bb1ef573dd77f20f4de64ba4b4c5112a364f0daf3424978f71b7ff36c61/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c03a8882de743114118d57225ecbfd9b55b207162f7326b89957a2928e3a925/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c03a8882de743114118d57225ecbfd9b55b207162f7326b89957a2928e3a925/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af02d4102d3d30d1c8d8279100a03aa5439e262ea74992ca242c0f330dcb4f3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af02d4102d3d30d1c8d8279100a03aa5439e262ea74992ca242c0f330dcb4f3d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b423824438a1755dab77a92ade9560b77f48bdca244bbf5fd64c0af4154268e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b423824438a1755dab77a92ade9560b77f48bdca244bbf5fd64c0af4154268e8/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/beac3c04fe0d07c4c840497823bbf4f74df7dcc3467bf87f339a69b70bb73ddf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/beac3c04fe0d07c4c840497823bbf4f74df7dcc3467bf87f339a69b70bb73ddf/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ccd809ce8e6cd82cf0867fe0496a4e5e03174f8099bdbf41a43f338214d4cf7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ccd809ce8e6cd82cf0867fe0496a4e5e03174f8099bdbf41a43f338214d4cf7e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef0f59abb1c1d15c9ef032f36c068cd2b1f91f73d54c6b7e197180ac6a5f12c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef0f59abb1c1d15c9ef032f36c068cd2b1f91f73d54c6b7e197180ac6a5f12c6/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f38450dda7be5a25d62612d25a520ea805998c5bfc99b696fa5c244732de00f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f38450dda7be5a25d62612d25a520ea805998c5bfc99b696fa5c244732de00f9/userdata/shm major:0 minor:371 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5fa2ad137caf5bd6425053cc5496d81379b1fb1e5e7ad5046731d516d42c366/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5fa2ad137caf5bd6425053cc5496d81379b1fb1e5e7ad5046731d516d42c366/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe46741258363e7b02e84970875d6fe7881a3c9c79580df4429238fde64e8890/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe46741258363e7b02e84970875d6fe7881a3c9c79580df4429238fde64e8890/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/125c3968-dca4-4163-b473-b5cc7ac5a23f/volumes/kubernetes.io~projected/kube-api-access-r2rxv:{mountpoint:/var/lib/kubelet/pods/125c3968-dca4-4163-b473-b5cc7ac5a23f/volumes/kubernetes.io~projected/kube-api-access-r2rxv major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4/volumes/kubernetes.io~projected/kube-api-access-px6wp:{mountpoint:/var/lib/kubelet/pods/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4/volumes/kubernetes.io~projected/kube-api-access-px6wp major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c57dcf4-5aa3-4992-b357-47b388f3db90/volumes/kubernetes.io~projected/kube-api-access-jqwfz:{mountpoint:/var/lib/kubelet/pods/2c57dcf4-5aa3-4992-b357-47b388f3db90/volumes/kubernetes.io~projected/kube-api-access-jqwfz major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f7f681b-d410-40e9-a4d0-487230d8dece/volumes/kubernetes.io~projected/kube-api-access-s4hm8:{mountpoint:/var/lib/kubelet/pods/2f7f681b-d410-40e9-a4d0-487230d8dece/volumes/kubernetes.io~projected/kube-api-access-s4hm8 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32d4bb3a-8abf-408a-8168-4581172f47f1/volumes/kubernetes.io~projected/kube-api-access-wz2vv:{mountpoint:/var/lib/kubelet/pods/32d4bb3a-8abf-408a-8168-4581172f47f1/volumes/kubernetes.io~projected/kube-api-access-wz2vv major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a492c60-d3a8-417b-b45f-254e82f2ae19/volumes/kubernetes.io~projected/kube-api-access-965xj:{mountpoint:/var/lib/kubelet/pods/3a492c60-d3a8-417b-b45f-254e82f2ae19/volumes/kubernetes.io~projected/kube-api-access-965xj major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d824491-3dcc-4c32-8378-7c8459b34086/volumes/kubernetes.io~projected/kube-api-access-mwx2t:{mountpoint:/var/lib/kubelet/pods/3d824491-3dcc-4c32-8378-7c8459b34086/volumes/kubernetes.io~projected/kube-api-access-mwx2t major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d824491-3dcc-4c32-8378-7c8459b34086/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/3d824491-3dcc-4c32-8378-7c8459b34086/volumes/kubernetes.io~secret/metrics-certs major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/571bdc94-d1ed-4ede-87fd-65af23ce24e9/volumes/kubernetes.io~projected/kube-api-access-zmkv6:{mountpoint:/var/lib/kubelet/pods/571bdc94-d1ed-4ede-87fd-65af23ce24e9/volumes/kubernetes.io~projected/kube-api-access-zmkv6 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/571bdc94-d1ed-4ede-87fd-65af23ce24e9/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/571bdc94-d1ed-4ede-87fd-65af23ce24e9/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a8d8001-c1b0-4489-9e25-fb8b67048d7d/volumes/kubernetes.io~projected/kube-api-access-dp42j:{mountpoint:/var/lib/kubelet/pods/5a8d8001-c1b0-4489-9e25-fb8b67048d7d/volumes/kubernetes.io~projected/kube-api-access-dp42j major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79/volumes/kubernetes.io~projected/kube-api-access-wmdcp:{mountpoint:/var/lib/kubelet/pods/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79/volumes/kubernetes.io~projected/kube-api-access-wmdcp major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volume-subpaths/web-config/alertmanager/9 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volume-subpaths/web-config/config-reloader/6 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~empty-dir/config-out major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~projected/kube-api-access-wfvkd:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~projected/kube-api-access-wfvkd major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~projected/tls-assets major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/config-volume major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/web-config major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af477ca4-5745-4345-8e68-21ac9c32221f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/af477ca4-5745-4345-8e68-21ac9c32221f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af477ca4-5745-4345-8e68-21ac9c32221f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/af477ca4-5745-4345-8e68-21ac9c32221f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af477ca4-5745-4345-8e68-21ac9c32221f/volumes/kubernetes.io~projected/kube-api-access-627v6:{mountpoint:/var/lib/kubelet/pods/af477ca4-5745-4345-8e68-21ac9c32221f/volumes/kubernetes.io~projected/kube-api-access-627v6 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b/volumes/kubernetes.io~projected/kube-api-access-86vpg:{mountpoint:/var/lib/kubelet/pods/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b/volumes/kubernetes.io~projected/kube-api-access-86vpg major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volume-subpaths/web-config/config-reloader/2 major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volume-subpaths/web-config/prometheus/15 major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~empty-dir/config-out major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~projected/kube-api-access-84tzx:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~projected/kube-api-access-84tzx major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~projected/tls-assets major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/config major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/web-config major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da04dce2-a300-44f6-ab49-5fb77c6c8a19/volumes/kubernetes.io~projected/kube-api-access-s8pqf:{mountpoint:/var/lib/kubelet/pods/da04dce2-a300-44f6-ab49-5fb77c6c8a19/volumes/kubernetes.io~projected/kube-api-access-s8pqf major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da04dce2-a300-44f6-ab49-5fb77c6c8a19/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/da04dce2-a300-44f6-ab49-5fb77c6c8a19/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df19de7c-3e77-48ce-b5e4-b4b56ab26017/volumes/kubernetes.io~projected/kube-api-access-pfhx7:{mountpoint:/var/lib/kubelet/pods/df19de7c-3e77-48ce-b5e4-b4b56ab26017/volumes/kubernetes.io~projected/kube-api-access-pfhx7 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df19de7c-3e77-48ce-b5e4-b4b56ab26017/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/df19de7c-3e77-48ce-b5e4-b4b56ab26017/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df19de7c-3e77-48ce-b5e4-b4b56ab26017/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/df19de7c-3e77-48ce-b5e4-b4b56ab26017/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df19de7c-3e77-48ce-b5e4-b4b56ab26017/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/df19de7c-3e77-48ce-b5e4-b4b56ab26017/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e263af32-db0f-4984-8e90-83347cdd15f7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e263af32-db0f-4984-8e90-83347cdd15f7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e263af32-db0f-4984-8e90-83347cdd15f7/volumes/kubernetes.io~projected/kube-api-access-x4nvm:{mountpoint:/var/lib/kubelet/pods/e263af32-db0f-4984-8e90-83347cdd15f7/volumes/kubernetes.io~projected/kube-api-access-x4nvm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e263af32-db0f-4984-8e90-83347cdd15f7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e263af32-db0f-4984-8e90-83347cdd15f7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee4a4694-e358-4491-b4e1-c05bd477f19a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ee4a4694-e358-4491-b4e1-c05bd477f19a/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f57a8017-198c-42a9-9fb4-481348d8568c/volumes/kubernetes.io~projected/kube-api-access-x4gt7:{mountpoint:/var/lib/kubelet/pods/f57a8017-198c-42a9-9fb4-481348d8568c/volumes/kubernetes.io~projected/kube-api-access-x4gt7 major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/dd631f4d07e57ad0149ee86269922157a3085c91e70c4f0311fd991cbb7b980d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/135acd938a7ac3c3369e3a50d02b99430dcb61fddf21e80fd16329bc8a32199c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d2629fab0748289d3e333eadb39c0b1c69f9646aac0dfea43bc0150255223456/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/91f23ac270872da97bc9544ba4e0a42eb90611dafa3d2da51bd30af1985805db/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ed74aecb9c3af06393d9dddd8808c1368bfcbeedd8076322644f02a25d663e62/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/2588b1b1a7994a2ec11727dfe09963ce33d490942fa66ca86f0ca0f217a8eb14/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/786c632e45498f82f2a76103f6e9b41c32e50b069670844fbff7f6ec0586043f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0b0eb41977d5598d4fde8087cbfda22f9b24859658f1e43d24a470f9a070ab1d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f938157915026404ea9edd266c24534b5da1926123149d99ee7f36ec328488aa/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e88b688c759789d3cb96c23c48331c2e731af4fb11b815e1ed5f79943d37cd1f/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/0e76500c36722812be8c55fb5687e5d3b058ed1933e99b666c8ea858827c6e5b/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/2def63a76ada52d8d5d9d3744967048293e8015175245cd37cd2005046ec5fe5/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/b7c5e060a739546526d6c1ad34bc90320a8ce8573dbadeb4ad8ee75ede10df30/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4cee34949a3edf8e08e07b35e7adb76f85ac3e692e2842fb46700f2abe7123e2/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2eb2b61460a56f0870a5eca6ed4ccd8ef30dcd2559d9e20914ae0e33c785367d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0b8e6b488fa0a27f5bf8145ee974e00b1f19bb3778e71569fc5263728beee3a0/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/183691dafc63da0b1a49792ee2484fbc7860fee6a7eee52705229da036953165/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/8fde04cde78af1e872be36a4d9be2b7f663a5723dbec88e1a04b2ba59c79d29d/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f93a1cbe60afb69bc7bcba18766d68200fd3b37c758b352e29caaf53ab9b01c5/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0a0e626ae1b41f1d0defa471859133c8ddf2cc1ec9556a17d9d5bcf963cd0e4a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f97319031f34d56fd4f8d98e64cb33e1b69704064934f07e41fc2c8aa87353ad/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/66fddbf3b6c52393d8b54501f6fb0c033b90979176f198713df7dc6a8a436e37/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/a62c8dbb89cd84ad2d6a95113409eea03c3e3fb4264f60601f083559213ec5a0/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/6d060c2dca63e8a8344139b2772e7a9b1171f40aa9897079a8e83b861f09aa52/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/35d24e124ca2d406c747321ad40eb8935872323ada47741405df647c2e7d7560/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/196bb73b3a2d39b951c7e9ceb480c2ea757de374e30f8ea56d30c0226fb5375d/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/bd142291dbe2b1b7253dbb6260a31c3ba4aad5f1e6845c47ec6ddffcd70ce48c/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/171225ec592e491589cb18434ca0cbb1ad41934f96f8efcc3e7a256a1c20d137/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/8bad2b154d225b4e354b154939511f75154d8d7b0bfa1326ce6938270c29c854/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/af8095c0f8f1463289ff5ee1d436fd721516ab3d35332a4db793c078a1e7920e/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/dfc57253932833425b484fc0377926a321baadf5a8d604fbaddc58545316d288/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/c7c66f7b0c9179bc485479bd836287624ad84eaecf9d0f3c4af3aad9ad07f7c7/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/cbd94e1a41308a6fa6bf0f265af8cdf5d214b6c8a03a77bd31775f429a0b46f0/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/2d6b3a56824fff2586253f0b7a49af83d1923caae31a7c37376284d1bc5ee884/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/e18187aa02992c1b2ed28b1e4ebb8f1615f133fbbae33191ef60cdb17f3e3783/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/64cd4dc86110ad91d92bdc26331f1bd5ac7ab7d189c7e6d32c4a5a6178543625/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/488f4fc23c6b345d81aab777459efc09b0f4c7fcb9c6a1fca982b6d4a6dca867/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/1102c9d1147b218d5d948dd2fde1356264b6a4c8eecab0ef482e5e61db9159c3/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/6de1b04669b89fc3b957318ff7a1899c1d79c46dca7cdeb291ee3e8f7c2c262a/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/0f35d86981f6f6079506e73c3b010b81aa1c3fc08502b2043d6e4ba96da5afe9/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/40f0cf3362e8c4cd104d9d75e0b7e610492fdb3926c696e05cd475323345114d/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/e0383a4c7f9970d21acf441509837054892365a089b3184956f98ecc1ec70708/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/f37e17c1003f06da1c043e7362108c000c1d3f720c9855c760a2cd0535c932a7/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/c55631d0ce648de46773216fe665a71fe99da1e34ae50e9b8b549f878ffea23f/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/53a3ef77c664afcb1f5d4bb8b4ce009b385f62847c2f758b9363385bab864e4d/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/23a5e73cb398cb28c6731467efe9697c837143a496db770c78d90b523f046ab8/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/cce95ca6045baa3b90bc9851f6332f74e0aae046f023038e82622aa4f337c0a2/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/47ff27f918b4c7198885a5cfa91d314047241b2d051eb22584ae04a77af083dc/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/5cd764cf6f1d270bec8d44d41f9e834ba55319e2af430e2ef34890ab675ca29a/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/2a69766b260ed82c6d4a77855695e8b00bd8e62f6912aa272a85f852d5c8e329/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/e8a85c706ebb203c185acafda7e4c89e94a8ea9166aad6971656111ef372dfec/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/3bf0e16d532be8052aeb3e13f755ddc109f5e258c5e5b6163e1b544b04afa6c2/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/c13f8c58f6977f2e84030ee76c762fa22a319039a9f32ae5e6f3f4239920d820/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/fcf4965236283d5e39205d2bb6d9c3066b31c58895b582a82cdfb42956717745/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/96db3ae30ee2464df7fa3430f77dfa31c93c58e39f092e9f9f2a4dbf09883163/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/8d31769d57c43714a9ff2276cc4ce000aa7a9108fc521cad96d9a04d6596c667/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/a7138ebcd0713e0d4eb40b91fe59ea3118ebeb5455c978771c07088c87ad7d13/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/b1ea558f15719cff5cf15b670fb13478bb6abbc42a18bed050b40bc6fa773bed/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/df8e4a6ad391dbe943ba61568d43e34fca2daee8e2680fb7d1c2b0d390f8d96c/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/47984f6b5ad7b938f18afe6738a761d364851485aeb203a1a0f7a1227f1d901f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/6f69fdfb75d6bf11e4c051eb9f3207426bda46c8f82ad3bf9740002f3f7ef537/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9afcdd033372e7daf9eaef860eb68a84cdf03d7e29bddadeb5354e1fa05aa563/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/d447248a9372dbfcb1196a922b329a47a1f7965cf23934e1cea3028862245753/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/e58c4f26a39aef7ccdbd7a8ab69118dfaf094ced5af81c759360661099fc870c/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/87071b7dee80c95983a6854081b3ce271cfdb9e7300b4ba143d913b498c61482/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8e8a085f202484a19686ce7b79c92d2f630fc739a0158f6ab0279bbc5ed4a3a3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/930e7530b90e4a909a0546626dfc276e725a2acb8eb83a939ca3848c39f04598/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/0657ef9feb68c061b5acaa84bea1773eb51aa2dc85a2b3fa2dbee2d0d1f44980/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 01:18:40.633266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.632834    5508 manager.go:217] Machine: {Timestamp:2026-11-03 01:18:40.63228653 +0000 UTC m=+0.057892990 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101168 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bd29b167c3ad55ed43a9ee71b9523 SystemUUID:ec2bd29b-167c-3ad5-5ed4-3a9ee71b9523 BootID:20520735-9dd0-4dd4-9a18-fb622990c068 Filesystems:[{Device:/var/lib/kubelet/pods/af477ca4-5745-4345-8e68-21ac9c32221f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:323 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:352 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d824491-3dcc-4c32-8378-7c8459b34086/volumes/kubernetes.io~projected/kube-api-access-mwx2t DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85ece4a25e76922c64587102669c48816d4b7c3f1a4dc7929269a6296bd3e586/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:287 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2f7f681b-d410-40e9-a4d0-487230d8dece/volumes/kubernetes.io~projected/kube-api-access-s4hm8 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~projected/kube-api-access-84tzx DeviceMajor:0 DeviceMinor:327 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f57a8017-198c-42a9-9fb4-481348d8568c/volumes/kubernetes.io~projected/kube-api-access-x4gt7 DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:247 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a8d8001-c1b0-4489-9e25-fb8b67048d7d/volumes/kubernetes.io~projected/kube-api-access-dp42j DeviceMajor:0 DeviceMinor:365 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a492c60-d3a8-417b-b45f-254e82f2ae19/volumes/kubernetes.io~projected/kube-api-access-965xj DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:320 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e263af32-db0f-4984-8e90-83347cdd15f7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/df19de7c-3e77-48ce-b5e4-b4b56ab26017/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:294 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/df19de7c-3e77-48ce-b5e4-b4b56ab26017/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:296 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f38450dda7be5a25d62612d25a520ea805998c5bfc99b696fa5c244732de00f9/userdata/shm DeviceMajor:0 DeviceMinor:371 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:330 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43a30ba52f58f20dddbae3241d362c4a7e8a1e3aeb090b9cbd6fe066a476c494/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/505cac9e6dc55bc5f598dfb85f7b1048f7a448fdb3ece07be3bc44096df40b8c/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:281 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79/volumes/kubernetes.io~projected/kube-api-access-wmdcp DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b/volumes/kubernetes.io~projected/kube-api-access-86vpg DeviceMajor:0 DeviceMinor:248 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/125c3968-dca4-4163-b473-b5cc7ac5a23f/volumes/kubernetes.io~projected/kube-api-access-r2rxv DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/32d4bb3a-8abf-408a-8168-4581172f47f1/volumes/kubernetes.io~projected/kube-api-access-wz2vv DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe46741258363e7b02e84970875d6fe7881a3c9c79580df4429238fde64e8890/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ccd809ce8e6cd82cf0867fe0496a4e5e03174f8099bdbf41a43f338214d4cf7e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da04dce2-a300-44f6-ab49-5fb77c6c8a19/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/378fbd1c82140a4a08b20ef9e4bae339b4e720a2852ece2091adb3c0b7f2a9f7/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:316 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d47f5ae5cf9b860ed0482d0634fb887e4bbaff1cd439d70bb2b1ccd7cb3b0ff/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af477ca4-5745-4345-8e68-21ac9c32221f/volumes/kubernetes.io~projected/kube-api-access-627v6 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/beac3c04fe0d07c4c840497823bbf4f74df7dcc3467bf87f339a69b70bb73ddf/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:243 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:328 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c57dcf4-5aa3-4992-b357-47b388f3db90/volumes/kubernetes.io~projected/kube-api-access-jqwfz DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e263af32-db0f-4984-8e90-83347cdd15f7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/df19de7c-3e77-48ce-b5e4-b4b56ab26017/volumes/kubernetes.io~projected/kube-api-access-pfhx7 DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:318 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/af477ca4-5745-4345-8e68-21ac9c32221f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~projected/kube-api-access-wfvkd DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/571bdc94-d1ed-4ede-87fd-65af23ce24e9/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:285 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3d824491-3dcc-4c32-8378-7c8459b34086/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/802890bbb04d593748e26313e577cb888c7d3021afd8886c63fb74d4a641ff18/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef0f59abb1c1d15c9ef032f36c068cd2b1f91f73d54c6b7e197180ac6a5f12c6/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e263af32-db0f-4984-8e90-83347cdd15f7/volumes/kubernetes.io~projected/kube-api-access-x4nvm DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:282 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:315 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3928afa6d9ebb017886812fdf9721e4bf2f3ffca76f5d478378f2b2a9c4ba709/userdata/shm DeviceMajor:0 DeviceMinor:369 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af02d4102d3d30d1c8d8279100a03aa5439e262ea74992ca242c0f330dcb4f3d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3dc67cea4c5f3357064a3154d04a1e9e6df73c9d41b25a0e72c2d4faa0b9ef7e/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:285 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b423824438a1755dab77a92ade9560b77f48bdca244bbf5fd64c0af4154268e8/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78fe8905288581090afb36532f05784b20ebca0c055ed6c8e1074a9d6ff519be/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/571bdc94-d1ed-4ede-87fd-65af23ce24e9/volumes/kubernetes.io~projected/kube-api-access-zmkv6 DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:319 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5fa2ad137caf5bd6425053cc5496d81379b1fb1e5e7ad5046731d516d42c366/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da04dce2-a300-44f6-ab49-5fb77c6c8a19/volumes/kubernetes.io~projected/kube-api-access-s8pqf DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:310 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53a809541a9c2f2c9dfb47ffbe524e464932d733927bfde7895be99685c0fe65/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:317 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c03a8882de743114118d57225ecbfd9b55b207162f7326b89957a2928e3a925/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4/volumes/kubernetes.io~projected/kube-api-access-px6wp DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96657bb1ef573dd77f20f4de64ba4b4c5112a364f0daf3424978f71b7ff36c61/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ee4a4694-e358-4491-b4e1-c05bd477f19a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/df19de7c-3e77-48ce-b5e4-b4b56ab26017/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:295 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8205c515c9a9f8f445dc87459de92b0bbc943cddc6bbdafcf0e5447bfcd49876/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:378fbd1c82140a4 MacAddress:2a:0c:19:b7:07:e0 Speed:10000 Mtu:8901} {Name:3928afa6d9ebb01 MacAddress:a2:4b:be:e0:04:2b Speed:10000 Mtu:8901} {Name:505cac9e6dc55bc MacAddress:f2:a4:69:14:f8:9b Speed:10000 Mtu:8901} {Name:53a809541a9c2f2 MacAddress:32:3c:dd:50:28:e8 Speed:10000 Mtu:8901} {Name:8205c515c9a9f8f MacAddress:5e:87:72:75:dc:35 Speed:10000 Mtu:8901} {Name:96657bb1ef573dd MacAddress:56:83:fb:cd:e3:11 Speed:10000 Mtu:8901} {Name:9c03a8882de7431 MacAddress:2a:fe:86:5d:9c:5f Speed:10000 Mtu:8901} {Name:b423824438a1755 MacAddress:9a:f5:5e:eb:6b:12 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f8:80:d4:57 Speed:0 Mtu:9001} {Name:br-int MacAddress:f2:d8:3f:be:22:f6 Speed:0 Mtu:8901} {Name:ef0f59abb1c1d15 MacAddress:ba:ec:04:89:7a:69 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f8:80:d4:57 Speed:0 Mtu:9001} {Name:f38450dda7be5a2 MacAddress:de:06:cc:c8:4e:39 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:76:58:ec:c9:f7:6e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ae:fb:d9:7c:d2:46 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 01:18:40.633482 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.633270    5508 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 01:18:40.633482 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.633316    5508 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 01:18:40.634652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.633708    5508 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 01:18:40.634652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.633751    5508 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-255.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 01:18:40.634652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.634078    5508 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 01:18:40.634652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.634089    5508 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 01:18:40.634652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.634109    5508 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 01:18:40.634652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.634128    5508 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 01:18:40.634652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.634263    5508 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 01:18:40.634652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.634559    5508 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 01:18:40.635004 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.634771    5508 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 01:18:40.635037 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.635024    5508 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 01:18:40.635065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.635056    5508 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 01:18:40.635095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.635074    5508 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 01:18:40.635095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.635090    5508 kubelet.go:405] "Adding apiserver pod source"
Nov 03 01:18:40.635156 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.635102    5508 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 01:18:40.635925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.635913    5508 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 01:18:40.635925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.635926    5508 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 01:18:40.636462 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.636444    5508 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 01:18:40.639097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639084    5508 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 01:18:40.639148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639098    5508 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 01:18:40.639203 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639194    5508 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 01:18:40.639412 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639401    5508 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 01:18:40.639445 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639431    5508 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 01:18:40.639536 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639527    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 01:18:40.639570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639541    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 01:18:40.639570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639548    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 01:18:40.639570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639553    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 01:18:40.639570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639559    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 01:18:40.639570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639565    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 01:18:40.639570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639570    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 01:18:40.639730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639576    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 01:18:40.639730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639583    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 01:18:40.639730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639588    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 01:18:40.639730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639601    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 01:18:40.639730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639611    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 01:18:40.639730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639633    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 01:18:40.639730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.639639    5508 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 01:18:40.643047 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.643031    5508 server.go:1263] "Started kubelet"
Nov 03 01:18:40.643262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.643117    5508 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 01:18:40.643262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.643194    5508 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 01:18:40.643416 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.643282    5508 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 01:18:40.643783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.643754    5508 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 01:18:40.644917 ip-10-0-140-255 systemd[1]: Started Kubernetes Kubelet.
Nov 03 01:18:40.646474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.644912    5508 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 01:18:40.649002 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.648980    5508 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 01:18:40.649115 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.649074    5508 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 01:18:40.649115 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.649095    5508 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 01:18:40.649115 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:40.649089    5508 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-255.us-west-2.compute.internal\" not found"
Nov 03 01:18:40.649265 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.649175    5508 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 01:18:40.651034 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.650999    5508 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 01:18:40.651522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.651340    5508 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 01:12:26 +0000 UTC" deadline="2028-05-20 18:32:46.241432337 +0000 UTC"
Nov 03 01:18:40.651522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.651410    5508 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13553h14m5.590026584s"
Nov 03 01:18:40.651645 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.651596    5508 factory.go:55] Registering systemd factory
Nov 03 01:18:40.651645 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.651613    5508 factory.go:223] Registration of the systemd container factory successfully
Nov 03 01:18:40.652292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652249    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="571bdc94-d1ed-4ede-87fd-65af23ce24e9" volumeName="kubernetes.io/configmap/571bdc94-d1ed-4ede-87fd-65af23ce24e9-config-volume" seLinuxMountContext=""
Nov 03 01:18:40.652387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652294    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a804a1ea-8545-4932-a98b-fba23d40688b" volumeName="kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-config-volume" seLinuxMountContext=""
Nov 03 01:18:40.652387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652309    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7d8a03d-d25a-448a-955f-9b7ea3f4e07b" volumeName="kubernetes.io/secret/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-tls" seLinuxMountContext=""
Nov 03 01:18:40.652387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652321    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 03 01:18:40.652387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652333    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config" seLinuxMountContext=""
Nov 03 01:18:40.652387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652374    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c57dcf4-5aa3-4992-b357-47b388f3db90" volumeName="kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652388    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f7f681b-d410-40e9-a4d0-487230d8dece" volumeName="kubernetes.io/configmap/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-daemon-config" seLinuxMountContext=""
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652401    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a8d8001-c1b0-4489-9e25-fb8b67048d7d" volumeName="kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j" seLinuxMountContext=""
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652416    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652428    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652439    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df19de7c-3e77-48ce-b5e4-b4b56ab26017" volumeName="kubernetes.io/configmap/df19de7c-3e77-48ce-b5e4-b4b56ab26017-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652451    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" volumeName="kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config" seLinuxMountContext=""
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652475    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a804a1ea-8545-4932-a98b-fba23d40688b" volumeName="kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-kube-api-access-wfvkd" seLinuxMountContext=""
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652487    5508 factory.go:153] Registering CRI-O factory
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652505    5508 factory.go:223] Registration of the crio container factory successfully
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652489    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f57a8017-198c-42a9-9fb4-481348d8568c" volumeName="kubernetes.io/configmap/f57a8017-198c-42a9-9fb4-481348d8568c-serviceca" seLinuxMountContext=""
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652575    5508 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652586    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="125c3968-dca4-4163-b473-b5cc7ac5a23f" volumeName="kubernetes.io/projected/125c3968-dca4-4163-b473-b5cc7ac5a23f-kube-api-access-r2rxv" seLinuxMountContext=""
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652597    5508 factory.go:103] Registering Raw factory
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652600    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c57dcf4-5aa3-4992-b357-47b388f3db90" volumeName="kubernetes.io/projected/2c57dcf4-5aa3-4992-b357-47b388f3db90-kube-api-access-jqwfz" seLinuxMountContext=""
Nov 03 01:18:40.652605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652615    5508 manager.go:1199] Started watching for new ooms in manager
Nov 03 01:18:40.653267 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652980    5508 manager.go:319] Starting recovery of all containers
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.652613    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a5d501e-fa40-467b-a4fa-9fc52f9c45fa" volumeName="kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653430    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config-out" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653457    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df19de7c-3e77-48ce-b5e4-b4b56ab26017" volumeName="kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-client-ca-bundle" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653470    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="571bdc94-d1ed-4ede-87fd-65af23ce24e9" volumeName="kubernetes.io/secret/571bdc94-d1ed-4ede-87fd-65af23ce24e9-metrics-tls" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653483    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a804a1ea-8545-4932-a98b-fba23d40688b" volumeName="kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-config-out" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653495    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a804a1ea-8545-4932-a98b-fba23d40688b" volumeName="kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653508    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-kube-api-access-84tzx" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653522    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653534    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-grpc-tls" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653547    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653560    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e263af32-db0f-4984-8e90-83347cdd15f7" volumeName="kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653572    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e263af32-db0f-4984-8e90-83347cdd15f7" volumeName="kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-env-overrides" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653615    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e263af32-db0f-4984-8e90-83347cdd15f7" volumeName="kubernetes.io/secret/e263af32-db0f-4984-8e90-83347cdd15f7-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653631    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f7f681b-d410-40e9-a4d0-487230d8dece" volumeName="kubernetes.io/projected/2f7f681b-d410-40e9-a4d0-487230d8dece-kube-api-access-s4hm8" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653643    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d824491-3dcc-4c32-8378-7c8459b34086" volumeName="kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653658    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a804a1ea-8545-4932-a98b-fba23d40688b" volumeName="kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-metrics-client-ca" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653675    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a804a1ea-8545-4932-a98b-fba23d40688b" volumeName="kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-tls-assets" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653687    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653699    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da04dce2-a300-44f6-ab49-5fb77c6c8a19" volumeName="kubernetes.io/projected/da04dce2-a300-44f6-ab49-5fb77c6c8a19-kube-api-access-s8pqf" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653712    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32d4bb3a-8abf-408a-8168-4581172f47f1" volumeName="kubernetes.io/projected/32d4bb3a-8abf-408a-8168-4581172f47f1-kube-api-access-wz2vv" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653723    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a492c60-d3a8-417b-b45f-254e82f2ae19" volumeName="kubernetes.io/projected/3a492c60-d3a8-417b-b45f-254e82f2ae19-kube-api-access-965xj" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653736    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63489afa-8dc5-48b4-a0b6-2c40bdd8ff79" volumeName="kubernetes.io/secret/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653748    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-tls-assets" seLinuxMountContext=""
Nov 03 01:18:40.653743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653761    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653774    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df19de7c-3e77-48ce-b5e4-b4b56ab26017" volumeName="kubernetes.io/projected/df19de7c-3e77-48ce-b5e4-b4b56ab26017-kube-api-access-pfhx7" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653787    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" volumeName="kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653799    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63489afa-8dc5-48b4-a0b6-2c40bdd8ff79" volumeName="kubernetes.io/projected/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-kube-api-access-wmdcp" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653811    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a804a1ea-8545-4932-a98b-fba23d40688b" volumeName="kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653822    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a804a1ea-8545-4932-a98b-fba23d40688b" volumeName="kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-cluster-tls-config" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653835    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653864    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da04dce2-a300-44f6-ab49-5fb77c6c8a19" volumeName="kubernetes.io/secret/da04dce2-a300-44f6-ab49-5fb77c6c8a19-cert" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653881    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df19de7c-3e77-48ce-b5e4-b4b56ab26017" volumeName="kubernetes.io/empty-dir/df19de7c-3e77-48ce-b5e4-b4b56ab26017-audit-log" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653897    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4a4694-e358-4491-b4e1-c05bd477f19a" volumeName="kubernetes.io/secret/ee4a4694-e358-4491-b4e1-c05bd477f19a-agent-certs" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653909    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" volumeName="kubernetes.io/empty-dir/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-tmp" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653923    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c57dcf4-5aa3-4992-b357-47b388f3db90" volumeName="kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-cni-binary-copy" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653947    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="571bdc94-d1ed-4ede-87fd-65af23ce24e9" volumeName="kubernetes.io/projected/571bdc94-d1ed-4ede-87fd-65af23ce24e9-kube-api-access-zmkv6" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653960    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a804a1ea-8545-4932-a98b-fba23d40688b" volumeName="kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653973    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af477ca4-5745-4345-8e68-21ac9c32221f" volumeName="kubernetes.io/empty-dir/af477ca4-5745-4345-8e68-21ac9c32221f-tmp" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653986    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7d8a03d-d25a-448a-955f-9b7ea3f4e07b" volumeName="kubernetes.io/configmap/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-metrics-client-ca" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.653999    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654029    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654042    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="125c3968-dca4-4163-b473-b5cc7ac5a23f" volumeName="kubernetes.io/empty-dir/125c3968-dca4-4163-b473-b5cc7ac5a23f-tmp-dir" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654055    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" volumeName="kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654071    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c57dcf4-5aa3-4992-b357-47b388f3db90" volumeName="kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654086    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="571bdc94-d1ed-4ede-87fd-65af23ce24e9" volumeName="kubernetes.io/empty-dir/571bdc94-d1ed-4ede-87fd-65af23ce24e9-tmp-dir" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654099    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-metrics-client-certs" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654113    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df19de7c-3e77-48ce-b5e4-b4b56ab26017" volumeName="kubernetes.io/configmap/df19de7c-3e77-48ce-b5e4-b4b56ab26017-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654127    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f57a8017-198c-42a9-9fb4-481348d8568c" volumeName="kubernetes.io/projected/f57a8017-198c-42a9-9fb4-481348d8568c-kube-api-access-x4gt7" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654141    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" volumeName="kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654157    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f7f681b-d410-40e9-a4d0-487230d8dece" volumeName="kubernetes.io/configmap/2f7f681b-d410-40e9-a4d0-487230d8dece-cni-binary-copy" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654170    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63489afa-8dc5-48b4-a0b6-2c40bdd8ff79" volumeName="kubernetes.io/configmap/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654183    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a804a1ea-8545-4932-a98b-fba23d40688b" volumeName="kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-main-db" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654195    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7d8a03d-d25a-448a-955f-9b7ea3f4e07b" volumeName="kubernetes.io/configmap/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654209    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7d8a03d-d25a-448a-955f-9b7ea3f4e07b" volumeName="kubernetes.io/secret/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654222    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654251    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df19de7c-3e77-48ce-b5e4-b4b56ab26017" volumeName="kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654266    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" volumeName="kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654280    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63489afa-8dc5-48b4-a0b6-2c40bdd8ff79" volumeName="kubernetes.io/empty-dir/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-data-volume" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654291    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af477ca4-5745-4345-8e68-21ac9c32221f" volumeName="kubernetes.io/empty-dir/af477ca4-5745-4345-8e68-21ac9c32221f-etc-tuned" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654304    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df19de7c-3e77-48ce-b5e4-b4b56ab26017" volumeName="kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654316    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e263af32-db0f-4984-8e90-83347cdd15f7" volumeName="kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-ovnkube-config" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654331    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" volumeName="kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654343    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a804a1ea-8545-4932-a98b-fba23d40688b" volumeName="kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-web-config" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654403    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7d8a03d-d25a-448a-955f-9b7ea3f4e07b" volumeName="kubernetes.io/empty-dir/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-textfile" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654416    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7d8a03d-d25a-448a-955f-9b7ea3f4e07b" volumeName="kubernetes.io/projected/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-kube-api-access-86vpg" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654428    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-db" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654441    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a804a1ea-8545-4932-a98b-fba23d40688b" volumeName="kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654452    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a804a1ea-8545-4932-a98b-fba23d40688b" volumeName="kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654464    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af477ca4-5745-4345-8e68-21ac9c32221f" volumeName="kubernetes.io/projected/af477ca4-5745-4345-8e68-21ac9c32221f-kube-api-access-627v6" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654475    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654487    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" volumeName="kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-web-config" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654500    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e263af32-db0f-4984-8e90-83347cdd15f7" volumeName="kubernetes.io/projected/e263af32-db0f-4984-8e90-83347cdd15f7-kube-api-access-x4nvm" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654510    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee4a4694-e358-4491-b4e1-c05bd477f19a" volumeName="kubernetes.io/configmap/ee4a4694-e358-4491-b4e1-c05bd477f19a-konnectivity-ca" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654522    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32d4bb3a-8abf-408a-8168-4581172f47f1" volumeName="kubernetes.io/configmap/32d4bb3a-8abf-408a-8168-4581172f47f1-iptables-alerter-script" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654533    5508 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d824491-3dcc-4c32-8378-7c8459b34086" volumeName="kubernetes.io/projected/3d824491-3dcc-4c32-8378-7c8459b34086-kube-api-access-mwx2t" seLinuxMountContext=""
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654545    5508 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 01:18:40.655032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.654552    5508 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 01:18:40.682668 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.682603    5508 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 01:18:40.683943 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.683918    5508 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 01:18:40.683943 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.683942    5508 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 01:18:40.684130 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.683966    5508 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 01:18:40.684130 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:40.684006    5508 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 01:18:40.694850 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.694273    5508 generic.go:356] "Generic (PLEG): container finished" podID="b7d8a03d-d25a-448a-955f-9b7ea3f4e07b" containerID="e52d33eba4e2367878a39bf15e5f9650f46ecd5413210a0ba324eb2c3639e168" exitCode=0
Nov 03 01:18:40.710056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.710021    5508 generic.go:356] "Generic (PLEG): container finished" podID="a804a1ea-8545-4932-a98b-fba23d40688b" containerID="5d9c4f61324f1587b1cb3764bbedf414f734f3e32409c48ab3b04449dfe8a22c" exitCode=0
Nov 03 01:18:40.720732 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.720699    5508 generic.go:356] "Generic (PLEG): container finished" podID="2c57dcf4-5aa3-4992-b357-47b388f3db90" containerID="2ee77a8535902aae0d33bd7822cda2d7b539c999517479538319fbf49fd26aa1" exitCode=0
Nov 03 01:18:40.720732 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.720726    5508 generic.go:356] "Generic (PLEG): container finished" podID="2c57dcf4-5aa3-4992-b357-47b388f3db90" containerID="875bb96059a4fc0adaf42f12c6172991544deca6ba35a0c0e75d75f391d71088" exitCode=0
Nov 03 01:18:40.720732 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.720734    5508 generic.go:356] "Generic (PLEG): container finished" podID="2c57dcf4-5aa3-4992-b357-47b388f3db90" containerID="58861f089f8441c24593827d53971956c1d0570b7a85ac5fda92968ad7dcf7dc" exitCode=0
Nov 03 01:18:40.720732 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.720742    5508 generic.go:356] "Generic (PLEG): container finished" podID="2c57dcf4-5aa3-4992-b357-47b388f3db90" containerID="2225b0e970fda624a9e6c39a207bdc3947ef3e9d572abe0ee930fce0dc18e6fb" exitCode=0
Nov 03 01:18:40.721064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.720752    5508 generic.go:356] "Generic (PLEG): container finished" podID="2c57dcf4-5aa3-4992-b357-47b388f3db90" containerID="a302200e3662c2c7b13cb7b62b70a0de5dcb1dc4fd80d18fb81302807db923e2" exitCode=0
Nov 03 01:18:40.721064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.720764    5508 generic.go:356] "Generic (PLEG): container finished" podID="2c57dcf4-5aa3-4992-b357-47b388f3db90" containerID="84284ba03e65fc56b0b2da0ca8b11dc86f03666733ab714fb1543bd473b7019f" exitCode=0
Nov 03 01:18:40.725675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.725642    5508 generic.go:356] "Generic (PLEG): container finished" podID="bd3c5a92e769172fde5847bb75e6fd4d" containerID="0bab3edfc0f815357e2d56c2a6bacd1b4a446e9d135e35231ae74b4f35a623ec" exitCode=0
Nov 03 01:18:40.735659 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.735627    5508 generic.go:356] "Generic (PLEG): container finished" podID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerID="b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4" exitCode=0
Nov 03 01:18:40.749218 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:40.749150    5508 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-255.us-west-2.compute.internal\" not found"
Nov 03 01:18:40.758071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.758047    5508 manager.go:324] Recovery completed
Nov 03 01:18:40.779848 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.779820    5508 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:18:40.781728 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.781705    5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:18:40.781818 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.781748    5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:18:40.781818 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.781759    5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:18:40.783997 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.783972    5508 cpu_manager.go:226] "Starting" policy="none"
Nov 03 01:18:40.783997 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.783989    5508 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:18:40.784130 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.783990    5508 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 01:18:40.784130 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.784095    5508 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 01:18:40.784130 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:40.784103    5508 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 01:18:40.784387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.784239    5508 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 01:18:40.784387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.784250    5508 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 01:18:40.784387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.784388    5508 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 01:18:40.784522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.784394    5508 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 01:18:40.784522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.784401    5508 policy_none.go:50] "Start"
Nov 03 01:18:40.784522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.784420    5508 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 01:18:40.784522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.784430    5508 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 01:18:40.785000 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.784986    5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:18:40.785069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.785012    5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:18:40.785069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.785023    5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:18:40.785168 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.785156    5508 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 01:18:40.785198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.785170    5508 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 01:18:40.785198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.785176    5508 policy_none.go:44] "Start"
Nov 03 01:18:40.789682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.789656    5508 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 01:18:40.792997 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.792985    5508 manager.go:341] "Starting Device Plugin manager"
Nov 03 01:18:40.793089 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:40.793007    5508 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 01:18:40.793089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.793015    5508 server.go:87] "Starting device plugin registration server"
Nov 03 01:18:40.793268 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.793256    5508 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 01:18:40.793329 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.793267    5508 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 01:18:40.793401 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.793327    5508 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 01:18:40.793477 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.793438    5508 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 01:18:40.793595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.793486    5508 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 01:18:40.793595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.793493    5508 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 01:18:40.793693 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.793547    5508 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T01:18:40.793471039Z","UUID":"c8355c7e-d1a0-417d-98d5-c27e2768b0e7","Handler":null,"Name":"","Endpoint":""}
Nov 03 01:18:40.795294 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:40.795274    5508 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 01:18:40.795415 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:40.795315    5508 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-255.us-west-2.compute.internal\" not found"
Nov 03 01:18:40.795634 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.795620    5508 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 01:18:40.795700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.795653    5508 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 01:18:40.836323 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.836287    5508 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:18:40.836525 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.836457    5508 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 01:18:40.836525 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.836482    5508 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:18:40.836720 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.836701    5508 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:18:40.894109 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.894083    5508 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 01:18:40.895156 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.895129    5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 01:18:40.895247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.895163    5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 01:18:40.895247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.895177    5508 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-255.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 01:18:40.895325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.895254    5508 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:40.928093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.928066    5508 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 01:18:40.984711 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.984657    5508 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"]
Nov 03 01:18:40.985314 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.985267    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal" event={"ID":"0eb27a029b0b668ca0885e8f9e1efac2","Type":"ContainerStarted","Data":"8b9657b6479991cce2f4f877eab66e4c1f9dd243cebbe76b19f493d26246f6c4"}
Nov 03 01:18:40.985430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.985325    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal" event={"ID":"0eb27a029b0b668ca0885e8f9e1efac2","Type":"ContainerStarted","Data":"fe46741258363e7b02e84970875d6fe7881a3c9c79580df4429238fde64e8890"}
Nov 03 01:18:40.985430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.985328    5508 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:40.985430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.985379    5508 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:40.985430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.985402    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal" event={"ID":"bd3c5a92e769172fde5847bb75e6fd4d","Type":"ContainerStarted","Data":"07afb357b6e0480360e728d76adc242e764fa276ec056dd07a2d57e4afd7b6c8"}
Nov 03 01:18:40.985430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.985415    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal" event={"ID":"bd3c5a92e769172fde5847bb75e6fd4d","Type":"ContainerDied","Data":"0bab3edfc0f815357e2d56c2a6bacd1b4a446e9d135e35231ae74b4f35a623ec"}
Nov 03 01:18:40.985430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:40.985428    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal" event={"ID":"bd3c5a92e769172fde5847bb75e6fd4d","Type":"ContainerStarted","Data":"78fe8905288581090afb36532f05784b20ebca0c055ed6c8e1074a9d6ff519be"}
Nov 03 01:18:41.028750 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.028690    5508 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.028831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.028767    5508 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.041809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.041780    5508 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 01:18:41.041939 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.041832    5508 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.042243 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.042228    5508 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 01:18:41.042281 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.042268    5508 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.138376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.138312    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd3c5a92e769172fde5847bb75e6fd4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"bd3c5a92e769172fde5847bb75e6fd4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.138545 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.138387    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd3c5a92e769172fde5847bb75e6fd4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"bd3c5a92e769172fde5847bb75e6fd4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.138545 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.138436    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0eb27a029b0b668ca0885e8f9e1efac2-config\") pod \"kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"0eb27a029b0b668ca0885e8f9e1efac2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.239099 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.239063    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd3c5a92e769172fde5847bb75e6fd4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"bd3c5a92e769172fde5847bb75e6fd4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.239099 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.239099    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd3c5a92e769172fde5847bb75e6fd4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"bd3c5a92e769172fde5847bb75e6fd4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.239261 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.239117    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0eb27a029b0b668ca0885e8f9e1efac2-config\") pod \"kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"0eb27a029b0b668ca0885e8f9e1efac2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.239261 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.239169    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0eb27a029b0b668ca0885e8f9e1efac2-config\") pod \"kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"0eb27a029b0b668ca0885e8f9e1efac2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.239261 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.239181    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd3c5a92e769172fde5847bb75e6fd4d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"bd3c5a92e769172fde5847bb75e6fd4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.239261 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.239181    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd3c5a92e769172fde5847bb75e6fd4d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" (UID: \"bd3c5a92e769172fde5847bb75e6fd4d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.635279 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.635240    5508 apiserver.go:52] "Watching apiserver"
Nov 03 01:18:41.731713 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.731678    5508 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 01:18:41.732343 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.732312    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal","openshift-multus/multus-46tlx","openshift-network-operator/iptables-alerter-7xkjx","kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal","openshift-console/console-65599784d8-h59hz","openshift-cluster-node-tuning-operator/tuned-vqm5v","openshift-ingress-canary/ingress-canary-rvpjl","openshift-multus/multus-additional-cni-plugins-qv8q5","openshift-multus/network-metrics-daemon-vxv8x","openshift-ovn-kubernetes/ovnkube-node-hr6kr","openshift-monitoring/metrics-server-67f9494f54-92n2g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj","openshift-dns/dns-default-bds75","openshift-monitoring/alertmanager-main-0","openshift-dns/node-resolver-287jt","openshift-image-registry/node-ca-8flnl","openshift-monitoring/node-exporter-m24b6","openshift-monitoring/prometheus-k8s-0","kube-system/global-pull-secret-syncer-b4htm","kube-system/konnectivity-agent-g4jmf","openshift-network-diagnostics/network-check-target-ttbvc","openshift-insights/insights-runtime-extractor-5pxx9"]
Nov 03 01:18:41.745875 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.745841    5508 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.746229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.746205    5508 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.751064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.751044    5508 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 01:18:41.782743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.782707    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:18:41.782990 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.782966    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 01:18:41.783093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.782968    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:18:41.783093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.782998    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 01:18:41.783093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783053    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 01:18:41.783093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.782982    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 01:18:41.783248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783092    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 01:18:41.783248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783094    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-th4bh\""
Nov 03 01:18:41.783248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783113    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bdhgr\""
Nov 03 01:18:41.783248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783094    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mmkcs\""
Nov 03 01:18:41.783248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783143    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 01:18:41.783248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783181    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 01:18:41.783248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783226    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2fc25\""
Nov 03 01:18:41.783248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783225    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 01:18:41.783602 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783300    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 01:18:41.783602 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783360    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 01:18:41.783602 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783484    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-9tz7d\""
Nov 03 01:18:41.783602 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783498    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 01:18:41.783602 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783527    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 01:18:41.783797 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783611    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 01:18:41.783797 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783662    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 01:18:41.783797 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783746    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 01:18:41.783893 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783798    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 01:18:41.783893 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783829    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 01:18:41.783893 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783852    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fqzf6\""
Nov 03 01:18:41.783893 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783891    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r4kvh\""
Nov 03 01:18:41.784007 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.783949    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7sd2v\""
Nov 03 01:18:41.784038 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.784022    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 01:18:41.784095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.784082    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nkj8v\""
Nov 03 01:18:41.784308 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.784295    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 01:18:41.784377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.784309    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 01:18:41.784490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.784475    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t8fl2\""
Nov 03 01:18:41.784490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.784485    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 01:18:41.788707 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.788691    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qwk4g\""
Nov 03 01:18:41.788770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.788698    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 01:18:41.788770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.788757    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 01:18:41.788845 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.788774    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 01:18:41.788845 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.788830    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tcxvw\""
Nov 03 01:18:41.788911 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.788847    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 01:18:41.788911 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.788881    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 01:18:41.789044 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.788942    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r69hg\""
Nov 03 01:18:41.789090 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.789044    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c92gz\""
Nov 03 01:18:41.789142 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.789128    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 01:18:41.789416 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.789401    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 01:18:41.789480 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.789417    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 01:18:41.789540 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.789528    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 01:18:41.789540 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.789534    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 01:18:41.789632 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.789558    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 01:18:41.789632 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.789579    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 01:18:41.789632 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.789588    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 01:18:41.789632 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.789584    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 01:18:41.826638 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.826610    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 01:18:41.826746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.826646    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 01:18:41.826746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.826656    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 01:18:41.826746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.826658    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 01:18:41.826746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.826659    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 01:18:41.826746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.826712    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 01:18:41.829573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.829550    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 01:18:41.829573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.829562    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 01:18:41.829723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.829560    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 01:18:41.829723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.829655    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 01:18:41.829723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.829660    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 01:18:41.829723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.829705    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9f8wp\""
Nov 03 01:18:41.829862 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.829729    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 01:18:41.829862 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.829736    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 01:18:41.829862 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.829779    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 01:18:41.830286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.830264    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-whk5z\""
Nov 03 01:18:41.830433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.830273    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 01:18:41.830433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.830309    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 01:18:41.830433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.830374    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-94q6h1no3itrh\""
Nov 03 01:18:41.830433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.830377    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 01:18:41.830433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.830431    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w7g4c\""
Nov 03 01:18:41.830714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.830477    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tnwmc\""
Nov 03 01:18:41.830714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.830573    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tp979\""
Nov 03 01:18:41.830800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.830782    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 01:18:41.834686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.834670    5508 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 01:18:41.834783 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.834711    5508 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.835262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.835248    5508 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 01:18:41.835313 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.835273    5508 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-255.us-west-2.compute.internal"
Nov 03 01:18:41.843567 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843542    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.843648 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843579    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-lib-modules\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.843648 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843599    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwx2t\" (UniqueName: \"kubernetes.io/projected/3d824491-3dcc-4c32-8378-7c8459b34086-kube-api-access-mwx2t\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:41.843648 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843621    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.843780 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843647    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.843780 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843685    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-client-ca-bundle\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.843780 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843755    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:41.843922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843791    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:41.843922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843821    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.843922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843849    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.843922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843851    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.843922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843872    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af477ca4-5745-4345-8e68-21ac9c32221f-tmp\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.843922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843902    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a4694-e358-4491-b4e1-c05bd477f19a-konnectivity-ca\") pod \"konnectivity-agent-g4jmf\" (UID: \"ee4a4694-e358-4491-b4e1-c05bd477f19a\") " pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:18:41.844184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843935    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.844184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843957    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-secret-metrics-server-tls\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.844184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843898    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-client-ca-bundle\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.844184 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.843976    5508 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 01:18:41.844184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843980    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df19de7c-3e77-48ce-b5e4-b4b56ab26017-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.844184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.843982    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.844184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844063    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d824491-3dcc-4c32-8378-7c8459b34086-metrics-certs\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:41.844184 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.844072    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:42.344049799 +0000 UTC m=+1.769656262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"service-ca" not registered
Nov 03 01:18:41.844184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844101    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.844184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844127    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ee4a4694-e358-4491-b4e1-c05bd477f19a-konnectivity-ca\") pod \"konnectivity-agent-g4jmf\" (UID: \"ee4a4694-e358-4491-b4e1-c05bd477f19a\") " pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:18:41.844184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844142    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844289    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6wp\" (UniqueName: \"kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844303    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df19de7c-3e77-48ce-b5e4-b4b56ab26017-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844310    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af477ca4-5745-4345-8e68-21ac9c32221f-tmp\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844327    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f57a8017-198c-42a9-9fb4-481348d8568c-host\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844300    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-secret-metrics-server-tls\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844368    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-netns\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844402    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-etc-kubernetes\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844422    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-var-lib-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844456    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-wtmp\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844488    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-data-volume\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844527    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-ovn\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844555    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-etc-selinux\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844579    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-config-out\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.844596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844556    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-data-volume\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844609    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-web-config\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844628    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844643    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844659    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4nvm\" (UniqueName: \"kubernetes.io/projected/e263af32-db0f-4984-8e90-83347cdd15f7-kube-api-access-x4nvm\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844661    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-config-out\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844675    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844729    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844751    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844766    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844784    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844808    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844833    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-tmp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844875    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844905    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844893    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-tmp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844929    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844955    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-run\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.844999    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-cni-bin\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845021    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-systemd-units\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845039    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845056    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845058    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845075    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845101    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysconfig\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845109    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.845161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845158    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-kubernetes\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.845177    5508 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845201    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-kubelet-config\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845219    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfhx7\" (UniqueName: \"kubernetes.io/projected/df19de7c-3e77-48ce-b5e4-b4b56ab26017-kube-api-access-pfhx7\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845242    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845258    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845273    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-k8s-cni-cncf-io\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.845293    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:42.345279876 +0000 UTC m=+1.770886318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-config" not registered
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845343    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-daemon-config\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845387    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-registration-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845403    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/571bdc94-d1ed-4ede-87fd-65af23ce24e9-tmp-dir\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845419    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8pqf\" (UniqueName: \"kubernetes.io/projected/da04dce2-a300-44f6-ab49-5fb77c6c8a19-kube-api-access-s8pqf\") pod \"ingress-canary-rvpjl\" (UID: \"da04dce2-a300-44f6-ab49-5fb77c6c8a19\") " pod="openshift-ingress-canary/ingress-canary-rvpjl"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845445    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-metrics-client-ca\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845468    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86vpg\" (UniqueName: \"kubernetes.io/projected/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-kube-api-access-86vpg\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845492    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845524    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845545    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-conf-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845569    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-textfile\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845596    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqwfz\" (UniqueName: \"kubernetes.io/projected/2c57dcf4-5aa3-4992-b357-47b388f3db90-kube-api-access-jqwfz\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.845603    5508 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845631    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/571bdc94-d1ed-4ede-87fd-65af23ce24e9-tmp-dir\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.845686    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:42.345675484 +0000 UTC m=+1.771281930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845696    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-textfile\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845703    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-secret-metrics-server-client-certs\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845731    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32d4bb3a-8abf-408a-8168-4581172f47f1-iptables-alerter-script\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845753    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-host\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845761    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845777    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af477ca4-5745-4345-8e68-21ac9c32221f-etc-tuned\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845879    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-daemon-config\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845903    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-kubelet\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845948    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-slash\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845963    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af477ca4-5745-4345-8e68-21ac9c32221f-etc-tuned\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845959    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df19de7c-3e77-48ce-b5e4-b4b56ab26017-secret-metrics-server-client-certs\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.845973    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-log-socket\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846011    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-sys\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846044    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846057    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32d4bb3a-8abf-408a-8168-4581172f47f1-iptables-alerter-script\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846078    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846096    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.846113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846132    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-socket-dir-parent\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846164    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/571bdc94-d1ed-4ede-87fd-65af23ce24e9-config-volume\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846192    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfvkd\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-kube-api-access-wfvkd\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846224    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df19de7c-3e77-48ce-b5e4-b4b56ab26017-audit-log\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846226    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846253    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846285    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-run-netns\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846299    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df19de7c-3e77-48ce-b5e4-b4b56ab26017-audit-log\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846311    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846372    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-env-overrides\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846381    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/571bdc94-d1ed-4ede-87fd-65af23ce24e9-config-volume\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846433    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-cni-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846460    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846511    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846538    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-env-overrides\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846545    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-cni-multus\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846572    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-multus-certs\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846599    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-ovnkube-script-lib\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846622    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846670    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846725    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/125c3968-dca4-4163-b473-b5cc7ac5a23f-hosts-file\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846744    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df19de7c-3e77-48ce-b5e4-b4b56ab26017-metrics-server-audit-profiles\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846761    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysctl-d\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846776    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f7f681b-d410-40e9-a4d0-487230d8dece-cni-binary-copy\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846793    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2rxv\" (UniqueName: \"kubernetes.io/projected/125c3968-dca4-4163-b473-b5cc7ac5a23f-kube-api-access-r2rxv\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846798    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-ovnkube-script-lib\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846819    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846846    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846875    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32d4bb3a-8abf-408a-8168-4581172f47f1-host-slash\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846919    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-sys\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846934    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846940    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-etc-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.846980    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-socket-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847008    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f7f681b-d410-40e9-a4d0-487230d8dece-cni-binary-copy\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847009    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee4a4694-e358-4491-b4e1-c05bd477f19a-agent-certs\") pod \"konnectivity-agent-g4jmf\" (UID: \"ee4a4694-e358-4491-b4e1-c05bd477f19a\") " pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847054    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571bdc94-d1ed-4ede-87fd-65af23ce24e9-metrics-tls\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847071    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da04dce2-a300-44f6-ab49-5fb77c6c8a19-cert\") pod \"ingress-canary-rvpjl\" (UID: \"da04dce2-a300-44f6-ab49-5fb77c6c8a19\") " pod="openshift-ingress-canary/ingress-canary-rvpjl"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847090    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-ovnkube-config\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847108    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847126    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gt7\" (UniqueName: \"kubernetes.io/projected/f57a8017-198c-42a9-9fb4-481348d8568c-kube-api-access-x4gt7\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847143    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-device-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847168    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-kubelet\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847190    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-dbus\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847218    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ee4a4694-e358-4491-b4e1-c05bd477f19a-agent-certs\") pod \"konnectivity-agent-g4jmf\" (UID: \"ee4a4694-e358-4491-b4e1-c05bd477f19a\") " pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.847229    5508 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847263    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/571bdc94-d1ed-4ede-87fd-65af23ce24e9-metrics-tls\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.847276    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:42.347265092 +0000 UTC m=+1.772871547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847288    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da04dce2-a300-44f6-ab49-5fb77c6c8a19-cert\") pod \"ingress-canary-rvpjl\" (UID: \"da04dce2-a300-44f6-ab49-5fb77c6c8a19\") " pod="openshift-ingress-canary/ingress-canary-rvpjl"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847301    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-web-config\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847336    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-627v6\" (UniqueName: \"kubernetes.io/projected/af477ca4-5745-4345-8e68-21ac9c32221f-kube-api-access-627v6\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847382    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847386    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e263af32-db0f-4984-8e90-83347cdd15f7-ovnkube-config\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847407    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/125c3968-dca4-4163-b473-b5cc7ac5a23f-tmp-dir\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847432    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847461    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-system-cni-dir\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.847467    5508 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847488    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp42j\" (UniqueName: \"kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j\") pod \"network-check-target-ttbvc\" (UID: \"5a8d8001-c1b0-4489-9e25-fb8b67048d7d\") " pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847493    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/125c3968-dca4-4163-b473-b5cc7ac5a23f-tmp-dir\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847514    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.847555    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:42.347540452 +0000 UTC m=+1.773146912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847576    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdcp\" (UniqueName: \"kubernetes.io/projected/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-kube-api-access-wmdcp\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847606    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847636    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-cni-binary-copy\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847672    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysctl-conf\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847697    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847701    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847718    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.847704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847745    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-accelerators-collector-config\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847775    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-systemd\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847801    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-var-lib-kubelet\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847852    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-system-cni-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847893    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-cnibin\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847922    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4hm8\" (UniqueName: \"kubernetes.io/projected/2f7f681b-d410-40e9-a4d0-487230d8dece-kube-api-access-s4hm8\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847939    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-original-pull-secret\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.847962    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848036    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-cni-binary-copy\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848056    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmkv6\" (UniqueName: \"kubernetes.io/projected/571bdc94-d1ed-4ede-87fd-65af23ce24e9-kube-api-access-zmkv6\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848121    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-root\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848144    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config-out\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848162    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84tzx\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-kube-api-access-84tzx\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848215    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz2vv\" (UniqueName: \"kubernetes.io/projected/32d4bb3a-8abf-408a-8168-4581172f47f1-kube-api-access-wz2vv\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848233    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-crio-socket\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848234    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-accelerators-collector-config\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848268    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f57a8017-198c-42a9-9fb4-481348d8568c-serviceca\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848253    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config-out\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848313    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-tls\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848339    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848376    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-systemd\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848403    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-node-log\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848427    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-cni-netd\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848476    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848505    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-cnibin\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848536    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848532    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-tls\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848554    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848569    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-hostroot\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848591    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-cni-bin\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848636    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-sys-fs\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848663    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-965xj\" (UniqueName: \"kubernetes.io/projected/3a492c60-d3a8-417b-b45f-254e82f2ae19-kube-api-access-965xj\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848679    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-modprobe-d\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848688    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c57dcf4-5aa3-4992-b357-47b388f3db90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848697    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e263af32-db0f-4984-8e90-83347cdd15f7-ovn-node-metrics-cert\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848901    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e263af32-db0f-4984-8e90-83347cdd15f7-ovn-node-metrics-cert\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.849999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.848916    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f57a8017-198c-42a9-9fb4-481348d8568c-serviceca\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:18:41.881642 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.881603    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 01:18:41.881790 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.881756    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 01:18:41.881790 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.881758    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 01:18:41.881912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.881802    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 01:18:41.881912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.881810    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 01:18:41.881912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.881847    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 01:18:41.881912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.881896    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 01:18:41.882072 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.882043    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 01:18:41.882072 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.882058    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 01:18:41.882190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.882058    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fsir5cbmlb4a0\""
Nov 03 01:18:41.882190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.882145    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 01:18:41.882190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.882151    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 01:18:41.882344 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.882327    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 01:18:41.882410 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.882378    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 01:18:41.882410 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.882388    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 01:18:41.882476 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.882462    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 01:18:41.882573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.882562    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 01:18:41.883903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.883889    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 01:18:41.884326 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.884310    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 01:18:41.884474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.884460    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.884874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.884859    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-web-config\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.884968 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.884954    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.885024 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.884978    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.885079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.885036    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.885258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.885228    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.885381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.885275    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.885454 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.885426    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.885494 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.885474    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.885494 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.885481    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.885558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.885532    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.886206 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.886192    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-metrics-client-ca\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.886319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.886304    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.886488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.886467    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.887451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.887403    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.887451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.887413    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.887600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.887497    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.887600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.887535    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-web-config\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.887600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.887548    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df19de7c-3e77-48ce-b5e4-b4b56ab26017-metrics-server-audit-profiles\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.888746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.888728    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.899375 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.899329    5508 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 01:18:41.899375 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.899365    5508 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 01:18:41.899496 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.899379    5508 projected.go:196] Error preparing data for projected volume kube-api-access-px6wp for pod openshift-console/console-65599784d8-h59hz: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 01:18:41.899496 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:41.899472    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:42.399452782 +0000 UTC m=+1.825059259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-px6wp" (UniqueName: "kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 01:18:41.899632 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.899614    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwx2t\" (UniqueName: \"kubernetes.io/projected/3d824491-3dcc-4c32-8378-7c8459b34086-kube-api-access-mwx2t\") pod \"network-metrics-daemon-vxv8x\" (UID: \"3d824491-3dcc-4c32-8378-7c8459b34086\") " pod="openshift-multus/network-metrics-daemon-vxv8x"
Nov 03 01:18:41.900776 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.900756    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqwfz\" (UniqueName: \"kubernetes.io/projected/2c57dcf4-5aa3-4992-b357-47b388f3db90-kube-api-access-jqwfz\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.900874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.900855    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86vpg\" (UniqueName: \"kubernetes.io/projected/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-kube-api-access-86vpg\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.900920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.900881    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfhx7\" (UniqueName: \"kubernetes.io/projected/df19de7c-3e77-48ce-b5e4-b4b56ab26017-kube-api-access-pfhx7\") pod \"metrics-server-67f9494f54-92n2g\" (UID: \"df19de7c-3e77-48ce-b5e4-b4b56ab26017\") " pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:18:41.900920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.900898    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8pqf\" (UniqueName: \"kubernetes.io/projected/da04dce2-a300-44f6-ab49-5fb77c6c8a19-kube-api-access-s8pqf\") pod \"ingress-canary-rvpjl\" (UID: \"da04dce2-a300-44f6-ab49-5fb77c6c8a19\") " pod="openshift-ingress-canary/ingress-canary-rvpjl"
Nov 03 01:18:41.901077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.901062    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4nvm\" (UniqueName: \"kubernetes.io/projected/e263af32-db0f-4984-8e90-83347cdd15f7-kube-api-access-x4nvm\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.903016 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.903001    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4gt7\" (UniqueName: \"kubernetes.io/projected/f57a8017-198c-42a9-9fb4-481348d8568c-kube-api-access-x4gt7\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:18:41.903610 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.903595    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2rxv\" (UniqueName: \"kubernetes.io/projected/125c3968-dca4-4163-b473-b5cc7ac5a23f-kube-api-access-r2rxv\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:18:41.904110 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.904095    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-965xj\" (UniqueName: \"kubernetes.io/projected/3a492c60-d3a8-417b-b45f-254e82f2ae19-kube-api-access-965xj\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.904570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.904556    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4hm8\" (UniqueName: \"kubernetes.io/projected/2f7f681b-d410-40e9-a4d0-487230d8dece-kube-api-access-s4hm8\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.904917 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.904903    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfvkd\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-kube-api-access-wfvkd\") pod \"alertmanager-main-0\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:18:41.905779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.905763    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmkv6\" (UniqueName: \"kubernetes.io/projected/571bdc94-d1ed-4ede-87fd-65af23ce24e9-kube-api-access-zmkv6\") pod \"dns-default-bds75\" (UID: \"571bdc94-d1ed-4ede-87fd-65af23ce24e9\") " pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:41.906146 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.906130    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84tzx\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-kube-api-access-84tzx\") pod \"prometheus-k8s-0\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:41.906545 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.906525    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdcp\" (UniqueName: \"kubernetes.io/projected/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-kube-api-access-wmdcp\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:41.906778 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.906764    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-627v6\" (UniqueName: \"kubernetes.io/projected/af477ca4-5745-4345-8e68-21ac9c32221f-kube-api-access-627v6\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.906933 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.906913    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz2vv\" (UniqueName: \"kubernetes.io/projected/32d4bb3a-8abf-408a-8168-4581172f47f1-kube-api-access-wz2vv\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:18:41.907088 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.907075    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp42j\" (UniqueName: \"kubernetes.io/projected/5a8d8001-c1b0-4489-9e25-fb8b67048d7d-kube-api-access-dp42j\") pod \"network-check-target-ttbvc\" (UID: \"5a8d8001-c1b0-4489-9e25-fb8b67048d7d\") " pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:41.949607 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949585    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.949696 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949621    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f57a8017-198c-42a9-9fb4-481348d8568c-host\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:18:41.949696 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949635    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-netns\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.949696 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949657    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-etc-kubernetes\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.949696 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949680    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-var-lib-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.949903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949702    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f57a8017-198c-42a9-9fb4-481348d8568c-host\") pod \"node-ca-8flnl\" (UID: \"f57a8017-198c-42a9-9fb4-481348d8568c\") " pod="openshift-image-registry/node-ca-8flnl"
Nov 03 01:18:41.949903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949704    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-wtmp\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.949903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949757    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-node-exporter-wtmp\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.949903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949766    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-var-lib-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.949903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949766    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.949903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949720    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-etc-kubernetes\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.949903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949720    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-netns\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.949903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949750    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-ovn\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.949903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949796    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-ovn\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.949903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949852    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-etc-selinux\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.949903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949900    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-etc-selinux\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949909    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949931    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-run\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949947    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-cni-bin\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949951    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.949964    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-systemd-units\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950024    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-cni-bin\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950025    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-run\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950029    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysconfig\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950057    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-systemd-units\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950067    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysconfig\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950088    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-kubernetes\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950106    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-kubelet-config\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950124    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-k8s-cni-cncf-io\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950145    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-registration-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950163    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-k8s-cni-cncf-io\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950179    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-kubernetes\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950178    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-kubelet-config\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950188    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-conf-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950202    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-registration-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950206    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-conf-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950216    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-host\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950250    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-kubelet\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.950258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950264    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-slash\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950280    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-log-socket\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950300    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-sys\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950306    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-kubelet\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950280    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-host\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950310    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-slash\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950332    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-sys\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950340    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-log-socket\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950391    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-socket-dir-parent\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950421    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-run-netns\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950437    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-cni-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950450    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-cni-multus\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950477    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-cni-multus\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950475    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-run-netns\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950503    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-multus-certs\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950511    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-cni-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950523    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/125c3968-dca4-4163-b473-b5cc7ac5a23f-hosts-file\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950503    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-multus-socket-dir-parent\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950544    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysctl-d\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950562    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-run-multus-certs\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950579    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32d4bb3a-8abf-408a-8168-4581172f47f1-host-slash\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950598    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-sys\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950602    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32d4bb3a-8abf-408a-8168-4581172f47f1-host-slash\") pod \"iptables-alerter-7xkjx\" (UID: \"32d4bb3a-8abf-408a-8168-4581172f47f1\") " pod="openshift-network-operator/iptables-alerter-7xkjx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950597    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/125c3968-dca4-4163-b473-b5cc7ac5a23f-hosts-file\") pod \"node-resolver-287jt\" (UID: \"125c3968-dca4-4163-b473-b5cc7ac5a23f\") " pod="openshift-dns/node-resolver-287jt"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950611    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-etc-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950630    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-etc-openvswitch\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950638    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-socket-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950647    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-sys\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950672    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysctl-d\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950683    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-device-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950691    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-socket-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950706    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-kubelet\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950722    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-dbus\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950730    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-device-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950751    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-system-cni-dir\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950780    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950792    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a5d501e-fa40-467b-a4fa-9fc52f9c45fa-dbus\") pod \"global-pull-secret-syncer-b4htm\" (UID: \"4a5d501e-fa40-467b-a4fa-9fc52f9c45fa\") " pod="kube-system/global-pull-secret-syncer-b4htm"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950807    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysctl-conf\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950813    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-system-cni-dir\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950783    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-host-var-lib-kubelet\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950834    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-systemd\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950830    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950855    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-var-lib-kubelet\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950875    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-systemd\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950876    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-system-cni-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950904    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-cnibin\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950915    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-system-cni-dir\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950925    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950918    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-var-lib-kubelet\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950939    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-sysctl-conf\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950943    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-cnibin\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950954    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-root\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950972    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-kubelet-dir\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950981    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-crio-socket\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.950991    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b7d8a03d-d25a-448a-955f-9b7ea3f4e07b-root\") pod \"node-exporter-m24b6\" (UID: \"b7d8a03d-d25a-448a-955f-9b7ea3f4e07b\") " pod="openshift-monitoring/node-exporter-m24b6"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951005    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-systemd\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951025    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-node-log\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.951014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951025    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-run-systemd\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951043    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-cni-netd\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951076    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/63489afa-8dc5-48b4-a0b6-2c40bdd8ff79-crio-socket\") pod \"insights-runtime-extractor-5pxx9\" (UID: \"63489afa-8dc5-48b4-a0b6-2c40bdd8ff79\") " pod="openshift-insights/insights-runtime-extractor-5pxx9"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951080    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951085    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-node-log\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951098    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-cni-netd\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951120    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951131    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-cnibin\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951147    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-hostroot\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951162    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-cni-bin\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951175    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-sys-fs\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951180    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c57dcf4-5aa3-4992-b357-47b388f3db90-cnibin\") pod \"multus-additional-cni-plugins-qv8q5\" (UID: \"2c57dcf4-5aa3-4992-b357-47b388f3db90\") " pod="openshift-multus/multus-additional-cni-plugins-qv8q5"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951190    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-modprobe-d\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951206    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-lib-modules\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951213    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e263af32-db0f-4984-8e90-83347cdd15f7-host-cni-bin\") pod \"ovnkube-node-hr6kr\" (UID: \"e263af32-db0f-4984-8e90-83347cdd15f7\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951222    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2f7f681b-d410-40e9-a4d0-487230d8dece-hostroot\") pod \"multus-46tlx\" (UID: \"2f7f681b-d410-40e9-a4d0-487230d8dece\") " pod="openshift-multus/multus-46tlx"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951238    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a492c60-d3a8-417b-b45f-254e82f2ae19-sys-fs\") pod \"aws-ebs-csi-driver-node-db7sj\" (UID: \"3a492c60-d3a8-417b-b45f-254e82f2ae19\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db7sj"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951318    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-lib-modules\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:41.952682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:41.951364    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af477ca4-5745-4345-8e68-21ac9c32221f-etc-modprobe-d\") pod \"tuned-vqm5v\" (UID: \"af477ca4-5745-4345-8e68-21ac9c32221f\") " pod="openshift-cluster-node-tuning-operator/tuned-vqm5v"
Nov 03 01:18:42.355070 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:42.355039    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:42.355218 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:42.355080    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:42.355218 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:42.355121    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:42.355218 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:42.355154    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:42.355218 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.355181    5508 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 01:18:42.355413 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.355245    5508 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 01:18:42.355413 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.355248    5508 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 01:18:42.355413 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.355257    5508 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 01:18:42.355413 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.355256    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:43.35523656 +0000 UTC m=+2.780843007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 01:18:42.355413 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:42.355192    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:42.355413 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.355319    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:43.355307175 +0000 UTC m=+2.780913618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 01:18:42.355413 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.355245    5508 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 01:18:42.355413 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.355378    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:43.355339042 +0000 UTC m=+2.780945495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"service-ca" not registered
Nov 03 01:18:42.355413 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.355402    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:43.355392953 +0000 UTC m=+2.780999399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-config" not registered
Nov 03 01:18:42.355664 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.355428    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:43.355410021 +0000 UTC m=+2.781016484 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 01:18:42.458139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:42.456691    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6wp\" (UniqueName: \"kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:42.458139 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.456912    5508 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 01:18:42.458139 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.456933    5508 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 01:18:42.458139 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.456946    5508 projected.go:196] Error preparing data for projected volume kube-api-access-px6wp for pod openshift-console/console-65599784d8-h59hz: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 01:18:42.458139 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:42.456998    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:43.45698283 +0000 UTC m=+2.882589275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-px6wp" (UniqueName: "kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 01:18:42.858080 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:42.858050    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bds75"
Nov 03 01:18:43.365451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:43.365415    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:43.365451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:43.365455    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:43.365720 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:43.365492    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:43.365720 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.365564    5508 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 01:18:43.365720 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.365574    5508 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 01:18:43.365720 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.365555    5508 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 01:18:43.365720 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.365640    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:45.365627084 +0000 UTC m=+4.791233527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 01:18:43.365720 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.365666    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:45.365649092 +0000 UTC m=+4.791255535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"service-ca" not registered
Nov 03 01:18:43.365720 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:43.365636    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:43.365720 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.365683    5508 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 01:18:43.365959 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.365685    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:45.365675889 +0000 UTC m=+4.791282332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 01:18:43.365959 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.365755    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:45.365745108 +0000 UTC m=+4.791351550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-config" not registered
Nov 03 01:18:43.365959 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:43.365784    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:43.365959 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.365833    5508 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 01:18:43.365959 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.365857    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:45.365850703 +0000 UTC m=+4.791457146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 01:18:43.466756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:43.466722    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6wp\" (UniqueName: \"kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:43.466904 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.466880    5508 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 01:18:43.466904 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.466903    5508 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 01:18:43.466980 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.466912    5508 projected.go:196] Error preparing data for projected volume kube-api-access-px6wp for pod openshift-console/console-65599784d8-h59hz: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 01:18:43.466980 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.466960    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:45.466946433 +0000 UTC m=+4.892552894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-px6wp" (UniqueName: "kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 01:18:43.563760 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:43.563730    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:18:43.812159 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:43.812124    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:18:45.251192 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:45.251159    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:45.389625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:45.389585    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:45.389799 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:45.389644    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:45.389799 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:45.389667    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:45.389799 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:45.389701    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:45.389799 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.389710    5508 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 01:18:45.389799 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.389767    5508 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 01:18:45.389799 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.389788    5508 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 01:18:45.389799 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.389789    5508 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 01:18:45.390015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:45.389721    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:45.390015 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.389791    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:49.389772582 +0000 UTC m=+8.815379041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"service-ca" not registered
Nov 03 01:18:45.390015 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.389851    5508 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 01:18:45.390015 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.389858    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:49.389844165 +0000 UTC m=+8.815450612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 01:18:45.390015 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.389912    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:49.389905808 +0000 UTC m=+8.815512251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 01:18:45.390015 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.389929    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:49.389922897 +0000 UTC m=+8.815529340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-config" not registered
Nov 03 01:18:45.390015 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.389944    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:49.389938967 +0000 UTC m=+8.815545409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 01:18:45.491066 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:45.491030    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6wp\" (UniqueName: \"kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:45.491212 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.491192    5508 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 01:18:45.491249 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.491215    5508 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 01:18:45.491249 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.491225    5508 projected.go:196] Error preparing data for projected volume kube-api-access-px6wp for pod openshift-console/console-65599784d8-h59hz: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 01:18:45.491312 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:45.491277    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:49.491264295 +0000 UTC m=+8.916870752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-px6wp" (UniqueName: "kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 01:18:45.839182 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:45.839105    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:45.839343 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:45.839306    5508 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 01:18:49.437850 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:49.437805    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:49.437868    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:49.437907    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.437965    5508 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.437984    5508 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.438003    5508 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.438017    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:57.43800515 +0000 UTC m=+16.863611606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-config" not registered
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:49.438066    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.438110    5508 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.438135    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:57.438120515 +0000 UTC m=+16.863726961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.438150    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:57.438143022 +0000 UTC m=+16.863749466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"service-ca" not registered
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:49.438164    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.438204    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:57.438190139 +0000 UTC m=+16.863796590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.438243    5508 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 01:18:49.438339 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.438271    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:57.438263333 +0000 UTC m=+16.863869816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 01:18:49.539191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:49.539153    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6wp\" (UniqueName: \"kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:49.539419 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.539312    5508 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 01:18:49.539419 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.539335    5508 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 01:18:49.539419 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.539363    5508 projected.go:196] Error preparing data for projected volume kube-api-access-px6wp for pod openshift-console/console-65599784d8-h59hz: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 01:18:49.539419 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:49.539413    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:57.539401023 +0000 UTC m=+16.965007465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-px6wp" (UniqueName: "kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 01:18:50.151947 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:50.151910    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:18:51.927554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:51.927522    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g4jmf"
Nov 03 01:18:52.284459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:52.284382    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr6kr"
Nov 03 01:18:57.513563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:57.513525    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:57.513582    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.513650    5508 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.513707    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:19:13.51368983 +0000 UTC m=+32.939296286 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-config" not registered
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.513712    5508 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.513762    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:19:13.513751241 +0000 UTC m=+32.939357696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"service-ca" not registered
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:57.513788    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:57.513846    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:57.513895    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.513912    5508 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.513968    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:19:13.513956676 +0000 UTC m=+32.939563119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"oauth-serving-cert" not registered
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.514003    5508 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.514006    5508 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.514041    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:19:13.514029818 +0000 UTC m=+32.939636272 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-oauth-config" not registered
Nov 03 01:18:57.514095 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.514054    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:19:13.514048651 +0000 UTC m=+32.939655097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : object "openshift-console"/"console-serving-cert" not registered
Nov 03 01:18:57.614559 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:18:57.614526    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6wp\" (UniqueName: \"kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp\") pod \"console-65599784d8-h59hz\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") " pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:18:57.614721 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.614686    5508 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Nov 03 01:18:57.614721 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.614707    5508 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Nov 03 01:18:57.614792 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.614724    5508 projected.go:196] Error preparing data for projected volume kube-api-access-px6wp for pod openshift-console/console-65599784d8-h59hz: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 01:18:57.614792 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:18:57.614780    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp podName:2a0e61d4-866a-4d9c-a885-8b232e7ba7c4 nodeName:}" failed. No retries permitted until 2026-11-03 01:19:13.614766647 +0000 UTC m=+33.040373094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-px6wp" (UniqueName: "kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp") pod "console-65599784d8-h59hz" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Nov 03 01:19:01.872529 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:01.872497    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-67f9494f54-92n2g"
Nov 03 01:19:06.761978 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:06.761923    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-65599784d8-h59hz" podUID="2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" containerName="console" containerID="cri-o://a3ac9b969ec40464f79a0fdfb528b0f1a1a0ee24a5109cd75a22aeb83cc3a7f6" gracePeriod=15
Nov 03 01:19:07.009664 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.009639    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:19:07.122049 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.121950    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca\") pod \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") "
Nov 03 01:19:07.122049 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.121996    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert\") pod \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") "
Nov 03 01:19:07.122049 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.122039    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-tmp\") pod \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") "
Nov 03 01:19:07.122317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.122071    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config\") pod \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") "
Nov 03 01:19:07.122516 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.122464    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca" pod "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:07.122699 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.122541    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp\" (UniqueName: \"kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp\") pod \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") "
Nov 03 01:19:07.122699 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.122649    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert\") pod \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") "
Nov 03 01:19:07.122837 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.122700    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config\") pod \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\" (UID: \"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4\") "
Nov 03 01:19:07.122907 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.122891    5508 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:07.123137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.123101    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-tmp" pod "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:19:07.123249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.123234    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config" pod "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:07.123488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.123464    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert" pod "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:07.124677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.124651    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config" pod "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:07.124865 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.124844    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert" pod "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:07.125026 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.125007    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp" pod "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" (UID: "2a0e61d4-866a-4d9c-a885-8b232e7ba7c4"). InnerVolumeSpecName "kube-api-access-px6wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:19:07.223577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.223552    5508 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-tmp\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:07.223577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.223576    5508 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-oauth-config\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:07.223718 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.223587    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px6wp\" (UniqueName: \"kubernetes.io/projected/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-kube-api-access-px6wp\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:07.223718 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.223597    5508 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-oauth-serving-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:07.223718 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.223608    5508 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-config\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:07.223718 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.223616    5508 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4-console-serving-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:07.827733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.827699    5508 generic.go:356] "Generic (PLEG): container finished" podID="2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" containerID="a3ac9b969ec40464f79a0fdfb528b0f1a1a0ee24a5109cd75a22aeb83cc3a7f6" exitCode=0
Nov 03 01:19:07.828190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.827741    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65599784d8-h59hz"
Nov 03 01:19:07.828190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.827774    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65599784d8-h59hz" event={"ID":"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4","Type":"ContainerDied","Data":"a3ac9b969ec40464f79a0fdfb528b0f1a1a0ee24a5109cd75a22aeb83cc3a7f6"}
Nov 03 01:19:07.828190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.827832    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65599784d8-h59hz" event={"ID":"2a0e61d4-866a-4d9c-a885-8b232e7ba7c4","Type":"ContainerDied","Data":"378fbd1c82140a4a08b20ef9e4bae339b4e720a2852ece2091adb3c0b7f2a9f7"}
Nov 03 01:19:07.828190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.827858    5508 scope.go:122] "RemoveContainer" containerID="a3ac9b969ec40464f79a0fdfb528b0f1a1a0ee24a5109cd75a22aeb83cc3a7f6"
Nov 03 01:19:07.836375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.836338    5508 scope.go:122] "RemoveContainer" containerID="a3ac9b969ec40464f79a0fdfb528b0f1a1a0ee24a5109cd75a22aeb83cc3a7f6"
Nov 03 01:19:07.836637 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:07.836620    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ac9b969ec40464f79a0fdfb528b0f1a1a0ee24a5109cd75a22aeb83cc3a7f6\": container with ID starting with a3ac9b969ec40464f79a0fdfb528b0f1a1a0ee24a5109cd75a22aeb83cc3a7f6 not found: ID does not exist" containerID="a3ac9b969ec40464f79a0fdfb528b0f1a1a0ee24a5109cd75a22aeb83cc3a7f6"
Nov 03 01:19:07.836684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.836647    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ac9b969ec40464f79a0fdfb528b0f1a1a0ee24a5109cd75a22aeb83cc3a7f6"} err="failed to get container status \"a3ac9b969ec40464f79a0fdfb528b0f1a1a0ee24a5109cd75a22aeb83cc3a7f6\": rpc error: code = NotFound desc = could not find container \"a3ac9b969ec40464f79a0fdfb528b0f1a1a0ee24a5109cd75a22aeb83cc3a7f6\": container with ID starting with a3ac9b969ec40464f79a0fdfb528b0f1a1a0ee24a5109cd75a22aeb83cc3a7f6 not found: ID does not exist"
Nov 03 01:19:07.979363 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:07.979308    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65599784d8-h59hz"]
Nov 03 01:19:08.025571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:08.025527    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65599784d8-h59hz"]
Nov 03 01:19:08.688632 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:08.688602    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" path="/var/lib/kubelet/pods/2a0e61d4-866a-4d9c-a885-8b232e7ba7c4/volumes"
Nov 03 01:19:10.154982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.154953    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ttbvc"
Nov 03 01:19:10.457889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.457786    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 01:19:10.458130 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.458103    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="alertmanager" containerID="cri-o://e62cfc33997f80e5b94efa81757d525e6d8659ef2fe2bf243892de12a3273b24" gracePeriod=120
Nov 03 01:19:10.458190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.458139    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="kube-rbac-proxy-metric" containerID="cri-o://547b58a4fd6c9f80ff3eed4793cab2dccb486153ecaa80af60d02579587c6ed5" gracePeriod=120
Nov 03 01:19:10.458190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.458170    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="kube-rbac-proxy-web" containerID="cri-o://b99934dab8545eb6799b349e221c084fd820bc7bad4588312519565be1f66955" gracePeriod=120
Nov 03 01:19:10.458308 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.458228    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="kube-rbac-proxy" containerID="cri-o://5100f6594561a64c0236ecccf5ec997ddffd41a353045cbe99752753db474db4" gracePeriod=120
Nov 03 01:19:10.458409 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.458309    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="config-reloader" containerID="cri-o://b2a51c8299dbebd40251d0f5c0887804f83031a0f2cd548b7fbcee729d8bc0d1" gracePeriod=120
Nov 03 01:19:10.458500 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.458448    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="prom-label-proxy" containerID="cri-o://ad308c4395e03f4f9176d4a62afa47ac4591f5acef95b857f19c93f7832cec87" gracePeriod=120
Nov 03 01:19:10.841319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.841237    5508 generic.go:356] "Generic (PLEG): container finished" podID="a804a1ea-8545-4932-a98b-fba23d40688b" containerID="ad308c4395e03f4f9176d4a62afa47ac4591f5acef95b857f19c93f7832cec87" exitCode=0
Nov 03 01:19:10.841319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.841259    5508 generic.go:356] "Generic (PLEG): container finished" podID="a804a1ea-8545-4932-a98b-fba23d40688b" containerID="5100f6594561a64c0236ecccf5ec997ddffd41a353045cbe99752753db474db4" exitCode=0
Nov 03 01:19:10.841319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.841267    5508 generic.go:356] "Generic (PLEG): container finished" podID="a804a1ea-8545-4932-a98b-fba23d40688b" containerID="b99934dab8545eb6799b349e221c084fd820bc7bad4588312519565be1f66955" exitCode=0
Nov 03 01:19:10.841319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.841273    5508 generic.go:356] "Generic (PLEG): container finished" podID="a804a1ea-8545-4932-a98b-fba23d40688b" containerID="b2a51c8299dbebd40251d0f5c0887804f83031a0f2cd548b7fbcee729d8bc0d1" exitCode=0
Nov 03 01:19:10.841319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.841279    5508 generic.go:356] "Generic (PLEG): container finished" podID="a804a1ea-8545-4932-a98b-fba23d40688b" containerID="e62cfc33997f80e5b94efa81757d525e6d8659ef2fe2bf243892de12a3273b24" exitCode=0
Nov 03 01:19:10.841649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.841315    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerDied","Data":"ad308c4395e03f4f9176d4a62afa47ac4591f5acef95b857f19c93f7832cec87"}
Nov 03 01:19:10.841649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.841426    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerDied","Data":"5100f6594561a64c0236ecccf5ec997ddffd41a353045cbe99752753db474db4"}
Nov 03 01:19:10.841649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.841456    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerDied","Data":"b99934dab8545eb6799b349e221c084fd820bc7bad4588312519565be1f66955"}
Nov 03 01:19:10.841649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.841481    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerDied","Data":"b2a51c8299dbebd40251d0f5c0887804f83031a0f2cd548b7fbcee729d8bc0d1"}
Nov 03 01:19:10.841649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:10.841506    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerDied","Data":"e62cfc33997f80e5b94efa81757d525e6d8659ef2fe2bf243892de12a3273b24"}
Nov 03 01:19:11.746627 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.746599    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:11.847943 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.847874    5508 generic.go:356] "Generic (PLEG): container finished" podID="a804a1ea-8545-4932-a98b-fba23d40688b" containerID="547b58a4fd6c9f80ff3eed4793cab2dccb486153ecaa80af60d02579587c6ed5" exitCode=0
Nov 03 01:19:11.848076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.847948    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:11.848076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.847950    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerDied","Data":"547b58a4fd6c9f80ff3eed4793cab2dccb486153ecaa80af60d02579587c6ed5"}
Nov 03 01:19:11.848076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.848010    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a804a1ea-8545-4932-a98b-fba23d40688b","Type":"ContainerDied","Data":"96657bb1ef573dd77f20f4de64ba4b4c5112a364f0daf3424978f71b7ff36c61"}
Nov 03 01:19:11.848076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.848041    5508 scope.go:122] "RemoveContainer" containerID="ad308c4395e03f4f9176d4a62afa47ac4591f5acef95b857f19c93f7832cec87"
Nov 03 01:19:11.855136 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.855120    5508 scope.go:122] "RemoveContainer" containerID="547b58a4fd6c9f80ff3eed4793cab2dccb486153ecaa80af60d02579587c6ed5"
Nov 03 01:19:11.861551 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.861534    5508 scope.go:122] "RemoveContainer" containerID="5100f6594561a64c0236ecccf5ec997ddffd41a353045cbe99752753db474db4"
Nov 03 01:19:11.867523 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.867509    5508 scope.go:122] "RemoveContainer" containerID="b99934dab8545eb6799b349e221c084fd820bc7bad4588312519565be1f66955"
Nov 03 01:19:11.868388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.868374    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-web-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-web-config\") pod \"a804a1ea-8545-4932-a98b-fba23d40688b\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") "
Nov 03 01:19:11.868446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.868399    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-trusted-ca-bundle\") pod \"a804a1ea-8545-4932-a98b-fba23d40688b\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") "
Nov 03 01:19:11.868446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.868427    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a804a1ea-8545-4932-a98b-fba23d40688b\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") "
Nov 03 01:19:11.868548 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.868456    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-metrics-client-ca\") pod \"a804a1ea-8545-4932-a98b-fba23d40688b\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") "
Nov 03 01:19:11.868605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.868557    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-config-out\") pod \"a804a1ea-8545-4932-a98b-fba23d40688b\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") "
Nov 03 01:19:11.868605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.868583    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-web\") pod \"a804a1ea-8545-4932-a98b-fba23d40688b\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") "
Nov 03 01:19:11.868715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.868621    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-config-volume\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-config-volume\") pod \"a804a1ea-8545-4932-a98b-fba23d40688b\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") "
Nov 03 01:19:11.868715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.868647    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-tls-assets\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-tls-assets\") pod \"a804a1ea-8545-4932-a98b-fba23d40688b\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") "
Nov 03 01:19:11.868715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.868700    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-cluster-tls-config\") pod \"a804a1ea-8545-4932-a98b-fba23d40688b\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") "
Nov 03 01:19:11.868815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.868727    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-main-tls\") pod \"a804a1ea-8545-4932-a98b-fba23d40688b\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") "
Nov 03 01:19:11.868815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.868756    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy\") pod \"a804a1ea-8545-4932-a98b-fba23d40688b\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") "
Nov 03 01:19:11.868815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.868774    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-main-db\") pod \"a804a1ea-8545-4932-a98b-fba23d40688b\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") "
Nov 03 01:19:11.868815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.868793    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-kube-api-access-wfvkd\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-kube-api-access-wfvkd\") pod \"a804a1ea-8545-4932-a98b-fba23d40688b\" (UID: \"a804a1ea-8545-4932-a98b-fba23d40688b\") "
Nov 03 01:19:11.869318 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.869284    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-trusted-ca-bundle" pod "a804a1ea-8545-4932-a98b-fba23d40688b" (UID: "a804a1ea-8545-4932-a98b-fba23d40688b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:11.869770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.869616    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-metrics-client-ca" pod "a804a1ea-8545-4932-a98b-fba23d40688b" (UID: "a804a1ea-8545-4932-a98b-fba23d40688b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:11.870236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.870188    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-main-db" pod "a804a1ea-8545-4932-a98b-fba23d40688b" (UID: "a804a1ea-8545-4932-a98b-fba23d40688b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:19:11.872228 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.872174    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy" pod "a804a1ea-8545-4932-a98b-fba23d40688b" (UID: "a804a1ea-8545-4932-a98b-fba23d40688b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:11.873106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.873070    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-config-volume" pod "a804a1ea-8545-4932-a98b-fba23d40688b" (UID: "a804a1ea-8545-4932-a98b-fba23d40688b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:11.873541 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.873512    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-tls-assets" pod "a804a1ea-8545-4932-a98b-fba23d40688b" (UID: "a804a1ea-8545-4932-a98b-fba23d40688b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:19:11.873710 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.873680    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-config-out" pod "a804a1ea-8545-4932-a98b-fba23d40688b" (UID: "a804a1ea-8545-4932-a98b-fba23d40688b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:19:11.874100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.874070    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-metric" pod "a804a1ea-8545-4932-a98b-fba23d40688b" (UID: "a804a1ea-8545-4932-a98b-fba23d40688b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:11.874200 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.874171    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-web" pod "a804a1ea-8545-4932-a98b-fba23d40688b" (UID: "a804a1ea-8545-4932-a98b-fba23d40688b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:11.874309 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.874268    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-main-tls" pod "a804a1ea-8545-4932-a98b-fba23d40688b" (UID: "a804a1ea-8545-4932-a98b-fba23d40688b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:11.874479 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.874360    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-kube-api-access-wfvkd" pod "a804a1ea-8545-4932-a98b-fba23d40688b" (UID: "a804a1ea-8545-4932-a98b-fba23d40688b"). InnerVolumeSpecName "kube-api-access-wfvkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:19:11.876721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.876673    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-cluster-tls-config" pod "a804a1ea-8545-4932-a98b-fba23d40688b" (UID: "a804a1ea-8545-4932-a98b-fba23d40688b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:11.881757 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.881729    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-web-config" pod "a804a1ea-8545-4932-a98b-fba23d40688b" (UID: "a804a1ea-8545-4932-a98b-fba23d40688b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:11.889716 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.889696    5508 scope.go:122] "RemoveContainer" containerID="b2a51c8299dbebd40251d0f5c0887804f83031a0f2cd548b7fbcee729d8bc0d1"
Nov 03 01:19:11.897239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.897220    5508 scope.go:122] "RemoveContainer" containerID="e62cfc33997f80e5b94efa81757d525e6d8659ef2fe2bf243892de12a3273b24"
Nov 03 01:19:11.903856 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.903841    5508 scope.go:122] "RemoveContainer" containerID="5d9c4f61324f1587b1cb3764bbedf414f734f3e32409c48ab3b04449dfe8a22c"
Nov 03 01:19:11.910040 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.910022    5508 scope.go:122] "RemoveContainer" containerID="ad308c4395e03f4f9176d4a62afa47ac4591f5acef95b857f19c93f7832cec87"
Nov 03 01:19:11.910305 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:11.910286    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad308c4395e03f4f9176d4a62afa47ac4591f5acef95b857f19c93f7832cec87\": container with ID starting with ad308c4395e03f4f9176d4a62afa47ac4591f5acef95b857f19c93f7832cec87 not found: ID does not exist" containerID="ad308c4395e03f4f9176d4a62afa47ac4591f5acef95b857f19c93f7832cec87"
Nov 03 01:19:11.910365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.910315    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad308c4395e03f4f9176d4a62afa47ac4591f5acef95b857f19c93f7832cec87"} err="failed to get container status \"ad308c4395e03f4f9176d4a62afa47ac4591f5acef95b857f19c93f7832cec87\": rpc error: code = NotFound desc = could not find container \"ad308c4395e03f4f9176d4a62afa47ac4591f5acef95b857f19c93f7832cec87\": container with ID starting with ad308c4395e03f4f9176d4a62afa47ac4591f5acef95b857f19c93f7832cec87 not found: ID does not exist"
Nov 03 01:19:11.910365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.910332    5508 scope.go:122] "RemoveContainer" containerID="547b58a4fd6c9f80ff3eed4793cab2dccb486153ecaa80af60d02579587c6ed5"
Nov 03 01:19:11.910597 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:11.910581    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547b58a4fd6c9f80ff3eed4793cab2dccb486153ecaa80af60d02579587c6ed5\": container with ID starting with 547b58a4fd6c9f80ff3eed4793cab2dccb486153ecaa80af60d02579587c6ed5 not found: ID does not exist" containerID="547b58a4fd6c9f80ff3eed4793cab2dccb486153ecaa80af60d02579587c6ed5"
Nov 03 01:19:11.910636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.910601    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547b58a4fd6c9f80ff3eed4793cab2dccb486153ecaa80af60d02579587c6ed5"} err="failed to get container status \"547b58a4fd6c9f80ff3eed4793cab2dccb486153ecaa80af60d02579587c6ed5\": rpc error: code = NotFound desc = could not find container \"547b58a4fd6c9f80ff3eed4793cab2dccb486153ecaa80af60d02579587c6ed5\": container with ID starting with 547b58a4fd6c9f80ff3eed4793cab2dccb486153ecaa80af60d02579587c6ed5 not found: ID does not exist"
Nov 03 01:19:11.910636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.910614    5508 scope.go:122] "RemoveContainer" containerID="5100f6594561a64c0236ecccf5ec997ddffd41a353045cbe99752753db474db4"
Nov 03 01:19:11.910794 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:11.910780    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5100f6594561a64c0236ecccf5ec997ddffd41a353045cbe99752753db474db4\": container with ID starting with 5100f6594561a64c0236ecccf5ec997ddffd41a353045cbe99752753db474db4 not found: ID does not exist" containerID="5100f6594561a64c0236ecccf5ec997ddffd41a353045cbe99752753db474db4"
Nov 03 01:19:11.910831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.910798    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5100f6594561a64c0236ecccf5ec997ddffd41a353045cbe99752753db474db4"} err="failed to get container status \"5100f6594561a64c0236ecccf5ec997ddffd41a353045cbe99752753db474db4\": rpc error: code = NotFound desc = could not find container \"5100f6594561a64c0236ecccf5ec997ddffd41a353045cbe99752753db474db4\": container with ID starting with 5100f6594561a64c0236ecccf5ec997ddffd41a353045cbe99752753db474db4 not found: ID does not exist"
Nov 03 01:19:11.910831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.910810    5508 scope.go:122] "RemoveContainer" containerID="b99934dab8545eb6799b349e221c084fd820bc7bad4588312519565be1f66955"
Nov 03 01:19:11.911037 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:11.911020    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99934dab8545eb6799b349e221c084fd820bc7bad4588312519565be1f66955\": container with ID starting with b99934dab8545eb6799b349e221c084fd820bc7bad4588312519565be1f66955 not found: ID does not exist" containerID="b99934dab8545eb6799b349e221c084fd820bc7bad4588312519565be1f66955"
Nov 03 01:19:11.911088 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.911042    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99934dab8545eb6799b349e221c084fd820bc7bad4588312519565be1f66955"} err="failed to get container status \"b99934dab8545eb6799b349e221c084fd820bc7bad4588312519565be1f66955\": rpc error: code = NotFound desc = could not find container \"b99934dab8545eb6799b349e221c084fd820bc7bad4588312519565be1f66955\": container with ID starting with b99934dab8545eb6799b349e221c084fd820bc7bad4588312519565be1f66955 not found: ID does not exist"
Nov 03 01:19:11.911088 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.911057    5508 scope.go:122] "RemoveContainer" containerID="b2a51c8299dbebd40251d0f5c0887804f83031a0f2cd548b7fbcee729d8bc0d1"
Nov 03 01:19:11.911274 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:11.911259    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2a51c8299dbebd40251d0f5c0887804f83031a0f2cd548b7fbcee729d8bc0d1\": container with ID starting with b2a51c8299dbebd40251d0f5c0887804f83031a0f2cd548b7fbcee729d8bc0d1 not found: ID does not exist" containerID="b2a51c8299dbebd40251d0f5c0887804f83031a0f2cd548b7fbcee729d8bc0d1"
Nov 03 01:19:11.911317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.911277    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2a51c8299dbebd40251d0f5c0887804f83031a0f2cd548b7fbcee729d8bc0d1"} err="failed to get container status \"b2a51c8299dbebd40251d0f5c0887804f83031a0f2cd548b7fbcee729d8bc0d1\": rpc error: code = NotFound desc = could not find container \"b2a51c8299dbebd40251d0f5c0887804f83031a0f2cd548b7fbcee729d8bc0d1\": container with ID starting with b2a51c8299dbebd40251d0f5c0887804f83031a0f2cd548b7fbcee729d8bc0d1 not found: ID does not exist"
Nov 03 01:19:11.911317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.911289    5508 scope.go:122] "RemoveContainer" containerID="e62cfc33997f80e5b94efa81757d525e6d8659ef2fe2bf243892de12a3273b24"
Nov 03 01:19:11.911509 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:11.911495    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e62cfc33997f80e5b94efa81757d525e6d8659ef2fe2bf243892de12a3273b24\": container with ID starting with e62cfc33997f80e5b94efa81757d525e6d8659ef2fe2bf243892de12a3273b24 not found: ID does not exist" containerID="e62cfc33997f80e5b94efa81757d525e6d8659ef2fe2bf243892de12a3273b24"
Nov 03 01:19:11.911558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.911511    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e62cfc33997f80e5b94efa81757d525e6d8659ef2fe2bf243892de12a3273b24"} err="failed to get container status \"e62cfc33997f80e5b94efa81757d525e6d8659ef2fe2bf243892de12a3273b24\": rpc error: code = NotFound desc = could not find container \"e62cfc33997f80e5b94efa81757d525e6d8659ef2fe2bf243892de12a3273b24\": container with ID starting with e62cfc33997f80e5b94efa81757d525e6d8659ef2fe2bf243892de12a3273b24 not found: ID does not exist"
Nov 03 01:19:11.911558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.911521    5508 scope.go:122] "RemoveContainer" containerID="5d9c4f61324f1587b1cb3764bbedf414f734f3e32409c48ab3b04449dfe8a22c"
Nov 03 01:19:11.911733 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:11.911717    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9c4f61324f1587b1cb3764bbedf414f734f3e32409c48ab3b04449dfe8a22c\": container with ID starting with 5d9c4f61324f1587b1cb3764bbedf414f734f3e32409c48ab3b04449dfe8a22c not found: ID does not exist" containerID="5d9c4f61324f1587b1cb3764bbedf414f734f3e32409c48ab3b04449dfe8a22c"
Nov 03 01:19:11.911772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.911735    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9c4f61324f1587b1cb3764bbedf414f734f3e32409c48ab3b04449dfe8a22c"} err="failed to get container status \"5d9c4f61324f1587b1cb3764bbedf414f734f3e32409c48ab3b04449dfe8a22c\": rpc error: code = NotFound desc = could not find container \"5d9c4f61324f1587b1cb3764bbedf414f734f3e32409c48ab3b04449dfe8a22c\": container with ID starting with 5d9c4f61324f1587b1cb3764bbedf414f734f3e32409c48ab3b04449dfe8a22c not found: ID does not exist"
Nov 03 01:19:11.970055 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.970029    5508 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-cluster-tls-config\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:11.970055 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.970054    5508 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-main-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:11.970198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.970066    5508 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:11.970198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.970076    5508 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-main-db\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:11.970198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.970085    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfvkd\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-kube-api-access-wfvkd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:11.970198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.970095    5508 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-web-config\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:11.970198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.970109    5508 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:11.970198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.970122    5508 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:11.970198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.970132    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a804a1ea-8545-4932-a98b-fba23d40688b-metrics-client-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:11.970198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.970140    5508 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a804a1ea-8545-4932-a98b-fba23d40688b-config-out\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:11.970198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.970150    5508 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:11.970198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.970159    5508 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a804a1ea-8545-4932-a98b-fba23d40688b-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:11.970198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:11.970167    5508 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a804a1ea-8545-4932-a98b-fba23d40688b-tls-assets\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:12.264857 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.264819    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 01:19:12.268708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.268686    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 01:19:12.298696 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.298665    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 01:19:12.298962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.298948    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="prom-label-proxy"
Nov 03 01:19:12.299005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.298964    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="prom-label-proxy"
Nov 03 01:19:12.299005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.298976    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="kube-rbac-proxy"
Nov 03 01:19:12.299005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.298981    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="kube-rbac-proxy"
Nov 03 01:19:12.299005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.298990    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="kube-rbac-proxy-web"
Nov 03 01:19:12.299005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.298995    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="kube-rbac-proxy-web"
Nov 03 01:19:12.299005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299001    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="config-reloader"
Nov 03 01:19:12.299005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299006    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="config-reloader"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299012    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="alertmanager"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299017    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="alertmanager"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299024    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="kube-rbac-proxy-metric"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299028    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="kube-rbac-proxy-metric"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299034    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="init-config-reloader"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299039    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="init-config-reloader"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299056    5508 cpu_manager.go:414] "containerMap: removing container" podUID="2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" containerName="console"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299061    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" containerName="console"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299105    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="alertmanager"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299113    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="kube-rbac-proxy-metric"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299118    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="init-config-reloader"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299125    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0e61d4-866a-4d9c-a885-8b232e7ba7c4" containerName="console"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299132    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="prom-label-proxy"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299137    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="kube-rbac-proxy"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299141    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="kube-rbac-proxy-web"
Nov 03 01:19:12.299204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.299145    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" containerName="config-reloader"
Nov 03 01:19:12.333175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.333146    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.373553 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.373519    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-config-volume\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.373742 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.373569    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.373742 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.373621    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84681513-1a85-478c-8ba7-b2e37090dbc0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.373742 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.373661    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.373742 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.373741    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84681513-1a85-478c-8ba7-b2e37090dbc0-config-out\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.373928 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.373760    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.373928 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.373782    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.373928 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.373806    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84681513-1a85-478c-8ba7-b2e37090dbc0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.374019 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.373953    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22xn8\" (UniqueName: \"kubernetes.io/projected/84681513-1a85-478c-8ba7-b2e37090dbc0-kube-api-access-22xn8\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.374019 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.373980    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84681513-1a85-478c-8ba7-b2e37090dbc0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.374019 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.373999    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84681513-1a85-478c-8ba7-b2e37090dbc0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.374125 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.374024    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-web-config\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.374125 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.374082    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.382430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.382313    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 01:19:12.382430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.382313    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 01:19:12.382430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.382374    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 01:19:12.382430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.382374    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 01:19:12.382430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.382401    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 01:19:12.382430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.382402    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 01:19:12.382430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.382393    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 01:19:12.382722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.382402    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 01:19:12.382722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.382455    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tnwmc\""
Nov 03 01:19:12.383325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.383309    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 01:19:12.429039 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.428977    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 01:19:12.474977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.474950    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.474977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.474980    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.475139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.475006    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84681513-1a85-478c-8ba7-b2e37090dbc0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.475139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.475045    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22xn8\" (UniqueName: \"kubernetes.io/projected/84681513-1a85-478c-8ba7-b2e37090dbc0-kube-api-access-22xn8\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.475139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.475073    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84681513-1a85-478c-8ba7-b2e37090dbc0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.475139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.475107    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84681513-1a85-478c-8ba7-b2e37090dbc0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.475139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.475131    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-web-config\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.475407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.475162    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.475407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.475195    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-config-volume\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.475407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.475229    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.475407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.475252    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84681513-1a85-478c-8ba7-b2e37090dbc0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.475407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.475280    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.475407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.475318    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84681513-1a85-478c-8ba7-b2e37090dbc0-config-out\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.475691 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.475394    5508 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 01:19:12.475768 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.475741    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84681513-1a85-478c-8ba7-b2e37090dbc0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.476218 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.476001    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84681513-1a85-478c-8ba7-b2e37090dbc0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.476218 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.476046    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84681513-1a85-478c-8ba7-b2e37090dbc0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.478287 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.478264    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.478407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.478285    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-config-volume\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.478407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.478386    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84681513-1a85-478c-8ba7-b2e37090dbc0-config-out\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.478533 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.478436    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-web-config\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.478664 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.478646    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.478710 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.478692    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.478847 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.478827    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84681513-1a85-478c-8ba7-b2e37090dbc0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.479289 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.479269    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.479711 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.479693    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84681513-1a85-478c-8ba7-b2e37090dbc0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.531900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.531871    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22xn8\" (UniqueName: \"kubernetes.io/projected/84681513-1a85-478c-8ba7-b2e37090dbc0-kube-api-access-22xn8\") pod \"alertmanager-main-0\" (UID: \"84681513-1a85-478c-8ba7-b2e37090dbc0\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.654864 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.654839    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 01:19:12.688870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:12.688801    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a804a1ea-8545-4932-a98b-fba23d40688b" path="/var/lib/kubelet/pods/a804a1ea-8545-4932-a98b-fba23d40688b/volumes"
Nov 03 01:19:13.081700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:13.081613    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 01:19:13.089545 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:19:13.089514    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84681513_1a85_478c_8ba7_b2e37090dbc0.slice/crio-b116ca905c4239ac9f091e0c5d2964d0c4679feb7db43a4bb0a5837287e25110 WatchSource:0}: Error finding container b116ca905c4239ac9f091e0c5d2964d0c4679feb7db43a4bb0a5837287e25110: Status 404 returned error can't find the container with id b116ca905c4239ac9f091e0c5d2964d0c4679feb7db43a4bb0a5837287e25110
Nov 03 01:19:13.810672 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:13.810633    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:19:13.857603 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:13.857571    5508 generic.go:356] "Generic (PLEG): container finished" podID="84681513-1a85-478c-8ba7-b2e37090dbc0" containerID="e31783d514efede91d8667eba4ea2597f8eb5135b56eaed821c1dbcefc3d184d" exitCode=0
Nov 03 01:19:13.857734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:13.857621    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84681513-1a85-478c-8ba7-b2e37090dbc0","Type":"ContainerDied","Data":"e31783d514efede91d8667eba4ea2597f8eb5135b56eaed821c1dbcefc3d184d"}
Nov 03 01:19:13.857734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:13.857676    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84681513-1a85-478c-8ba7-b2e37090dbc0","Type":"ContainerStarted","Data":"b116ca905c4239ac9f091e0c5d2964d0c4679feb7db43a4bb0a5837287e25110"}
Nov 03 01:19:14.214892 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.214849    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-dd6c498f-58kkc"]
Nov 03 01:19:14.218327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.218301    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.267443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.267407    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 01:19:14.267704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.267584    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 01:19:14.267704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.267598    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 01:19:14.267704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.267612    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 01:19:14.267704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.267598    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 01:19:14.267704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.267637    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fvdmd\""
Nov 03 01:19:14.290550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.290513    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/502e7655-ead5-4ace-a420-cf4a546bcfc7-metrics-client-ca\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.290646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.290560    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/502e7655-ead5-4ace-a420-cf4a546bcfc7-serving-certs-ca-bundle\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.290646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.290578    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/502e7655-ead5-4ace-a420-cf4a546bcfc7-federate-client-tls\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.290741 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.290716    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/502e7655-ead5-4ace-a420-cf4a546bcfc7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.290779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.290752    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/502e7655-ead5-4ace-a420-cf4a546bcfc7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.290809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.290783    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn7ds\" (UniqueName: \"kubernetes.io/projected/502e7655-ead5-4ace-a420-cf4a546bcfc7-kube-api-access-rn7ds\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.290841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.290828    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/502e7655-ead5-4ace-a420-cf4a546bcfc7-secret-telemeter-client\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.290874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.290847    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/502e7655-ead5-4ace-a420-cf4a546bcfc7-telemeter-client-tls\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.314411 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.314328    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 01:19:14.317148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.317109    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-dd6c498f-58kkc"]
Nov 03 01:19:14.391358 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.391303    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/502e7655-ead5-4ace-a420-cf4a546bcfc7-secret-telemeter-client\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.391358 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.391340    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/502e7655-ead5-4ace-a420-cf4a546bcfc7-telemeter-client-tls\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.391619 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.391411    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/502e7655-ead5-4ace-a420-cf4a546bcfc7-metrics-client-ca\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.391619 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.391441    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/502e7655-ead5-4ace-a420-cf4a546bcfc7-serving-certs-ca-bundle\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.391619 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.391460    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/502e7655-ead5-4ace-a420-cf4a546bcfc7-federate-client-tls\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.391619 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.391605    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/502e7655-ead5-4ace-a420-cf4a546bcfc7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.391828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.391635    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/502e7655-ead5-4ace-a420-cf4a546bcfc7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.391828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.391689    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn7ds\" (UniqueName: \"kubernetes.io/projected/502e7655-ead5-4ace-a420-cf4a546bcfc7-kube-api-access-rn7ds\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.392408 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.392326    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/502e7655-ead5-4ace-a420-cf4a546bcfc7-metrics-client-ca\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.392408 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.392342    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/502e7655-ead5-4ace-a420-cf4a546bcfc7-serving-certs-ca-bundle\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.392902 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.392879    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/502e7655-ead5-4ace-a420-cf4a546bcfc7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.394039 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.394017    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/502e7655-ead5-4ace-a420-cf4a546bcfc7-federate-client-tls\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.394126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.394109    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/502e7655-ead5-4ace-a420-cf4a546bcfc7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.394321 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.394303    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/502e7655-ead5-4ace-a420-cf4a546bcfc7-secret-telemeter-client\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.394373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.394332    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/502e7655-ead5-4ace-a420-cf4a546bcfc7-telemeter-client-tls\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.449294 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.449257    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn7ds\" (UniqueName: \"kubernetes.io/projected/502e7655-ead5-4ace-a420-cf4a546bcfc7-kube-api-access-rn7ds\") pod \"telemeter-client-dd6c498f-58kkc\" (UID: \"502e7655-ead5-4ace-a420-cf4a546bcfc7\") " pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.534222 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.534186    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc"
Nov 03 01:19:14.864451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.864336    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84681513-1a85-478c-8ba7-b2e37090dbc0","Type":"ContainerStarted","Data":"3f9561ae089d8bf09892ee404ffd1181a8c08c0f90975465a137e136d0ca6f8c"}
Nov 03 01:19:14.864451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.864433    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84681513-1a85-478c-8ba7-b2e37090dbc0","Type":"ContainerStarted","Data":"0b4a6a25a1751f508a1c84f754a0fa42f45763c456a61db267986c915ff20d1f"}
Nov 03 01:19:14.864674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.864475    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84681513-1a85-478c-8ba7-b2e37090dbc0","Type":"ContainerStarted","Data":"01d2495c87290e4a576d4a916bc4f3542c362a9c1eb27fe98fe651c1b3c492ea"}
Nov 03 01:19:14.864674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.864509    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84681513-1a85-478c-8ba7-b2e37090dbc0","Type":"ContainerStarted","Data":"63e192ff52d7ab0a5e80e4b28c00cd58303ab8f0cd694389c8864532bf0fa203"}
Nov 03 01:19:14.864674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.864540    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84681513-1a85-478c-8ba7-b2e37090dbc0","Type":"ContainerStarted","Data":"aea5a06d3ef3cfb5cd6f3d30c2e11b55fde9db00f91c97e6c7efba7e083037ed"}
Nov 03 01:19:14.864674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.864575    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84681513-1a85-478c-8ba7-b2e37090dbc0","Type":"ContainerStarted","Data":"a5b4520a7d4fc6635829a29e2738f8a770b7f4b275ee85fc75fa1781c8fc9747"}
Nov 03 01:19:14.959784 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.959742    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-dd6c498f-58kkc"]
Nov 03 01:19:14.969044 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:19:14.969014    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502e7655_ead5_4ace_a420_cf4a546bcfc7.slice/crio-ffa68015d43515fddd7725732cbe7a3089afc9640c5e9deded0251a81495593d WatchSource:0}: Error finding container ffa68015d43515fddd7725732cbe7a3089afc9640c5e9deded0251a81495593d: Status 404 returned error can't find the container with id ffa68015d43515fddd7725732cbe7a3089afc9640c5e9deded0251a81495593d
Nov 03 01:19:14.971698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.971680    5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:19:14.987070 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:14.987008    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.975414337 podStartE2EDuration="2.986994843s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.580506ms" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:14.986032093 +0000 UTC m=+34.411638587" watchObservedRunningTime="2026-11-03 01:19:14.986994843 +0000 UTC m=+34.412601305"
Nov 03 01:19:15.868680 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:15.868644    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc" event={"ID":"502e7655-ead5-4ace-a420-cf4a546bcfc7","Type":"ContainerStarted","Data":"ffa68015d43515fddd7725732cbe7a3089afc9640c5e9deded0251a81495593d"}
Nov 03 01:19:16.333128 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.333076    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 01:19:16.333679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.333480    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="prometheus" containerID="cri-o://321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c" gracePeriod=600
Nov 03 01:19:16.333679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.333545    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="config-reloader" containerID="cri-o://267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba" gracePeriod=600
Nov 03 01:19:16.333679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.333543    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037" gracePeriod=600
Nov 03 01:19:16.333679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.333587    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="thanos-sidecar" containerID="cri-o://511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff" gracePeriod=600
Nov 03 01:19:16.333679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.333488    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="kube-rbac-proxy" containerID="cri-o://85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71" gracePeriod=600
Nov 03 01:19:16.333970 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.333549    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="kube-rbac-proxy-web" containerID="cri-o://d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1" gracePeriod=600
Nov 03 01:19:16.592409 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.592335    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:16.712256 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712220    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-0\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712270    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-1\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712317    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-web-config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-web-config\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712361    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712388    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-2\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712422    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-metrics-client-certs\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712453    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-kubelet-serving-ca-bundle\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712764 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712483    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-grpc-tls\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712764 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712528    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-tls-assets\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-tls-assets\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712764 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712558    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-thanos-prometheus-http-client-file\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712764 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712588    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-metrics-client-ca\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712764 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712635    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-db\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712764 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712663    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-tls\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712764 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712701    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config-out\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.712764 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712751    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-serving-certs-ca-bundle\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.713135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712743    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-1" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:16.713135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712795    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.713135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712826    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-kube-api-access-84tzx\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-kube-api-access-84tzx\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.713135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712858    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.713135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712888    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-kube-rbac-proxy\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.713135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712895    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-kubelet-serving-ca-bundle" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:16.713135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.712926    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-trusted-ca-bundle\") pod \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\" (UID: \"bf8fe558-c1cf-4f0c-96d0-e930abeb857b\") "
Nov 03 01:19:16.713135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.713092    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-2" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:16.713135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.713116    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-metrics-client-ca" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:16.713595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.713208    5508 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.713595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.713225    5508 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.713595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.713244    5508 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.713595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.713266    5508 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-metrics-client-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.713595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.713532    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-trusted-ca-bundle" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:16.715093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.714136    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-serving-certs-ca-bundle" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:16.715093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.714297    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-0" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:19:16.717597 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.717567    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-grpc-tls" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:16.717857 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.717831    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-thanos-prometheus-http-client-file" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:16.718293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.718271    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-db" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:19:16.718485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.718443    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-metrics-client-certs" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:16.719042 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.719017    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:16.719541 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.719505    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config-out" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:19:16.720429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.720397    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-kube-rbac-proxy-web" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:16.720620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.720590    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-tls-assets" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:19:16.722213 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.722185    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-kube-api-access-84tzx" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "kube-api-access-84tzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:19:16.722608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.722580    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-kube-rbac-proxy" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:16.722693 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.722659    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-thanos-sidecar-tls" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:16.723479 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.721633    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-tls" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:16.732922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.732887    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-web-config" pod "bf8fe558-c1cf-4f0c-96d0-e930abeb857b" (UID: "bf8fe558-c1cf-4f0c-96d0-e930abeb857b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:16.814382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814327    5508 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814382    5508 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-web-config\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814395    5508 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814405    5508 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-metrics-client-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814415    5508 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-grpc-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814424    5508 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-tls-assets\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814432    5508 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814441    5508 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-k8s-db\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814450    5508 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814458    5508 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config-out\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814469    5508 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814477    5508 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-config\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814490    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84tzx\" (UniqueName: \"kubernetes.io/projected/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-kube-api-access-84tzx\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814501    5508 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814510    5508 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-secret-kube-rbac-proxy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.814594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.814519    5508 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf8fe558-c1cf-4f0c-96d0-e930abeb857b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:19:16.876199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876116    5508 generic.go:356] "Generic (PLEG): container finished" podID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerID="0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037" exitCode=0
Nov 03 01:19:16.876199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876149    5508 generic.go:356] "Generic (PLEG): container finished" podID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerID="85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71" exitCode=0
Nov 03 01:19:16.876199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876165    5508 generic.go:356] "Generic (PLEG): container finished" podID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerID="d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1" exitCode=0
Nov 03 01:19:16.876199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876174    5508 generic.go:356] "Generic (PLEG): container finished" podID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerID="511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff" exitCode=0
Nov 03 01:19:16.876199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876185    5508 generic.go:356] "Generic (PLEG): container finished" podID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerID="267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba" exitCode=0
Nov 03 01:19:16.876199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876193    5508 generic.go:356] "Generic (PLEG): container finished" podID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerID="321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c" exitCode=0
Nov 03 01:19:16.876807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876201    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerDied","Data":"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"}
Nov 03 01:19:16.876807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876230    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:16.876807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876294    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerDied","Data":"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"}
Nov 03 01:19:16.876807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876342    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerDied","Data":"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"}
Nov 03 01:19:16.876807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876377    5508 scope.go:122] "RemoveContainer" containerID="0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"
Nov 03 01:19:16.876807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876407    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerDied","Data":"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"}
Nov 03 01:19:16.876807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876448    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerDied","Data":"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"}
Nov 03 01:19:16.876807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876492    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerDied","Data":"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"}
Nov 03 01:19:16.876807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.876530    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bf8fe558-c1cf-4f0c-96d0-e930abeb857b","Type":"ContainerDied","Data":"b423824438a1755dab77a92ade9560b77f48bdca244bbf5fd64c0af4154268e8"}
Nov 03 01:19:16.886073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.886050    5508 scope.go:122] "RemoveContainer" containerID="85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"
Nov 03 01:19:16.894497 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.894471    5508 scope.go:122] "RemoveContainer" containerID="d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"
Nov 03 01:19:16.902388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.902362    5508 scope.go:122] "RemoveContainer" containerID="511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"
Nov 03 01:19:16.911328 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.911305    5508 scope.go:122] "RemoveContainer" containerID="267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"
Nov 03 01:19:16.919214 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.919191    5508 scope.go:122] "RemoveContainer" containerID="321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"
Nov 03 01:19:16.928782 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.928760    5508 scope.go:122] "RemoveContainer" containerID="b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"
Nov 03 01:19:16.937633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.937609    5508 scope.go:122] "RemoveContainer" containerID="0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"
Nov 03 01:19:16.937970 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:16.937946    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037\": container with ID starting with 0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037 not found: ID does not exist" containerID="0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"
Nov 03 01:19:16.938068 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.937982    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"} err="failed to get container status \"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037\": rpc error: code = NotFound desc = could not find container \"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037\": container with ID starting with 0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037 not found: ID does not exist"
Nov 03 01:19:16.938068 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.938010    5508 scope.go:122] "RemoveContainer" containerID="85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"
Nov 03 01:19:16.938333 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:16.938311    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71\": container with ID starting with 85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71 not found: ID does not exist" containerID="85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"
Nov 03 01:19:16.938492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.938344    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"} err="failed to get container status \"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71\": rpc error: code = NotFound desc = could not find container \"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71\": container with ID starting with 85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71 not found: ID does not exist"
Nov 03 01:19:16.938492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.938384    5508 scope.go:122] "RemoveContainer" containerID="d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"
Nov 03 01:19:16.938784 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:16.938750    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1\": container with ID starting with d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1 not found: ID does not exist" containerID="d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"
Nov 03 01:19:16.938844 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.938782    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"} err="failed to get container status \"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1\": rpc error: code = NotFound desc = could not find container \"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1\": container with ID starting with d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1 not found: ID does not exist"
Nov 03 01:19:16.938844 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.938803    5508 scope.go:122] "RemoveContainer" containerID="511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"
Nov 03 01:19:16.939101 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:16.939076    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff\": container with ID starting with 511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff not found: ID does not exist" containerID="511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"
Nov 03 01:19:16.939181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.939106    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"} err="failed to get container status \"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff\": rpc error: code = NotFound desc = could not find container \"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff\": container with ID starting with 511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff not found: ID does not exist"
Nov 03 01:19:16.939181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.939125    5508 scope.go:122] "RemoveContainer" containerID="267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"
Nov 03 01:19:16.939449 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:16.939429    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba\": container with ID starting with 267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba not found: ID does not exist" containerID="267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"
Nov 03 01:19:16.939508 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.939455    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"} err="failed to get container status \"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba\": rpc error: code = NotFound desc = could not find container \"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba\": container with ID starting with 267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba not found: ID does not exist"
Nov 03 01:19:16.939508 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.939472    5508 scope.go:122] "RemoveContainer" containerID="321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"
Nov 03 01:19:16.939742 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:16.939718    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c\": container with ID starting with 321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c not found: ID does not exist" containerID="321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"
Nov 03 01:19:16.939812 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.939749    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"} err="failed to get container status \"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c\": rpc error: code = NotFound desc = could not find container \"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c\": container with ID starting with 321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c not found: ID does not exist"
Nov 03 01:19:16.939812 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.939770    5508 scope.go:122] "RemoveContainer" containerID="b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"
Nov 03 01:19:16.940037 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:16.940017    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4\": container with ID starting with b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4 not found: ID does not exist" containerID="b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"
Nov 03 01:19:16.940077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.940042    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"} err="failed to get container status \"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4\": rpc error: code = NotFound desc = could not find container \"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4\": container with ID starting with b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4 not found: ID does not exist"
Nov 03 01:19:16.940077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.940060    5508 scope.go:122] "RemoveContainer" containerID="0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"
Nov 03 01:19:16.940385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.940364    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"} err="failed to get container status \"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037\": rpc error: code = NotFound desc = could not find container \"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037\": container with ID starting with 0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037 not found: ID does not exist"
Nov 03 01:19:16.940448 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.940385    5508 scope.go:122] "RemoveContainer" containerID="85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"
Nov 03 01:19:16.940629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.940610    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"} err="failed to get container status \"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71\": rpc error: code = NotFound desc = could not find container \"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71\": container with ID starting with 85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71 not found: ID does not exist"
Nov 03 01:19:16.940675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.940631    5508 scope.go:122] "RemoveContainer" containerID="d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"
Nov 03 01:19:16.940879 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.940857    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"} err="failed to get container status \"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1\": rpc error: code = NotFound desc = could not find container \"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1\": container with ID starting with d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1 not found: ID does not exist"
Nov 03 01:19:16.940958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.940879    5508 scope.go:122] "RemoveContainer" containerID="511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"
Nov 03 01:19:16.941077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.941058    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"} err="failed to get container status \"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff\": rpc error: code = NotFound desc = could not find container \"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff\": container with ID starting with 511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff not found: ID does not exist"
Nov 03 01:19:16.941122 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.941077    5508 scope.go:122] "RemoveContainer" containerID="267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"
Nov 03 01:19:16.941329 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.941302    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"} err="failed to get container status \"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba\": rpc error: code = NotFound desc = could not find container \"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba\": container with ID starting with 267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba not found: ID does not exist"
Nov 03 01:19:16.941430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.941328    5508 scope.go:122] "RemoveContainer" containerID="321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"
Nov 03 01:19:16.941614 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.941593    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"} err="failed to get container status \"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c\": rpc error: code = NotFound desc = could not find container \"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c\": container with ID starting with 321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c not found: ID does not exist"
Nov 03 01:19:16.941667 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.941616    5508 scope.go:122] "RemoveContainer" containerID="b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"
Nov 03 01:19:16.941954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.941934    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"} err="failed to get container status \"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4\": rpc error: code = NotFound desc = could not find container \"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4\": container with ID starting with b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4 not found: ID does not exist"
Nov 03 01:19:16.941954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.941955    5508 scope.go:122] "RemoveContainer" containerID="0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"
Nov 03 01:19:16.942281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.942260    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"} err="failed to get container status \"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037\": rpc error: code = NotFound desc = could not find container \"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037\": container with ID starting with 0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037 not found: ID does not exist"
Nov 03 01:19:16.942386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.942282    5508 scope.go:122] "RemoveContainer" containerID="85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"
Nov 03 01:19:16.942559 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.942539    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"} err="failed to get container status \"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71\": rpc error: code = NotFound desc = could not find container \"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71\": container with ID starting with 85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71 not found: ID does not exist"
Nov 03 01:19:16.942651 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.942561    5508 scope.go:122] "RemoveContainer" containerID="d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"
Nov 03 01:19:16.942847 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.942826    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"} err="failed to get container status \"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1\": rpc error: code = NotFound desc = could not find container \"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1\": container with ID starting with d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1 not found: ID does not exist"
Nov 03 01:19:16.942934 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.942848    5508 scope.go:122] "RemoveContainer" containerID="511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"
Nov 03 01:19:16.943075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.943050    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"} err="failed to get container status \"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff\": rpc error: code = NotFound desc = could not find container \"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff\": container with ID starting with 511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff not found: ID does not exist"
Nov 03 01:19:16.943121 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.943079    5508 scope.go:122] "RemoveContainer" containerID="267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"
Nov 03 01:19:16.943392 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.943333    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"} err="failed to get container status \"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba\": rpc error: code = NotFound desc = could not find container \"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba\": container with ID starting with 267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba not found: ID does not exist"
Nov 03 01:19:16.943392 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.943378    5508 scope.go:122] "RemoveContainer" containerID="321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"
Nov 03 01:19:16.943627 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.943601    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"} err="failed to get container status \"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c\": rpc error: code = NotFound desc = could not find container \"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c\": container with ID starting with 321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c not found: ID does not exist"
Nov 03 01:19:16.943627 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.943620    5508 scope.go:122] "RemoveContainer" containerID="b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"
Nov 03 01:19:16.943842 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.943824    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"} err="failed to get container status \"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4\": rpc error: code = NotFound desc = could not find container \"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4\": container with ID starting with b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4 not found: ID does not exist"
Nov 03 01:19:16.943906 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.943843    5508 scope.go:122] "RemoveContainer" containerID="0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"
Nov 03 01:19:16.944111 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.944067    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"} err="failed to get container status \"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037\": rpc error: code = NotFound desc = could not find container \"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037\": container with ID starting with 0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037 not found: ID does not exist"
Nov 03 01:19:16.944111 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.944091    5508 scope.go:122] "RemoveContainer" containerID="85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"
Nov 03 01:19:16.944374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.944336    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"} err="failed to get container status \"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71\": rpc error: code = NotFound desc = could not find container \"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71\": container with ID starting with 85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71 not found: ID does not exist"
Nov 03 01:19:16.944454 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.944375    5508 scope.go:122] "RemoveContainer" containerID="d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"
Nov 03 01:19:16.944622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.944596    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"} err="failed to get container status \"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1\": rpc error: code = NotFound desc = could not find container \"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1\": container with ID starting with d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1 not found: ID does not exist"
Nov 03 01:19:16.944693 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.944624    5508 scope.go:122] "RemoveContainer" containerID="511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"
Nov 03 01:19:16.944892 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.944869    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"} err="failed to get container status \"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff\": rpc error: code = NotFound desc = could not find container \"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff\": container with ID starting with 511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff not found: ID does not exist"
Nov 03 01:19:16.944959 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.944894    5508 scope.go:122] "RemoveContainer" containerID="267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"
Nov 03 01:19:16.945153 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.945130    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"} err="failed to get container status \"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba\": rpc error: code = NotFound desc = could not find container \"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba\": container with ID starting with 267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba not found: ID does not exist"
Nov 03 01:19:16.945153 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.945152    5508 scope.go:122] "RemoveContainer" containerID="321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"
Nov 03 01:19:16.945481 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.945451    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"} err="failed to get container status \"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c\": rpc error: code = NotFound desc = could not find container \"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c\": container with ID starting with 321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c not found: ID does not exist"
Nov 03 01:19:16.945481 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.945480    5508 scope.go:122] "RemoveContainer" containerID="b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"
Nov 03 01:19:16.945820 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.945802    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"} err="failed to get container status \"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4\": rpc error: code = NotFound desc = could not find container \"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4\": container with ID starting with b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4 not found: ID does not exist"
Nov 03 01:19:16.945820 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.945819    5508 scope.go:122] "RemoveContainer" containerID="0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"
Nov 03 01:19:16.946030 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.946012    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"} err="failed to get container status \"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037\": rpc error: code = NotFound desc = could not find container \"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037\": container with ID starting with 0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037 not found: ID does not exist"
Nov 03 01:19:16.946030 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.946028    5508 scope.go:122] "RemoveContainer" containerID="85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"
Nov 03 01:19:16.946322 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.946303    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"} err="failed to get container status \"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71\": rpc error: code = NotFound desc = could not find container \"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71\": container with ID starting with 85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71 not found: ID does not exist"
Nov 03 01:19:16.946322 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.946321    5508 scope.go:122] "RemoveContainer" containerID="d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"
Nov 03 01:19:16.946577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.946543    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"} err="failed to get container status \"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1\": rpc error: code = NotFound desc = could not find container \"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1\": container with ID starting with d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1 not found: ID does not exist"
Nov 03 01:19:16.946649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.946578    5508 scope.go:122] "RemoveContainer" containerID="511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"
Nov 03 01:19:16.946874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.946848    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"} err="failed to get container status \"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff\": rpc error: code = NotFound desc = could not find container \"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff\": container with ID starting with 511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff not found: ID does not exist"
Nov 03 01:19:16.946960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.946875    5508 scope.go:122] "RemoveContainer" containerID="267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"
Nov 03 01:19:16.947159 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.947136    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"} err="failed to get container status \"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba\": rpc error: code = NotFound desc = could not find container \"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba\": container with ID starting with 267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba not found: ID does not exist"
Nov 03 01:19:16.947240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.947159    5508 scope.go:122] "RemoveContainer" containerID="321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"
Nov 03 01:19:16.947415 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.947394    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"} err="failed to get container status \"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c\": rpc error: code = NotFound desc = could not find container \"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c\": container with ID starting with 321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c not found: ID does not exist"
Nov 03 01:19:16.947482 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.947416    5508 scope.go:122] "RemoveContainer" containerID="b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"
Nov 03 01:19:16.947708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.947689    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"} err="failed to get container status \"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4\": rpc error: code = NotFound desc = could not find container \"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4\": container with ID starting with b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4 not found: ID does not exist"
Nov 03 01:19:16.947708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.947708    5508 scope.go:122] "RemoveContainer" containerID="0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"
Nov 03 01:19:16.947977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.947956    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037"} err="failed to get container status \"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037\": rpc error: code = NotFound desc = could not find container \"0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037\": container with ID starting with 0cd7dff600928be5f93eb65b8cfbd1ef8de8824a02f876807bacdae1d6342037 not found: ID does not exist"
Nov 03 01:19:16.948098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.947977    5508 scope.go:122] "RemoveContainer" containerID="85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"
Nov 03 01:19:16.948250 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.948228    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71"} err="failed to get container status \"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71\": rpc error: code = NotFound desc = could not find container \"85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71\": container with ID starting with 85375b8ce1c2b511cd2f4b5c5281d1465e9b2b9d6e8dcae534980de3b27f6a71 not found: ID does not exist"
Nov 03 01:19:16.948299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.948252    5508 scope.go:122] "RemoveContainer" containerID="d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"
Nov 03 01:19:16.948546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.948521    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1"} err="failed to get container status \"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1\": rpc error: code = NotFound desc = could not find container \"d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1\": container with ID starting with d13c0e72a39f0cd026c57f215f7896ffd51309d2926a518065bdb3150381aec1 not found: ID does not exist"
Nov 03 01:19:16.948546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.948544    5508 scope.go:122] "RemoveContainer" containerID="511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"
Nov 03 01:19:16.948826 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.948748    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff"} err="failed to get container status \"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff\": rpc error: code = NotFound desc = could not find container \"511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff\": container with ID starting with 511ecf1f819b1cce4ccb169cb4c88ca65b926dedc95b1a252e4f0adfc959f0ff not found: ID does not exist"
Nov 03 01:19:16.948826 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.948770    5508 scope.go:122] "RemoveContainer" containerID="267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"
Nov 03 01:19:16.949021 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.948998    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba"} err="failed to get container status \"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba\": rpc error: code = NotFound desc = could not find container \"267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba\": container with ID starting with 267517c4c4f39bd6a39eba3a394414209c52c4171db7f06a3927e3a82ce20eba not found: ID does not exist"
Nov 03 01:19:16.949072 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.949024    5508 scope.go:122] "RemoveContainer" containerID="321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"
Nov 03 01:19:16.949309 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.949287    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c"} err="failed to get container status \"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c\": rpc error: code = NotFound desc = could not find container \"321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c\": container with ID starting with 321e6e74ea38e127a034ed240c7af58761db88ebd48cbdec2eceda50bfe9561c not found: ID does not exist"
Nov 03 01:19:16.949415 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.949322    5508 scope.go:122] "RemoveContainer" containerID="b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"
Nov 03 01:19:16.949679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:16.949651    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4"} err="failed to get container status \"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4\": rpc error: code = NotFound desc = could not find container \"b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4\": container with ID starting with b73d585107ba142ccd1f8b88c1744e59b41d06ec92063cd3adf7b8bee418d2a4 not found: ID does not exist"
Nov 03 01:19:17.032335 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.032300    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 01:19:17.036790 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.036762    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 01:19:17.067230 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067199    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 01:19:17.067555 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067539    5508 cpu_manager.go:414] "containerMap: removing container" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="config-reloader"
Nov 03 01:19:17.067644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067557    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="config-reloader"
Nov 03 01:19:17.067644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067570    5508 cpu_manager.go:414] "containerMap: removing container" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="kube-rbac-proxy"
Nov 03 01:19:17.067644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067576    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="kube-rbac-proxy"
Nov 03 01:19:17.067644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067584    5508 cpu_manager.go:414] "containerMap: removing container" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="init-config-reloader"
Nov 03 01:19:17.067644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067594    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="init-config-reloader"
Nov 03 01:19:17.067644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067611    5508 cpu_manager.go:414] "containerMap: removing container" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="prometheus"
Nov 03 01:19:17.067644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067619    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="prometheus"
Nov 03 01:19:17.067644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067638    5508 cpu_manager.go:414] "containerMap: removing container" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="kube-rbac-proxy-web"
Nov 03 01:19:17.067644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067645    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="kube-rbac-proxy-web"
Nov 03 01:19:17.068058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067655    5508 cpu_manager.go:414] "containerMap: removing container" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="thanos-sidecar"
Nov 03 01:19:17.068058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067662    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="thanos-sidecar"
Nov 03 01:19:17.068058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067672    5508 cpu_manager.go:414] "containerMap: removing container" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="kube-rbac-proxy-thanos"
Nov 03 01:19:17.068058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067678    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="kube-rbac-proxy-thanos"
Nov 03 01:19:17.068058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067745    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="kube-rbac-proxy-thanos"
Nov 03 01:19:17.068058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067762    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="config-reloader"
Nov 03 01:19:17.068058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067771    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="kube-rbac-proxy"
Nov 03 01:19:17.068058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067780    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="init-config-reloader"
Nov 03 01:19:17.068058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067790    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="prometheus"
Nov 03 01:19:17.068058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067800    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="kube-rbac-proxy-web"
Nov 03 01:19:17.068058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.067808    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" containerName="thanos-sidecar"
Nov 03 01:19:17.073186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.073162    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.115955 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.115920    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116142 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.115967    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89d794f0-b5b4-4c8b-ad25-165a99f69562-config-out\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116142 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116010    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116142 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116082    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116190    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116229    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-web-config\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116250    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116278    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89d794f0-b5b4-4c8b-ad25-165a99f69562-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116379    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116452 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116428    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r467x\" (UniqueName: \"kubernetes.io/projected/89d794f0-b5b4-4c8b-ad25-165a99f69562-kube-api-access-r467x\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116474    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116526 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116505    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116573    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116627 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116617    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116699 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116681    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116740 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116727    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116758    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116810    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116850 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116838    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-config\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.116908 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.116884    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.128407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.128240    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 01:19:17.128407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.128256    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 01:19:17.128407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.128316    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 01:19:17.129166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.128642    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 01:19:17.129166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.128653    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 01:19:17.129166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.128725    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 01:19:17.129166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.128751    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 01:19:17.129166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.128795    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 01:19:17.129166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.128807    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 01:19:17.129166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.128904    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 01:19:17.129166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.129002    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fsir5cbmlb4a0\""
Nov 03 01:19:17.129166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.129058    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 01:19:17.129166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.129073    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tp979\""
Nov 03 01:19:17.129166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.129127    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 01:19:17.175299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.175258    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 01:19:17.175481 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.175407    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 01:19:17.193625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.193576    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 01:19:17.217710 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.217676    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.217890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.217716    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.217890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.217746    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.217890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.217770    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-config\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.217890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.217791    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.217890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.217815    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.217890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.217844    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89d794f0-b5b4-4c8b-ad25-165a99f69562-config-out\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.217890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.217867    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.218240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.217895    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.218240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.217923    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.218240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.217965    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-web-config\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.218240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.217990    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.218240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.218036    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89d794f0-b5b4-4c8b-ad25-165a99f69562-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.218240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.218059    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.218240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.218083    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r467x\" (UniqueName: \"kubernetes.io/projected/89d794f0-b5b4-4c8b-ad25-165a99f69562-kube-api-access-r467x\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.218240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.218115    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.218240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.218143    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.218240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.218180    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.218767 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.218215    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.218767 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.218286    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.218767 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.218612    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.219849 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.219504    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.220582    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.220784    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.221081    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-config\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.221141    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.221175    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.221853    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.221883    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.222377    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.222414    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.222775    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-web-config\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.223367    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.223389    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.223737    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.224379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.224331    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89d794f0-b5b4-4c8b-ad25-165a99f69562-config-out\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.225123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.224755    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89d794f0-b5b4-4c8b-ad25-165a99f69562-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.225276 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.225253    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/89d794f0-b5b4-4c8b-ad25-165a99f69562-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.225634 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.225614    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89d794f0-b5b4-4c8b-ad25-165a99f69562-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.279320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.279293    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r467x\" (UniqueName: \"kubernetes.io/projected/89d794f0-b5b4-4c8b-ad25-165a99f69562-kube-api-access-r467x\") pod \"prometheus-k8s-0\" (UID: \"89d794f0-b5b4-4c8b-ad25-165a99f69562\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.402827 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.402790    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:17.845622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.845589    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 01:19:17.854807 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:19:17.854780    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d794f0_b5b4_4c8b_ad25_165a99f69562.slice/crio-f79c864adcd0b0394d0a17d8817c6fa25c87154871b03cd2927cd9160aff3d10 WatchSource:0}: Error finding container f79c864adcd0b0394d0a17d8817c6fa25c87154871b03cd2927cd9160aff3d10: Status 404 returned error can't find the container with id f79c864adcd0b0394d0a17d8817c6fa25c87154871b03cd2927cd9160aff3d10
Nov 03 01:19:17.881629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.881602    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc" event={"ID":"502e7655-ead5-4ace-a420-cf4a546bcfc7","Type":"ContainerStarted","Data":"9f420b224339038eda701beabe454ea1009a23fbea4da83414a6b027c94c3cd4"}
Nov 03 01:19:17.881957 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.881658    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc" event={"ID":"502e7655-ead5-4ace-a420-cf4a546bcfc7","Type":"ContainerStarted","Data":"8ea07642351cfaa7bb4e7e10d702f722bb1ffab573517354aa0f3341a42b0076"}
Nov 03 01:19:17.881957 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.881679    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc" event={"ID":"502e7655-ead5-4ace-a420-cf4a546bcfc7","Type":"ContainerStarted","Data":"6c65d3d7859fba26b838e47a7586a267f60b4d713638763cc132d490aab29e29"}
Nov 03 01:19:17.883517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.883495    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89d794f0-b5b4-4c8b-ad25-165a99f69562","Type":"ContainerStarted","Data":"f79c864adcd0b0394d0a17d8817c6fa25c87154871b03cd2927cd9160aff3d10"}
Nov 03 01:19:17.996812 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:17.996747    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-dd6c498f-58kkc" podStartSLOduration=1.376676795 podStartE2EDuration="3.996730729s" totalImagesPullingTime="2.620053934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:17.996096141 +0000 UTC m=+37.421702626" watchObservedRunningTime="2026-11-03 01:19:17.996730729 +0000 UTC m=+37.422337194"
Nov 03 01:19:18.689506 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:18.689472    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf8fe558-c1cf-4f0c-96d0-e930abeb857b" path="/var/lib/kubelet/pods/bf8fe558-c1cf-4f0c-96d0-e930abeb857b/volumes"
Nov 03 01:19:18.893563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:18.893532    5508 generic.go:356] "Generic (PLEG): container finished" podID="89d794f0-b5b4-4c8b-ad25-165a99f69562" containerID="67dcb0b02664973dcadd83c1ee76101bf80791a7f21dc87341625334740b188e" exitCode=0
Nov 03 01:19:18.893921 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:18.893620    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89d794f0-b5b4-4c8b-ad25-165a99f69562","Type":"ContainerDied","Data":"67dcb0b02664973dcadd83c1ee76101bf80791a7f21dc87341625334740b188e"}
Nov 03 01:19:19.900173 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:19.900141    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:19:19.900686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:19.900506    5508 generic.go:356] "Generic (PLEG): container finished" podID="89d794f0-b5b4-4c8b-ad25-165a99f69562" containerID="d2d771c288e66db263c320e4061c98d364fec73763a96b34bfa609b22529fb4e" exitCode=2
Nov 03 01:19:19.900686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:19.900578    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89d794f0-b5b4-4c8b-ad25-165a99f69562","Type":"ContainerStarted","Data":"ebef75f383137cdb3b49a2107ab51445177aa844c37d6b62c270a795f1d4f059"}
Nov 03 01:19:19.900686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:19.900624    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89d794f0-b5b4-4c8b-ad25-165a99f69562","Type":"ContainerStarted","Data":"5febf912d671597624569aee1f2800c105590dd2820b28223016009dc2db7dd4"}
Nov 03 01:19:19.900686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:19.900650    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89d794f0-b5b4-4c8b-ad25-165a99f69562","Type":"ContainerStarted","Data":"2c25bfecd66a5fb29830aa27d2985c81b66372d974a1a7fe678099fbe2c700e7"}
Nov 03 01:19:19.900686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:19.900672    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89d794f0-b5b4-4c8b-ad25-165a99f69562","Type":"ContainerStarted","Data":"dad40ab9ace2839aa42ba9999acb69abfc27bf54b0ae82637f8530d0adc45be7"}
Nov 03 01:19:19.900912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:19.900694    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89d794f0-b5b4-4c8b-ad25-165a99f69562","Type":"ContainerStarted","Data":"516e5a8a876dbc15c08d7fc26383f9488db3ea1eacb0c7af0b1cfd081e07c2c4"}
Nov 03 01:19:19.900912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:19.900721    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89d794f0-b5b4-4c8b-ad25-165a99f69562","Type":"ContainerDied","Data":"d2d771c288e66db263c320e4061c98d364fec73763a96b34bfa609b22529fb4e"}
Nov 03 01:19:19.900981 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:19.900938    5508 scope.go:122] "RemoveContainer" containerID="d2d771c288e66db263c320e4061c98d364fec73763a96b34bfa609b22529fb4e"
Nov 03 01:19:20.906224 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:20.906199    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:19:20.906708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:20.906682    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89d794f0-b5b4-4c8b-ad25-165a99f69562","Type":"ContainerStarted","Data":"45598ea0cc1578065e831cccd7762f5b7886c41cce69169758516b6edacff1e9"}
Nov 03 01:19:21.125139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:21.125091    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.111192669 podStartE2EDuration="4.125078761s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.886092ms" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:21.078913672 +0000 UTC m=+40.504520135" watchObservedRunningTime="2026-11-03 01:19:21.125078761 +0000 UTC m=+40.550685265"
Nov 03 01:19:22.403028 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:19:22.402997    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:19:43.809067 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:19:43.809029    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:20:01.801685 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:20:01.801653    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:20:01.816530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:20:01.816505    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:20:02.065796 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:20:02.065709    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 01:20:13.809886 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:20:13.809838    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:20:43.810026 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:20:43.809989    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:21:13.809960 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:21:13.809921    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:21:43.810738 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:21:43.810692    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:22:13.809628 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:22:13.809591    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:22:24.197003 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.196968    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"]
Nov 03 01:22:24.200590 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.200571    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"
Nov 03 01:22:24.295717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.295697    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 01:22:24.295835 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.295785    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 01:22:24.295835 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.295789    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-5p282\""
Nov 03 01:22:24.296500 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.296485    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"]
Nov 03 01:22:24.346387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.346364    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flb5j\" (UniqueName: \"kubernetes.io/projected/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-kube-api-access-flb5j\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq\" (UID: \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"
Nov 03 01:22:24.346486 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.346399    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq\" (UID: \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"
Nov 03 01:22:24.346486 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.346421    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq\" (UID: \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"
Nov 03 01:22:24.446842 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.446820    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flb5j\" (UniqueName: \"kubernetes.io/projected/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-kube-api-access-flb5j\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq\" (UID: \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"
Nov 03 01:22:24.446928 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.446863    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq\" (UID: \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"
Nov 03 01:22:24.446928 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.446886    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq\" (UID: \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"
Nov 03 01:22:24.447190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.447152    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq\" (UID: \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"
Nov 03 01:22:24.447229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.447186    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq\" (UID: \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"
Nov 03 01:22:24.504476 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.504451    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flb5j\" (UniqueName: \"kubernetes.io/projected/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-kube-api-access-flb5j\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq\" (UID: \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"
Nov 03 01:22:24.515488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.515467    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"
Nov 03 01:22:24.880460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:24.880381    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"]
Nov 03 01:22:24.889100 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:22:24.889072    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c74c0c_4f1a_4563_8e6d_f0495bfa0a4c.slice/crio-421af03f14a062f0a35df995f9ebaf5f2f1cb5b7f2ce5b2df3cd639f9575f154 WatchSource:0}: Error finding container 421af03f14a062f0a35df995f9ebaf5f2f1cb5b7f2ce5b2df3cd639f9575f154: Status 404 returned error can't find the container with id 421af03f14a062f0a35df995f9ebaf5f2f1cb5b7f2ce5b2df3cd639f9575f154
Nov 03 01:22:25.472896 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.472866    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"]
Nov 03 01:22:25.477258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.477241    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"
Nov 03 01:22:25.527810 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.527784    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq" event={"ID":"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c","Type":"ContainerStarted","Data":"421af03f14a062f0a35df995f9ebaf5f2f1cb5b7f2ce5b2df3cd639f9575f154"}
Nov 03 01:22:25.552936 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.552912    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d\" (UID: \"ac570087-5c20-4ba8-91ce-013f501a6337\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"
Nov 03 01:22:25.553045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.552970    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvlq\" (UniqueName: \"kubernetes.io/projected/ac570087-5c20-4ba8-91ce-013f501a6337-kube-api-access-6bvlq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d\" (UID: \"ac570087-5c20-4ba8-91ce-013f501a6337\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"
Nov 03 01:22:25.553108 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.553049    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d\" (UID: \"ac570087-5c20-4ba8-91ce-013f501a6337\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"
Nov 03 01:22:25.574834 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.574810    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-247px\""
Nov 03 01:22:25.577057 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.577031    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"]
Nov 03 01:22:25.577168 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.577117    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"]
Nov 03 01:22:25.581299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.581266    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"
Nov 03 01:22:25.631384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.631357    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"]
Nov 03 01:22:25.633679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.633478    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-7lmhq\""
Nov 03 01:22:25.653745 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.653719    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvlq\" (UniqueName: \"kubernetes.io/projected/ac570087-5c20-4ba8-91ce-013f501a6337-kube-api-access-6bvlq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d\" (UID: \"ac570087-5c20-4ba8-91ce-013f501a6337\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"
Nov 03 01:22:25.653853 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.653753    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gntll\" (UniqueName: \"kubernetes.io/projected/71ffe894-0908-4722-b283-5d5dc7442391-kube-api-access-gntll\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf\" (UID: \"71ffe894-0908-4722-b283-5d5dc7442391\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"
Nov 03 01:22:25.653853 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.653774    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d\" (UID: \"ac570087-5c20-4ba8-91ce-013f501a6337\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"
Nov 03 01:22:25.653954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.653938    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf\" (UID: \"71ffe894-0908-4722-b283-5d5dc7442391\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"
Nov 03 01:22:25.653999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.653980    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d\" (UID: \"ac570087-5c20-4ba8-91ce-013f501a6337\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"
Nov 03 01:22:25.654044 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.654013    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf\" (UID: \"71ffe894-0908-4722-b283-5d5dc7442391\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"
Nov 03 01:22:25.654199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.654182    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d\" (UID: \"ac570087-5c20-4ba8-91ce-013f501a6337\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"
Nov 03 01:22:25.654251 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.654236    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d\" (UID: \"ac570087-5c20-4ba8-91ce-013f501a6337\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"
Nov 03 01:22:25.672087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.672064    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"]
Nov 03 01:22:25.675450 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.675433    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"
Nov 03 01:22:25.711170 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.711145    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvlq\" (UniqueName: \"kubernetes.io/projected/ac570087-5c20-4ba8-91ce-013f501a6337-kube-api-access-6bvlq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d\" (UID: \"ac570087-5c20-4ba8-91ce-013f501a6337\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"
Nov 03 01:22:25.724686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.724635    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-fc7zd\""
Nov 03 01:22:25.730480 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.730459    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"]
Nov 03 01:22:25.754716 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.754690    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf\" (UID: \"71ffe894-0908-4722-b283-5d5dc7442391\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"
Nov 03 01:22:25.754815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.754720    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j\" (UID: \"7e8a3f80-6450-4566-8b80-6215b2914a0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"
Nov 03 01:22:25.754815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.754760    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fm42\" (UniqueName: \"kubernetes.io/projected/7e8a3f80-6450-4566-8b80-6215b2914a0a-kube-api-access-8fm42\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j\" (UID: \"7e8a3f80-6450-4566-8b80-6215b2914a0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"
Nov 03 01:22:25.754902 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.754879    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gntll\" (UniqueName: \"kubernetes.io/projected/71ffe894-0908-4722-b283-5d5dc7442391-kube-api-access-gntll\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf\" (UID: \"71ffe894-0908-4722-b283-5d5dc7442391\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"
Nov 03 01:22:25.754933 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.754920    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf\" (UID: \"71ffe894-0908-4722-b283-5d5dc7442391\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"
Nov 03 01:22:25.754989 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.754972    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j\" (UID: \"7e8a3f80-6450-4566-8b80-6215b2914a0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"
Nov 03 01:22:25.755064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.755047    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf\" (UID: \"71ffe894-0908-4722-b283-5d5dc7442391\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"
Nov 03 01:22:25.755202 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.755186    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf\" (UID: \"71ffe894-0908-4722-b283-5d5dc7442391\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"
Nov 03 01:22:25.773686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.773661    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"]
Nov 03 01:22:25.776999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.776986    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"
Nov 03 01:22:25.792158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.792140    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"
Nov 03 01:22:25.812617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.812595    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gntll\" (UniqueName: \"kubernetes.io/projected/71ffe894-0908-4722-b283-5d5dc7442391-kube-api-access-gntll\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf\" (UID: \"71ffe894-0908-4722-b283-5d5dc7442391\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"
Nov 03 01:22:25.826252 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.826219    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-t76zb\""
Nov 03 01:22:25.832814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.832789    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"]
Nov 03 01:22:25.856313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.856280    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fm42\" (UniqueName: \"kubernetes.io/projected/7e8a3f80-6450-4566-8b80-6215b2914a0a-kube-api-access-8fm42\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j\" (UID: \"7e8a3f80-6450-4566-8b80-6215b2914a0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"
Nov 03 01:22:25.856475 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.856325    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt\" (UID: \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"
Nov 03 01:22:25.856475 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.856388    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt\" (UID: \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"
Nov 03 01:22:25.856475 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.856464    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j\" (UID: \"7e8a3f80-6450-4566-8b80-6215b2914a0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"
Nov 03 01:22:25.856652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.856490    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j\" (UID: \"7e8a3f80-6450-4566-8b80-6215b2914a0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"
Nov 03 01:22:25.856652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.856519    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfk4b\" (UniqueName: \"kubernetes.io/projected/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-kube-api-access-lfk4b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt\" (UID: \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"
Nov 03 01:22:25.857162 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.857143    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j\" (UID: \"7e8a3f80-6450-4566-8b80-6215b2914a0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"
Nov 03 01:22:25.857325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.857312    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j\" (UID: \"7e8a3f80-6450-4566-8b80-6215b2914a0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"
Nov 03 01:22:25.895519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.895494    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"
Nov 03 01:22:25.914337 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.914307    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fm42\" (UniqueName: \"kubernetes.io/projected/7e8a3f80-6450-4566-8b80-6215b2914a0a-kube-api-access-8fm42\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j\" (UID: \"7e8a3f80-6450-4566-8b80-6215b2914a0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"
Nov 03 01:22:25.957963 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.957930    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt\" (UID: \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"
Nov 03 01:22:25.958093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.957989    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt\" (UID: \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"
Nov 03 01:22:25.958093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.958077    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfk4b\" (UniqueName: \"kubernetes.io/projected/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-kube-api-access-lfk4b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt\" (UID: \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"
Nov 03 01:22:25.958759 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.958734    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt\" (UID: \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"
Nov 03 01:22:25.958937 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.958916    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt\" (UID: \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"
Nov 03 01:22:25.990480 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:25.990410    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"
Nov 03 01:22:26.016366 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:26.016322    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfk4b\" (UniqueName: \"kubernetes.io/projected/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-kube-api-access-lfk4b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt\" (UID: \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"
Nov 03 01:22:26.092051 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:26.092006    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"
Nov 03 01:22:26.164515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:26.164447    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"]
Nov 03 01:22:26.173235 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:22:26.173197    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac570087_5c20_4ba8_91ce_013f501a6337.slice/crio-05fb29c2e1da96e3d1c7becf4ddaeb1d5b20f5c33f38dc784279fffaa23800c3 WatchSource:0}: Error finding container 05fb29c2e1da96e3d1c7becf4ddaeb1d5b20f5c33f38dc784279fffaa23800c3: Status 404 returned error can't find the container with id 05fb29c2e1da96e3d1c7becf4ddaeb1d5b20f5c33f38dc784279fffaa23800c3
Nov 03 01:22:26.268818 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:26.268757    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"]
Nov 03 01:22:26.277470 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:22:26.277440    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ffe894_0908_4722_b283_5d5dc7442391.slice/crio-917f93025d56264583c2eef2c2fec3213d283d7284b4b4dbc1799bd37941c36d WatchSource:0}: Error finding container 917f93025d56264583c2eef2c2fec3213d283d7284b4b4dbc1799bd37941c36d: Status 404 returned error can't find the container with id 917f93025d56264583c2eef2c2fec3213d283d7284b4b4dbc1799bd37941c36d
Nov 03 01:22:26.375673 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:26.375610    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"]
Nov 03 01:22:26.384907 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:22:26.384876    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8a3f80_6450_4566_8b80_6215b2914a0a.slice/crio-29db3d1facc97d2bd7c0909ec607159e76f9454c654ae4b9d9ee0c0c03bf4d3a WatchSource:0}: Error finding container 29db3d1facc97d2bd7c0909ec607159e76f9454c654ae4b9d9ee0c0c03bf4d3a: Status 404 returned error can't find the container with id 29db3d1facc97d2bd7c0909ec607159e76f9454c654ae4b9d9ee0c0c03bf4d3a
Nov 03 01:22:26.467849 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:26.467818    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"]
Nov 03 01:22:26.475884 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:22:26.475855    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6ecd27_0c07_4451_a8fa_c7f74eb2fe22.slice/crio-0a6e20e1453dcef41ca884664eaf99a723348732b5538a4bc75a66c350ddf5a1 WatchSource:0}: Error finding container 0a6e20e1453dcef41ca884664eaf99a723348732b5538a4bc75a66c350ddf5a1: Status 404 returned error can't find the container with id 0a6e20e1453dcef41ca884664eaf99a723348732b5538a4bc75a66c350ddf5a1
Nov 03 01:22:26.533086 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:26.533025    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j" event={"ID":"7e8a3f80-6450-4566-8b80-6215b2914a0a","Type":"ContainerStarted","Data":"29db3d1facc97d2bd7c0909ec607159e76f9454c654ae4b9d9ee0c0c03bf4d3a"}
Nov 03 01:22:26.534365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:26.534322    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt" event={"ID":"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22","Type":"ContainerStarted","Data":"0a6e20e1453dcef41ca884664eaf99a723348732b5538a4bc75a66c350ddf5a1"}
Nov 03 01:22:26.535763 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:26.535739    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf" event={"ID":"71ffe894-0908-4722-b283-5d5dc7442391","Type":"ContainerStarted","Data":"917f93025d56264583c2eef2c2fec3213d283d7284b4b4dbc1799bd37941c36d"}
Nov 03 01:22:26.537119 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:26.537091    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d" event={"ID":"ac570087-5c20-4ba8-91ce-013f501a6337","Type":"ContainerStarted","Data":"05fb29c2e1da96e3d1c7becf4ddaeb1d5b20f5c33f38dc784279fffaa23800c3"}
Nov 03 01:22:26.995430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:26.995398    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"]
Nov 03 01:22:27.000220 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.000198    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"
Nov 03 01:22:27.049274 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.049246    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-wwcg5\""
Nov 03 01:22:27.053831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.053805    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"]
Nov 03 01:22:27.068072 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.068050    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k\" (UID: \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"
Nov 03 01:22:27.068202 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.068117    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k\" (UID: \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"
Nov 03 01:22:27.068254 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.068209    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhtnk\" (UniqueName: \"kubernetes.io/projected/0c642bfc-c879-4cb1-83e5-75cf48f2427e-kube-api-access-nhtnk\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k\" (UID: \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"
Nov 03 01:22:27.169612 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.169585    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k\" (UID: \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"
Nov 03 01:22:27.169753 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.169651    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k\" (UID: \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"
Nov 03 01:22:27.169753 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.169711    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtnk\" (UniqueName: \"kubernetes.io/projected/0c642bfc-c879-4cb1-83e5-75cf48f2427e-kube-api-access-nhtnk\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k\" (UID: \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"
Nov 03 01:22:27.170012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.169982    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k\" (UID: \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"
Nov 03 01:22:27.170104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.170031    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k\" (UID: \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"
Nov 03 01:22:27.228074 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.228053    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhtnk\" (UniqueName: \"kubernetes.io/projected/0c642bfc-c879-4cb1-83e5-75cf48f2427e-kube-api-access-nhtnk\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k\" (UID: \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"
Nov 03 01:22:27.319257 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.319152    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"
Nov 03 01:22:27.705310 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.705282    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"]
Nov 03 01:22:27.719549 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:22:27.719520    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c642bfc_c879_4cb1_83e5_75cf48f2427e.slice/crio-dd3a063be953006b48ac3243a95fb9d6312b40d5a9a8eba56cfa7a0d92d81bf7 WatchSource:0}: Error finding container dd3a063be953006b48ac3243a95fb9d6312b40d5a9a8eba56cfa7a0d92d81bf7: Status 404 returned error can't find the container with id dd3a063be953006b48ac3243a95fb9d6312b40d5a9a8eba56cfa7a0d92d81bf7
Nov 03 01:22:27.833976 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.832458    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"]
Nov 03 01:22:27.837112 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.836492    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"
Nov 03 01:22:27.886238 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.886191    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-4lhp2\""
Nov 03 01:22:27.890786 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.890714    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"]
Nov 03 01:22:27.977571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.977251    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp\" (UID: \"3a212187-7678-4368-9b6a-da15c45f73ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"
Nov 03 01:22:27.977571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.977310    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp\" (UID: \"3a212187-7678-4368-9b6a-da15c45f73ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"
Nov 03 01:22:27.977571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:27.977365    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzdkr\" (UniqueName: \"kubernetes.io/projected/3a212187-7678-4368-9b6a-da15c45f73ac-kube-api-access-rzdkr\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp\" (UID: \"3a212187-7678-4368-9b6a-da15c45f73ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"
Nov 03 01:22:28.080006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:28.078903    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp\" (UID: \"3a212187-7678-4368-9b6a-da15c45f73ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"
Nov 03 01:22:28.080006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:28.078952    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp\" (UID: \"3a212187-7678-4368-9b6a-da15c45f73ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"
Nov 03 01:22:28.080006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:28.078988    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzdkr\" (UniqueName: \"kubernetes.io/projected/3a212187-7678-4368-9b6a-da15c45f73ac-kube-api-access-rzdkr\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp\" (UID: \"3a212187-7678-4368-9b6a-da15c45f73ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"
Nov 03 01:22:28.080006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:28.079701    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp\" (UID: \"3a212187-7678-4368-9b6a-da15c45f73ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"
Nov 03 01:22:28.080006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:28.079950    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp\" (UID: \"3a212187-7678-4368-9b6a-da15c45f73ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"
Nov 03 01:22:28.141006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:28.140975    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzdkr\" (UniqueName: \"kubernetes.io/projected/3a212187-7678-4368-9b6a-da15c45f73ac-kube-api-access-rzdkr\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp\" (UID: \"3a212187-7678-4368-9b6a-da15c45f73ac\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"
Nov 03 01:22:28.155906 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:28.155882    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"
Nov 03 01:22:28.548624 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:28.548493    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k" event={"ID":"0c642bfc-c879-4cb1-83e5-75cf48f2427e","Type":"ContainerStarted","Data":"dd3a063be953006b48ac3243a95fb9d6312b40d5a9a8eba56cfa7a0d92d81bf7"}
Nov 03 01:22:28.575412 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:28.575374    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"]
Nov 03 01:22:28.590290 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:22:28.590260    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a212187_7678_4368_9b6a_da15c45f73ac.slice/crio-b17da8a2436c24b101821063c17d81d67f033e6ff3f71c85a9b9c69df7bc2c0c WatchSource:0}: Error finding container b17da8a2436c24b101821063c17d81d67f033e6ff3f71c85a9b9c69df7bc2c0c: Status 404 returned error can't find the container with id b17da8a2436c24b101821063c17d81d67f033e6ff3f71c85a9b9c69df7bc2c0c
Nov 03 01:22:29.557727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.557617    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp" event={"ID":"3a212187-7678-4368-9b6a-da15c45f73ac","Type":"ContainerStarted","Data":"b17da8a2436c24b101821063c17d81d67f033e6ff3f71c85a9b9c69df7bc2c0c"}
Nov 03 01:22:29.667848 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.665718    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"]
Nov 03 01:22:29.670975 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.670037    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.720098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.720069    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 01:22:29.720342 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.720164    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 01:22:29.720342 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.720300    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 01:22:29.766000 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.765973    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-w2lkg\""
Nov 03 01:22:29.767647 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.767614    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"]
Nov 03 01:22:29.799172 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.799101    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.799172 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.799157    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.799483 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.799233    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05f8000-1622-4313-89df-5d55ce6b2f81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.799483 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.799298    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.799632 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.799492    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.799632 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.799524    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.799632 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.799568    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.799632 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.799622    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clhlm\" (UniqueName: \"kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-kube-api-access-clhlm\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.799893 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.799651    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.900284 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.900243    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.900472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.900319    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.900472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.900343    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.900472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.900388    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.900472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.900422    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clhlm\" (UniqueName: \"kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-kube-api-access-clhlm\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.900472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.900446    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.900717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.900508    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.900717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.900539    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.900717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.900580    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05f8000-1622-4313-89df-5d55ce6b2f81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.901538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.901513    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05f8000-1622-4313-89df-5d55ce6b2f81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.902545 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.902490    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.903083 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.903060    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.903697 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.903672    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.903821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.903731    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.912665 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.912637    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.915829 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.915803    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.971849 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.971819    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clhlm\" (UniqueName: \"kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-kube-api-access-clhlm\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.977479 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.977442    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-s9q4w\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:29.988485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:29.988283    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:30.457982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:30.457943    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"]
Nov 03 01:22:30.475868 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:22:30.475834    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05f8000_1622_4313_89df_5d55ce6b2f81.slice/crio-cefdb248c2c3eb4346144e37b8c397bdd093c7bf94d3cecc8231470cb2a29adb WatchSource:0}: Error finding container cefdb248c2c3eb4346144e37b8c397bdd093c7bf94d3cecc8231470cb2a29adb: Status 404 returned error can't find the container with id cefdb248c2c3eb4346144e37b8c397bdd093c7bf94d3cecc8231470cb2a29adb
Nov 03 01:22:30.579260 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:30.579184    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" event={"ID":"d05f8000-1622-4313-89df-5d55ce6b2f81","Type":"ContainerStarted","Data":"cefdb248c2c3eb4346144e37b8c397bdd093c7bf94d3cecc8231470cb2a29adb"}
Nov 03 01:22:37.638175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:37.638087    5508 generic.go:356] "Generic (PLEG): container finished" podID="06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c" containerID="9fda4c159b162c84d0450e9c9c4ef3fd40e0a47449eb88683c6088724caa5c67" exitCode=0
Nov 03 01:22:37.638175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:37.638160    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq" event={"ID":"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c","Type":"ContainerDied","Data":"9fda4c159b162c84d0450e9c9c4ef3fd40e0a47449eb88683c6088724caa5c67"}
Nov 03 01:22:39.155077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:39.154955    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:22:39.155077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:39.155051    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:22:39.155532 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:39.155160    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:22:39.655127 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:39.654294    5508 generic.go:356] "Generic (PLEG): container finished" podID="3a212187-7678-4368-9b6a-da15c45f73ac" containerID="00f19abdd8360ad67536b20499adcf3004f30c489a59d9fd400c6c259cf09cea" exitCode=0
Nov 03 01:22:39.655127 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:39.654404    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp" event={"ID":"3a212187-7678-4368-9b6a-da15c45f73ac","Type":"ContainerDied","Data":"00f19abdd8360ad67536b20499adcf3004f30c489a59d9fd400c6c259cf09cea"}
Nov 03 01:22:39.660328 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:39.660296    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" event={"ID":"d05f8000-1622-4313-89df-5d55ce6b2f81","Type":"ContainerStarted","Data":"ad1c888e0d2d24111a812a20b767307a08f7f2f80093c931370712b6334825d6"}
Nov 03 01:22:39.886962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:39.886554    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" podStartSLOduration=2.213029751 podStartE2EDuration="10.886532258s" totalImagesPullingTime="8.673502507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:39.884401448 +0000 UTC m=+239.310007938" watchObservedRunningTime="2026-11-03 01:22:39.886532258 +0000 UTC m=+239.312138724"
Nov 03 01:22:39.989327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:39.989238    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:40.666293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:40.666263    5508 generic.go:356] "Generic (PLEG): container finished" podID="da6ecd27-0c07-4451-a8fa-c7f74eb2fe22" containerID="a656920d1fd513f2f4b91ed95b62fa1317cb41337dfb1afc8f2642e3b3dc89d5" exitCode=0
Nov 03 01:22:40.666757 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:40.666341    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt" event={"ID":"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22","Type":"ContainerDied","Data":"a656920d1fd513f2f4b91ed95b62fa1317cb41337dfb1afc8f2642e3b3dc89d5"}
Nov 03 01:22:40.668461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:40.668436    5508 generic.go:356] "Generic (PLEG): container finished" podID="06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c" containerID="50e8a685e26216274b5184793f38ceb7dd9d2e996564a75463b93102ac1f9ee8" exitCode=0
Nov 03 01:22:40.668567 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:40.668471    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq" event={"ID":"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c","Type":"ContainerDied","Data":"50e8a685e26216274b5184793f38ceb7dd9d2e996564a75463b93102ac1f9ee8"}
Nov 03 01:22:40.670615 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:40.670538    5508 generic.go:356] "Generic (PLEG): container finished" podID="71ffe894-0908-4722-b283-5d5dc7442391" containerID="eaf484529c3a3953c685ab8c5acfaa8512850a0b9e15dbdefdc1814a9e1bf953" exitCode=0
Nov 03 01:22:40.671053 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:40.671038    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf" event={"ID":"71ffe894-0908-4722-b283-5d5dc7442391","Type":"ContainerDied","Data":"eaf484529c3a3953c685ab8c5acfaa8512850a0b9e15dbdefdc1814a9e1bf953"}
Nov 03 01:22:40.994495 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:40.994416    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:41.676182 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:41.676144    5508 generic.go:356] "Generic (PLEG): container finished" podID="7e8a3f80-6450-4566-8b80-6215b2914a0a" containerID="5a644b5326b6321316605a12dbe06822f78622201aeafce203be3cc4c3b5d1eb" exitCode=0
Nov 03 01:22:41.676641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:41.676238    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j" event={"ID":"7e8a3f80-6450-4566-8b80-6215b2914a0a","Type":"ContainerDied","Data":"5a644b5326b6321316605a12dbe06822f78622201aeafce203be3cc4c3b5d1eb"}
Nov 03 01:22:41.678582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:41.678496    5508 generic.go:356] "Generic (PLEG): container finished" podID="ac570087-5c20-4ba8-91ce-013f501a6337" containerID="37320cdeb2d099b01b4a6dd1f85aa1b7735cdacaa0eb819244e301af43b75d72" exitCode=0
Nov 03 01:22:41.678635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:41.678601    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d" event={"ID":"ac570087-5c20-4ba8-91ce-013f501a6337","Type":"ContainerDied","Data":"37320cdeb2d099b01b4a6dd1f85aa1b7735cdacaa0eb819244e301af43b75d72"}
Nov 03 01:22:41.680641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:41.680618    5508 generic.go:356] "Generic (PLEG): container finished" podID="0c642bfc-c879-4cb1-83e5-75cf48f2427e" containerID="e1b1ed9c9f6574866a13c9283a4960dc12249f2f278221479ac33f73e6e527aa" exitCode=0
Nov 03 01:22:41.680739 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:41.680695    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k" event={"ID":"0c642bfc-c879-4cb1-83e5-75cf48f2427e","Type":"ContainerDied","Data":"e1b1ed9c9f6574866a13c9283a4960dc12249f2f278221479ac33f73e6e527aa"}
Nov 03 01:22:41.681421 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:41.681056    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:41.682788 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:41.682766    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:22:43.690819 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:43.690784    5508 generic.go:356] "Generic (PLEG): container finished" podID="71ffe894-0908-4722-b283-5d5dc7442391" containerID="841cc53b2c6ed2b88acd4d9797ba025558d1ab2b1fd951d941663b116bd2ad26" exitCode=0
Nov 03 01:22:43.691224 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:43.691110    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf" event={"ID":"71ffe894-0908-4722-b283-5d5dc7442391","Type":"ContainerDied","Data":"841cc53b2c6ed2b88acd4d9797ba025558d1ab2b1fd951d941663b116bd2ad26"}
Nov 03 01:22:43.694665 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:43.694077    5508 generic.go:356] "Generic (PLEG): container finished" podID="da6ecd27-0c07-4451-a8fa-c7f74eb2fe22" containerID="cc10815bb02ffa87455166e5baae2685adafd573efb021c81107e9170ab9bc75" exitCode=0
Nov 03 01:22:43.694665 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:43.694171    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt" event={"ID":"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22","Type":"ContainerDied","Data":"cc10815bb02ffa87455166e5baae2685adafd573efb021c81107e9170ab9bc75"}
Nov 03 01:22:43.696301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:43.696278    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp" event={"ID":"3a212187-7678-4368-9b6a-da15c45f73ac","Type":"ContainerStarted","Data":"3edb5da56bfd90c05343abcc5124cc61c580ce3a40d19a7a1df111fe21a45767"}
Nov 03 01:22:43.813880 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:22:43.813845    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:22:44.702181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:44.702135    5508 generic.go:356] "Generic (PLEG): container finished" podID="7e8a3f80-6450-4566-8b80-6215b2914a0a" containerID="2d5911e75fe2499696791d60e901d586ccb42b45341d631485df4c8cfb7d23d7" exitCode=0
Nov 03 01:22:44.702657 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:44.702243    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j" event={"ID":"7e8a3f80-6450-4566-8b80-6215b2914a0a","Type":"ContainerDied","Data":"2d5911e75fe2499696791d60e901d586ccb42b45341d631485df4c8cfb7d23d7"}
Nov 03 01:22:44.704416 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:44.704329    5508 generic.go:356] "Generic (PLEG): container finished" podID="ac570087-5c20-4ba8-91ce-013f501a6337" containerID="4e017feb5c09452bdf551f5b0ca4c0709873eaa116526839ce929aee3f4b3523" exitCode=0
Nov 03 01:22:44.704486 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:44.704450    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d" event={"ID":"ac570087-5c20-4ba8-91ce-013f501a6337","Type":"ContainerDied","Data":"4e017feb5c09452bdf551f5b0ca4c0709873eaa116526839ce929aee3f4b3523"}
Nov 03 01:22:44.707033 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:44.707011    5508 generic.go:356] "Generic (PLEG): container finished" podID="3a212187-7678-4368-9b6a-da15c45f73ac" containerID="3edb5da56bfd90c05343abcc5124cc61c580ce3a40d19a7a1df111fe21a45767" exitCode=0
Nov 03 01:22:44.707124 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:44.707076    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp" event={"ID":"3a212187-7678-4368-9b6a-da15c45f73ac","Type":"ContainerDied","Data":"3edb5da56bfd90c05343abcc5124cc61c580ce3a40d19a7a1df111fe21a45767"}
Nov 03 01:22:45.720330 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:45.720289    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k" event={"ID":"0c642bfc-c879-4cb1-83e5-75cf48f2427e","Type":"ContainerStarted","Data":"1c30d439853eb1f3db78d5f8d1842bd70c646a3e55227987d6663720a2760ae2"}
Nov 03 01:22:46.733286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:46.733229    5508 generic.go:356] "Generic (PLEG): container finished" podID="0c642bfc-c879-4cb1-83e5-75cf48f2427e" containerID="1c30d439853eb1f3db78d5f8d1842bd70c646a3e55227987d6663720a2760ae2" exitCode=0
Nov 03 01:22:46.734974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:46.734385    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k" event={"ID":"0c642bfc-c879-4cb1-83e5-75cf48f2427e","Type":"ContainerDied","Data":"1c30d439853eb1f3db78d5f8d1842bd70c646a3e55227987d6663720a2760ae2"}
Nov 03 01:22:56.810750 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:56.810715    5508 generic.go:356] "Generic (PLEG): container finished" podID="06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c" containerID="64c989a84378002cb775e1dc2add436abb27cafece9f1730c36b8d5748d37e8b" exitCode=0
Nov 03 01:22:56.811215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:56.810819    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq" event={"ID":"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c","Type":"ContainerDied","Data":"64c989a84378002cb775e1dc2add436abb27cafece9f1730c36b8d5748d37e8b"}
Nov 03 01:22:58.037673 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.037333    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"
Nov 03 01:22:58.075237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.074464    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-util\" (UniqueName: \"kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-util\") pod \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\" (UID: \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\") "
Nov 03 01:22:58.075237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.074531    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-bundle\") pod \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\" (UID: \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\") "
Nov 03 01:22:58.075237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.074556    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-kube-api-access-flb5j\" (UniqueName: \"kubernetes.io/projected/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-kube-api-access-flb5j\") pod \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\" (UID: \"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c\") "
Nov 03 01:22:58.076832 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.075579    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-bundle" pod "06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c" (UID: "06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:22:58.080221 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.080004    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-kube-api-access-flb5j" pod "06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c" (UID: "06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c"). InnerVolumeSpecName "kube-api-access-flb5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:22:58.081770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.081743    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-util" pod "06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c" (UID: "06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:22:58.175671 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.175561    5508 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-util\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:22:58.175671 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.175599    5508 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-bundle\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:22:58.175671 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.175614    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flb5j\" (UniqueName: \"kubernetes.io/projected/06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c-kube-api-access-flb5j\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:22:58.825685 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.825654    5508 generic.go:356] "Generic (PLEG): container finished" podID="da6ecd27-0c07-4451-a8fa-c7f74eb2fe22" containerID="0b994f8573bca6fc34a1e7d43b453c6317f4d56a07837d29e6a74f53f3d5ea33" exitCode=0
Nov 03 01:22:58.825812 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.825765    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt" event={"ID":"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22","Type":"ContainerDied","Data":"0b994f8573bca6fc34a1e7d43b453c6317f4d56a07837d29e6a74f53f3d5ea33"}
Nov 03 01:22:58.828966 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.828941    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq" event={"ID":"06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c","Type":"ContainerDied","Data":"421af03f14a062f0a35df995f9ebaf5f2f1cb5b7f2ce5b2df3cd639f9575f154"}
Nov 03 01:22:58.829065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.828986    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="421af03f14a062f0a35df995f9ebaf5f2f1cb5b7f2ce5b2df3cd639f9575f154"
Nov 03 01:22:58.829065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:58.829060    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71txngq"
Nov 03 01:22:59.835485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:59.835454    5508 generic.go:356] "Generic (PLEG): container finished" podID="ac570087-5c20-4ba8-91ce-013f501a6337" containerID="1acc371761a72605b0b4644b703e47a8b3f56383da6d4d78adc44968f46306c4" exitCode=0
Nov 03 01:22:59.835852 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:59.835520    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d" event={"ID":"ac570087-5c20-4ba8-91ce-013f501a6337","Type":"ContainerDied","Data":"1acc371761a72605b0b4644b703e47a8b3f56383da6d4d78adc44968f46306c4"}
Nov 03 01:22:59.837725 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:59.837701    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp" event={"ID":"3a212187-7678-4368-9b6a-da15c45f73ac","Type":"ContainerStarted","Data":"7b8ec7c89824019d790eb042e07effbe8bdae79af06ea191c09de24234971ff0"}
Nov 03 01:22:59.839490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:59.839470    5508 generic.go:356] "Generic (PLEG): container finished" podID="71ffe894-0908-4722-b283-5d5dc7442391" containerID="c6ca85a3d5f6168813feb1be945c285edc85f95ed5e5726548fab8b832d3a685" exitCode=0
Nov 03 01:22:59.839566 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:59.839546    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf" event={"ID":"71ffe894-0908-4722-b283-5d5dc7442391","Type":"ContainerDied","Data":"c6ca85a3d5f6168813feb1be945c285edc85f95ed5e5726548fab8b832d3a685"}
Nov 03 01:22:59.841299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:22:59.841278    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j" event={"ID":"7e8a3f80-6450-4566-8b80-6215b2914a0a","Type":"ContainerStarted","Data":"e322675954fea0efe24c6a6c5faf6f8cb2835c87824b5746aa91ae280650aefb"}
Nov 03 01:23:00.032423 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.032366    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"
Nov 03 01:23:00.095839 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.095809    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-kube-api-access-lfk4b\" (UniqueName: \"kubernetes.io/projected/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-kube-api-access-lfk4b\") pod \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\" (UID: \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\") "
Nov 03 01:23:00.095982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.095849    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-util\" (UniqueName: \"kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-util\") pod \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\" (UID: \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\") "
Nov 03 01:23:00.095982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.095884    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-bundle\" (UniqueName: \"kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-bundle\") pod \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\" (UID: \"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22\") "
Nov 03 01:23:00.096471 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.096441    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-bundle" pod "da6ecd27-0c07-4451-a8fa-c7f74eb2fe22" (UID: "da6ecd27-0c07-4451-a8fa-c7f74eb2fe22"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:00.097896 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.097876    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-kube-api-access-lfk4b" pod "da6ecd27-0c07-4451-a8fa-c7f74eb2fe22" (UID: "da6ecd27-0c07-4451-a8fa-c7f74eb2fe22"). InnerVolumeSpecName "kube-api-access-lfk4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:23:00.100982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.100958    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-util" pod "da6ecd27-0c07-4451-a8fa-c7f74eb2fe22" (UID: "da6ecd27-0c07-4451-a8fa-c7f74eb2fe22"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:00.166432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.166386    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j" podStartSLOduration=4.2787986700000005 podStartE2EDuration="35.166370947s" totalImagesPullingTime="30.887572277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:00.165676503 +0000 UTC m=+259.591282978" watchObservedRunningTime="2026-11-03 01:23:00.166370947 +0000 UTC m=+259.591977409"
Nov 03 01:23:00.197285 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.197257    5508 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-bundle\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:00.197398 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.197289    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfk4b\" (UniqueName: \"kubernetes.io/projected/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-kube-api-access-lfk4b\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:00.197398 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.197304    5508 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/da6ecd27-0c07-4451-a8fa-c7f74eb2fe22-util\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:00.276128 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.276085    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp" podStartSLOduration=4.662291464 podStartE2EDuration="33.276072062s" totalImagesPullingTime="28.613780598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:00.274860372 +0000 UTC m=+259.700466858" watchObservedRunningTime="2026-11-03 01:23:00.276072062 +0000 UTC m=+259.701678547"
Nov 03 01:23:00.847322 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.847296    5508 generic.go:356] "Generic (PLEG): container finished" podID="7e8a3f80-6450-4566-8b80-6215b2914a0a" containerID="e322675954fea0efe24c6a6c5faf6f8cb2835c87824b5746aa91ae280650aefb" exitCode=0
Nov 03 01:23:00.847737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.847381    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j" event={"ID":"7e8a3f80-6450-4566-8b80-6215b2914a0a","Type":"ContainerDied","Data":"e322675954fea0efe24c6a6c5faf6f8cb2835c87824b5746aa91ae280650aefb"}
Nov 03 01:23:00.849149 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.849132    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt"
Nov 03 01:23:00.849149 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.849143    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jxjkt" event={"ID":"da6ecd27-0c07-4451-a8fa-c7f74eb2fe22","Type":"ContainerDied","Data":"0a6e20e1453dcef41ca884664eaf99a723348732b5538a4bc75a66c350ddf5a1"}
Nov 03 01:23:00.849302 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.849174    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a6e20e1453dcef41ca884664eaf99a723348732b5538a4bc75a66c350ddf5a1"
Nov 03 01:23:00.851553 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.851532    5508 generic.go:356] "Generic (PLEG): container finished" podID="3a212187-7678-4368-9b6a-da15c45f73ac" containerID="7b8ec7c89824019d790eb042e07effbe8bdae79af06ea191c09de24234971ff0" exitCode=0
Nov 03 01:23:00.851665 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.851618    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp" event={"ID":"3a212187-7678-4368-9b6a-da15c45f73ac","Type":"ContainerDied","Data":"7b8ec7c89824019d790eb042e07effbe8bdae79af06ea191c09de24234971ff0"}
Nov 03 01:23:00.853595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.853571    5508 generic.go:356] "Generic (PLEG): container finished" podID="0c642bfc-c879-4cb1-83e5-75cf48f2427e" containerID="0faa3a722cf3485c44f97f5449cf54a33c9d6ef1c7a5c340843ee3e335a1f0b1" exitCode=0
Nov 03 01:23:00.853721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.853595    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k" event={"ID":"0c642bfc-c879-4cb1-83e5-75cf48f2427e","Type":"ContainerDied","Data":"0faa3a722cf3485c44f97f5449cf54a33c9d6ef1c7a5c340843ee3e335a1f0b1"}
Nov 03 01:23:00.977565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:00.977545    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"
Nov 03 01:23:01.002419 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.002400    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"
Nov 03 01:23:01.103540 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.103520    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-bundle\" (UniqueName: \"kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-bundle\") pod \"71ffe894-0908-4722-b283-5d5dc7442391\" (UID: \"71ffe894-0908-4722-b283-5d5dc7442391\") "
Nov 03 01:23:01.103651 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.103561    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac570087-5c20-4ba8-91ce-013f501a6337-kube-api-access-6bvlq\" (UniqueName: \"kubernetes.io/projected/ac570087-5c20-4ba8-91ce-013f501a6337-kube-api-access-6bvlq\") pod \"ac570087-5c20-4ba8-91ce-013f501a6337\" (UID: \"ac570087-5c20-4ba8-91ce-013f501a6337\") "
Nov 03 01:23:01.103651 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.103629    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-util\" (UniqueName: \"kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-util\") pod \"71ffe894-0908-4722-b283-5d5dc7442391\" (UID: \"71ffe894-0908-4722-b283-5d5dc7442391\") "
Nov 03 01:23:01.103718 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.103652    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-util\" (UniqueName: \"kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-util\") pod \"ac570087-5c20-4ba8-91ce-013f501a6337\" (UID: \"ac570087-5c20-4ba8-91ce-013f501a6337\") "
Nov 03 01:23:01.103718 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.103694    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-bundle\") pod \"ac570087-5c20-4ba8-91ce-013f501a6337\" (UID: \"ac570087-5c20-4ba8-91ce-013f501a6337\") "
Nov 03 01:23:01.103797 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.103728    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ffe894-0908-4722-b283-5d5dc7442391-kube-api-access-gntll\" (UniqueName: \"kubernetes.io/projected/71ffe894-0908-4722-b283-5d5dc7442391-kube-api-access-gntll\") pod \"71ffe894-0908-4722-b283-5d5dc7442391\" (UID: \"71ffe894-0908-4722-b283-5d5dc7442391\") "
Nov 03 01:23:01.104231 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.104199    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-bundle" pod "ac570087-5c20-4ba8-91ce-013f501a6337" (UID: "ac570087-5c20-4ba8-91ce-013f501a6337"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:01.104532 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.104504    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-bundle" pod "71ffe894-0908-4722-b283-5d5dc7442391" (UID: "71ffe894-0908-4722-b283-5d5dc7442391"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:01.105632 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.105607    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac570087-5c20-4ba8-91ce-013f501a6337-kube-api-access-6bvlq" pod "ac570087-5c20-4ba8-91ce-013f501a6337" (UID: "ac570087-5c20-4ba8-91ce-013f501a6337"). InnerVolumeSpecName "kube-api-access-6bvlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:23:01.105853 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.105828    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ffe894-0908-4722-b283-5d5dc7442391-kube-api-access-gntll" pod "71ffe894-0908-4722-b283-5d5dc7442391" (UID: "71ffe894-0908-4722-b283-5d5dc7442391"). InnerVolumeSpecName "kube-api-access-gntll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:23:01.109062 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.109044    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-util" pod "71ffe894-0908-4722-b283-5d5dc7442391" (UID: "71ffe894-0908-4722-b283-5d5dc7442391"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:01.109436 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.109420    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-util" pod "ac570087-5c20-4ba8-91ce-013f501a6337" (UID: "ac570087-5c20-4ba8-91ce-013f501a6337"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:01.204681 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.204650    5508 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-util\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:01.204681 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.204676    5508 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-util\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:01.204681 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.204685    5508 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac570087-5c20-4ba8-91ce-013f501a6337-bundle\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:01.204896 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.204694    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gntll\" (UniqueName: \"kubernetes.io/projected/71ffe894-0908-4722-b283-5d5dc7442391-kube-api-access-gntll\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:01.204896 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.204703    5508 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71ffe894-0908-4722-b283-5d5dc7442391-bundle\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:01.204896 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.204712    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bvlq\" (UniqueName: \"kubernetes.io/projected/ac570087-5c20-4ba8-91ce-013f501a6337-kube-api-access-6bvlq\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:01.859401 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.859371    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d"
Nov 03 01:23:01.859401 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.859380    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bgh5d" event={"ID":"ac570087-5c20-4ba8-91ce-013f501a6337","Type":"ContainerDied","Data":"05fb29c2e1da96e3d1c7becf4ddaeb1d5b20f5c33f38dc784279fffaa23800c3"}
Nov 03 01:23:01.859826 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.859431    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05fb29c2e1da96e3d1c7becf4ddaeb1d5b20f5c33f38dc784279fffaa23800c3"
Nov 03 01:23:01.861365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.861314    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf" event={"ID":"71ffe894-0908-4722-b283-5d5dc7442391","Type":"ContainerDied","Data":"917f93025d56264583c2eef2c2fec3213d283d7284b4b4dbc1799bd37941c36d"}
Nov 03 01:23:01.861472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.861373    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azq9rf"
Nov 03 01:23:01.861472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.861389    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="917f93025d56264583c2eef2c2fec3213d283d7284b4b4dbc1799bd37941c36d"
Nov 03 01:23:01.989765 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:01.989741    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"
Nov 03 01:23:02.022407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.022386    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"
Nov 03 01:23:02.025117 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.025098    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"
Nov 03 01:23:02.111925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.111856    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8a3f80-6450-4566-8b80-6215b2914a0a-kube-api-access-8fm42\" (UniqueName: \"kubernetes.io/projected/7e8a3f80-6450-4566-8b80-6215b2914a0a-kube-api-access-8fm42\") pod \"7e8a3f80-6450-4566-8b80-6215b2914a0a\" (UID: \"7e8a3f80-6450-4566-8b80-6215b2914a0a\") "
Nov 03 01:23:02.111925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.111887    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-bundle\") pod \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\" (UID: \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\") "
Nov 03 01:23:02.112114 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.111929    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a212187-7678-4368-9b6a-da15c45f73ac-kube-api-access-rzdkr\" (UniqueName: \"kubernetes.io/projected/3a212187-7678-4368-9b6a-da15c45f73ac-kube-api-access-rzdkr\") pod \"3a212187-7678-4368-9b6a-da15c45f73ac\" (UID: \"3a212187-7678-4368-9b6a-da15c45f73ac\") "
Nov 03 01:23:02.112114 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.111957    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-bundle\") pod \"7e8a3f80-6450-4566-8b80-6215b2914a0a\" (UID: \"7e8a3f80-6450-4566-8b80-6215b2914a0a\") "
Nov 03 01:23:02.112114 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.111995    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c642bfc-c879-4cb1-83e5-75cf48f2427e-kube-api-access-nhtnk\" (UniqueName: \"kubernetes.io/projected/0c642bfc-c879-4cb1-83e5-75cf48f2427e-kube-api-access-nhtnk\") pod \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\" (UID: \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\") "
Nov 03 01:23:02.112114 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.112033    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-util\" (UniqueName: \"kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-util\") pod \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\" (UID: \"0c642bfc-c879-4cb1-83e5-75cf48f2427e\") "
Nov 03 01:23:02.112114 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.112094    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-util\" (UniqueName: \"kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-util\") pod \"3a212187-7678-4368-9b6a-da15c45f73ac\" (UID: \"3a212187-7678-4368-9b6a-da15c45f73ac\") "
Nov 03 01:23:02.112415 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.112125    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-bundle\") pod \"3a212187-7678-4368-9b6a-da15c45f73ac\" (UID: \"3a212187-7678-4368-9b6a-da15c45f73ac\") "
Nov 03 01:23:02.112415 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.112160    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-util\" (UniqueName: \"kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-util\") pod \"7e8a3f80-6450-4566-8b80-6215b2914a0a\" (UID: \"7e8a3f80-6450-4566-8b80-6215b2914a0a\") "
Nov 03 01:23:02.112415 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.112285    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-bundle" pod "0c642bfc-c879-4cb1-83e5-75cf48f2427e" (UID: "0c642bfc-c879-4cb1-83e5-75cf48f2427e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:02.112667 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.112648    5508 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-bundle\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:02.113018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.112981    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-bundle" pod "3a212187-7678-4368-9b6a-da15c45f73ac" (UID: "3a212187-7678-4368-9b6a-da15c45f73ac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:02.114093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.114072    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-bundle" pod "7e8a3f80-6450-4566-8b80-6215b2914a0a" (UID: "7e8a3f80-6450-4566-8b80-6215b2914a0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:02.114442 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.114413    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c642bfc-c879-4cb1-83e5-75cf48f2427e-kube-api-access-nhtnk" pod "0c642bfc-c879-4cb1-83e5-75cf48f2427e" (UID: "0c642bfc-c879-4cb1-83e5-75cf48f2427e"). InnerVolumeSpecName "kube-api-access-nhtnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:23:02.114819 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.114801    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8a3f80-6450-4566-8b80-6215b2914a0a-kube-api-access-8fm42" pod "7e8a3f80-6450-4566-8b80-6215b2914a0a" (UID: "7e8a3f80-6450-4566-8b80-6215b2914a0a"). InnerVolumeSpecName "kube-api-access-8fm42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:23:02.114954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.114932    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a212187-7678-4368-9b6a-da15c45f73ac-kube-api-access-rzdkr" pod "3a212187-7678-4368-9b6a-da15c45f73ac" (UID: "3a212187-7678-4368-9b6a-da15c45f73ac"). InnerVolumeSpecName "kube-api-access-rzdkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:23:02.119650 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.119621    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-util" pod "3a212187-7678-4368-9b6a-da15c45f73ac" (UID: "3a212187-7678-4368-9b6a-da15c45f73ac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:02.119783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.119765    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-util" pod "0c642bfc-c879-4cb1-83e5-75cf48f2427e" (UID: "0c642bfc-c879-4cb1-83e5-75cf48f2427e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:02.120356 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.120327    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-util" pod "7e8a3f80-6450-4566-8b80-6215b2914a0a" (UID: "7e8a3f80-6450-4566-8b80-6215b2914a0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:02.213400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.213369    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fm42\" (UniqueName: \"kubernetes.io/projected/7e8a3f80-6450-4566-8b80-6215b2914a0a-kube-api-access-8fm42\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:02.213400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.213395    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzdkr\" (UniqueName: \"kubernetes.io/projected/3a212187-7678-4368-9b6a-da15c45f73ac-kube-api-access-rzdkr\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:02.213633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.213431    5508 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-bundle\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:02.213633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.213443    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhtnk\" (UniqueName: \"kubernetes.io/projected/0c642bfc-c879-4cb1-83e5-75cf48f2427e-kube-api-access-nhtnk\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:02.213633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.213452    5508 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c642bfc-c879-4cb1-83e5-75cf48f2427e-util\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:02.213633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.213461    5508 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-util\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:02.213633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.213469    5508 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a212187-7678-4368-9b6a-da15c45f73ac-bundle\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:02.213633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.213476    5508 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e8a3f80-6450-4566-8b80-6215b2914a0a-util\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:02.689472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689443    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"]
Nov 03 01:23:02.689727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689715    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0c642bfc-c879-4cb1-83e5-75cf48f2427e" containerName="extract"
Nov 03 01:23:02.689769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689729    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c642bfc-c879-4cb1-83e5-75cf48f2427e" containerName="extract"
Nov 03 01:23:02.689769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689741    5508 cpu_manager.go:414] "containerMap: removing container" podUID="06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c" containerName="util"
Nov 03 01:23:02.689769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689746    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c" containerName="util"
Nov 03 01:23:02.689769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689752    5508 cpu_manager.go:414] "containerMap: removing container" podUID="da6ecd27-0c07-4451-a8fa-c7f74eb2fe22" containerName="util"
Nov 03 01:23:02.689769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689757    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6ecd27-0c07-4451-a8fa-c7f74eb2fe22" containerName="util"
Nov 03 01:23:02.689769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689764    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3a212187-7678-4368-9b6a-da15c45f73ac" containerName="util"
Nov 03 01:23:02.689769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689769    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a212187-7678-4368-9b6a-da15c45f73ac" containerName="util"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689777    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3a212187-7678-4368-9b6a-da15c45f73ac" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689781    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a212187-7678-4368-9b6a-da15c45f73ac" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689789    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ac570087-5c20-4ba8-91ce-013f501a6337" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689794    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac570087-5c20-4ba8-91ce-013f501a6337" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689799    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ac570087-5c20-4ba8-91ce-013f501a6337" containerName="extract"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689803    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac570087-5c20-4ba8-91ce-013f501a6337" containerName="extract"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689812    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ac570087-5c20-4ba8-91ce-013f501a6337" containerName="util"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689816    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac570087-5c20-4ba8-91ce-013f501a6337" containerName="util"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689822    5508 cpu_manager.go:414] "containerMap: removing container" podUID="71ffe894-0908-4722-b283-5d5dc7442391" containerName="extract"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689826    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ffe894-0908-4722-b283-5d5dc7442391" containerName="extract"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689835    5508 cpu_manager.go:414] "containerMap: removing container" podUID="06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c" containerName="extract"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689839    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c" containerName="extract"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689850    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7e8a3f80-6450-4566-8b80-6215b2914a0a" containerName="util"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689854    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8a3f80-6450-4566-8b80-6215b2914a0a" containerName="util"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689860    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0c642bfc-c879-4cb1-83e5-75cf48f2427e" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689864    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c642bfc-c879-4cb1-83e5-75cf48f2427e" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689871    5508 cpu_manager.go:414] "containerMap: removing container" podUID="71ffe894-0908-4722-b283-5d5dc7442391" containerName="util"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689875    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ffe894-0908-4722-b283-5d5dc7442391" containerName="util"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689881    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0c642bfc-c879-4cb1-83e5-75cf48f2427e" containerName="util"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689885    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c642bfc-c879-4cb1-83e5-75cf48f2427e" containerName="util"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689894    5508 cpu_manager.go:414] "containerMap: removing container" podUID="71ffe894-0908-4722-b283-5d5dc7442391" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689901    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ffe894-0908-4722-b283-5d5dc7442391" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689908    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7e8a3f80-6450-4566-8b80-6215b2914a0a" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689912    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8a3f80-6450-4566-8b80-6215b2914a0a" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689918    5508 cpu_manager.go:414] "containerMap: removing container" podUID="da6ecd27-0c07-4451-a8fa-c7f74eb2fe22" containerName="extract"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689922    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6ecd27-0c07-4451-a8fa-c7f74eb2fe22" containerName="extract"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689931    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7e8a3f80-6450-4566-8b80-6215b2914a0a" containerName="extract"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689939    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8a3f80-6450-4566-8b80-6215b2914a0a" containerName="extract"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689950    5508 cpu_manager.go:414] "containerMap: removing container" podUID="06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689955    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689961    5508 cpu_manager.go:414] "containerMap: removing container" podUID="da6ecd27-0c07-4451-a8fa-c7f74eb2fe22" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689965    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6ecd27-0c07-4451-a8fa-c7f74eb2fe22" containerName="pull"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689970    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3a212187-7678-4368-9b6a-da15c45f73ac" containerName="extract"
Nov 03 01:23:02.689988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.689974    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a212187-7678-4368-9b6a-da15c45f73ac" containerName="extract"
Nov 03 01:23:02.691132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.690025    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8a3f80-6450-4566-8b80-6215b2914a0a" containerName="extract"
Nov 03 01:23:02.691132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.690032    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6ecd27-0c07-4451-a8fa-c7f74eb2fe22" containerName="extract"
Nov 03 01:23:02.691132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.690038    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a212187-7678-4368-9b6a-da15c45f73ac" containerName="extract"
Nov 03 01:23:02.691132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.690044    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c642bfc-c879-4cb1-83e5-75cf48f2427e" containerName="extract"
Nov 03 01:23:02.691132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.690050    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac570087-5c20-4ba8-91ce-013f501a6337" containerName="extract"
Nov 03 01:23:02.691132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.690055    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ffe894-0908-4722-b283-5d5dc7442391" containerName="extract"
Nov 03 01:23:02.691132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.690060    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c74c0c-4f1a-4563-8e6d-f0495bfa0a4c" containerName="extract"
Nov 03 01:23:02.694383 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.694367    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.743689 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.743668    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 01:23:02.743689 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.743668    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 01:23:02.771957 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.771933    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 01:23:02.772055 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.771982    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 01:23:02.772239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.772222    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rqx2g\""
Nov 03 01:23:02.772313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.772302    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 01:23:02.818737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.818718    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-webhook-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.818857 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.818755    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-manifests\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.818951 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.818930    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-tmp\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.819059 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.819044    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mtwd\" (UniqueName: \"kubernetes.io/projected/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-kube-api-access-7mtwd\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.819120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.819083    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.819198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.819174    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-metrics-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.867134 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.867117    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j"
Nov 03 01:23:02.867458 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.867131    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cscd7j" event={"ID":"7e8a3f80-6450-4566-8b80-6215b2914a0a","Type":"ContainerDied","Data":"29db3d1facc97d2bd7c0909ec607159e76f9454c654ae4b9d9ee0c0c03bf4d3a"}
Nov 03 01:23:02.867458 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.867170    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29db3d1facc97d2bd7c0909ec607159e76f9454c654ae4b9d9ee0c0c03bf4d3a"
Nov 03 01:23:02.869128 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.869107    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp"
Nov 03 01:23:02.869243 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.869127    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cc4swp" event={"ID":"3a212187-7678-4368-9b6a-da15c45f73ac","Type":"ContainerDied","Data":"b17da8a2436c24b101821063c17d81d67f033e6ff3f71c85a9b9c69df7bc2c0c"}
Nov 03 01:23:02.869243 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.869168    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b17da8a2436c24b101821063c17d81d67f033e6ff3f71c85a9b9c69df7bc2c0c"
Nov 03 01:23:02.871077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.871056    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k" event={"ID":"0c642bfc-c879-4cb1-83e5-75cf48f2427e","Type":"ContainerDied","Data":"dd3a063be953006b48ac3243a95fb9d6312b40d5a9a8eba56cfa7a0d92d81bf7"}
Nov 03 01:23:02.871153 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.871077    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446fc4k"
Nov 03 01:23:02.871153 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.871091    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd3a063be953006b48ac3243a95fb9d6312b40d5a9a8eba56cfa7a0d92d81bf7"
Nov 03 01:23:02.919564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.919538    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-tmp\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.919673 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.919582    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mtwd\" (UniqueName: \"kubernetes.io/projected/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-kube-api-access-7mtwd\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.919673 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.919615    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.919673 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.919660    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-metrics-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.919825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.919790    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-webhook-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.919877 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.919829    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-manifests\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.920147 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.920122    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-manifests\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.921783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.921738    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-tmp\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.921946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.921930    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.922344 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.922324    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-webhook-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.922433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.922411    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-metrics-cert\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:02.935842 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.935814    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"]
Nov 03 01:23:02.976833 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:02.976782    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mtwd\" (UniqueName: \"kubernetes.io/projected/7bae5fbb-f54b-41ae-91a6-3917a0e40e9d-kube-api-access-7mtwd\") pod \"opendatahub-operator-controller-manager-797bb7c459-nc8cr\" (UID: \"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:03.012809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:03.012786    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:03.505405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:03.505317    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"]
Nov 03 01:23:03.528159 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:23:03.528131    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bae5fbb_f54b_41ae_91a6_3917a0e40e9d.slice/crio-b8ba7731dce353ca09ebd44fc2a6b593ad3b58793fde4c43f312ca07eaa3f91f WatchSource:0}: Error finding container b8ba7731dce353ca09ebd44fc2a6b593ad3b58793fde4c43f312ca07eaa3f91f: Status 404 returned error can't find the container with id b8ba7731dce353ca09ebd44fc2a6b593ad3b58793fde4c43f312ca07eaa3f91f
Nov 03 01:23:03.881908 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:03.881824    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr" event={"ID":"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d","Type":"ContainerStarted","Data":"b8ba7731dce353ca09ebd44fc2a6b593ad3b58793fde4c43f312ca07eaa3f91f"}
Nov 03 01:23:08.906715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:08.906681    5508 generic.go:356] "Generic (PLEG): container finished" podID="7bae5fbb-f54b-41ae-91a6-3917a0e40e9d" containerID="4896ef43b7f8c8d01816bceb2d410a9a16fe45c18f1fbd24b4011a7348821a61" exitCode=0
Nov 03 01:23:08.907164 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:08.906761    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr" event={"ID":"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d","Type":"ContainerDied","Data":"4896ef43b7f8c8d01816bceb2d410a9a16fe45c18f1fbd24b4011a7348821a61"}
Nov 03 01:23:08.924062 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:08.924041    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79"]
Nov 03 01:23:08.927388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:08.927375    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79"
Nov 03 01:23:09.015946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.015912    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:23:09.050799 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.050776    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 01:23:09.051013 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.050994    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-sh8t9\""
Nov 03 01:23:09.052141 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.052123    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79"]
Nov 03 01:23:09.072678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.072658    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ddfb216-f909-4945-9d1d-37c381afe636-tmp\") pod \"jobset-operator-674cd59dc5-9bp79\" (UID: \"0ddfb216-f909-4945-9d1d-37c381afe636\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79"
Nov 03 01:23:09.072748 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.072705    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6spr\" (UniqueName: \"kubernetes.io/projected/0ddfb216-f909-4945-9d1d-37c381afe636-kube-api-access-c6spr\") pod \"jobset-operator-674cd59dc5-9bp79\" (UID: \"0ddfb216-f909-4945-9d1d-37c381afe636\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79"
Nov 03 01:23:09.173186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.173113    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6spr\" (UniqueName: \"kubernetes.io/projected/0ddfb216-f909-4945-9d1d-37c381afe636-kube-api-access-c6spr\") pod \"jobset-operator-674cd59dc5-9bp79\" (UID: \"0ddfb216-f909-4945-9d1d-37c381afe636\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79"
Nov 03 01:23:09.173186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.173173    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ddfb216-f909-4945-9d1d-37c381afe636-tmp\") pod \"jobset-operator-674cd59dc5-9bp79\" (UID: \"0ddfb216-f909-4945-9d1d-37c381afe636\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79"
Nov 03 01:23:09.173532 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.173516    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ddfb216-f909-4945-9d1d-37c381afe636-tmp\") pod \"jobset-operator-674cd59dc5-9bp79\" (UID: \"0ddfb216-f909-4945-9d1d-37c381afe636\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79"
Nov 03 01:23:09.234084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.234048    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6spr\" (UniqueName: \"kubernetes.io/projected/0ddfb216-f909-4945-9d1d-37c381afe636-kube-api-access-c6spr\") pod \"jobset-operator-674cd59dc5-9bp79\" (UID: \"0ddfb216-f909-4945-9d1d-37c381afe636\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79"
Nov 03 01:23:09.245897 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.245875    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79"
Nov 03 01:23:09.618774 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.618703    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79"]
Nov 03 01:23:09.626522 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:23:09.626496    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ddfb216_f909_4945_9d1d_37c381afe636.slice/crio-b78b269528c66a3feade6560eff42498a50986ef6b71a4c9d61e183cc665a5ee WatchSource:0}: Error finding container b78b269528c66a3feade6560eff42498a50986ef6b71a4c9d61e183cc665a5ee: Status 404 returned error can't find the container with id b78b269528c66a3feade6560eff42498a50986ef6b71a4c9d61e183cc665a5ee
Nov 03 01:23:09.912514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.912483    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79" event={"ID":"0ddfb216-f909-4945-9d1d-37c381afe636","Type":"ContainerStarted","Data":"b78b269528c66a3feade6560eff42498a50986ef6b71a4c9d61e183cc665a5ee"}
Nov 03 01:23:09.914374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.914333    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr" event={"ID":"7bae5fbb-f54b-41ae-91a6-3917a0e40e9d","Type":"ContainerStarted","Data":"8cdd1012ecb10a25d3b8994ec9c95a7aaf091a49479fd5fa36a33a841e86fa84"}
Nov 03 01:23:09.914505 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:09.914494    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:10.095757 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:10.095706    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr" podStartSLOduration=2.375382913 podStartE2EDuration="8.0956926s" totalImagesPullingTime="5.317410248s" totalInitContainerRuntime="402.899439ms" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:10.094409627 +0000 UTC m=+269.520016091" watchObservedRunningTime="2026-11-03 01:23:10.0956926 +0000 UTC m=+269.521299063"
Nov 03 01:23:13.809727 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:23:13.809683    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:23:13.932079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:13.932044    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79" event={"ID":"0ddfb216-f909-4945-9d1d-37c381afe636","Type":"ContainerStarted","Data":"9cce59ef589bab65e910534bfbf417ac113ad910763656056da2044d5e02f7de"}
Nov 03 01:23:14.079629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:14.079531    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-9bp79" podStartSLOduration=2.806509131 podStartE2EDuration="6.079515321s" totalImagesPullingTime="3.27300619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:14.077965981 +0000 UTC m=+273.503572444" watchObservedRunningTime="2026-11-03 01:23:14.079515321 +0000 UTC m=+273.505121785"
Nov 03 01:23:18.893357 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:18.893276    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw"]
Nov 03 01:23:18.896508 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:18.896486    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw"
Nov 03 01:23:18.946635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:18.946609    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 01:23:18.946635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:18.946629    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 01:23:18.992581 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:18.992543    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-wrrhl\""
Nov 03 01:23:18.994171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:18.994152    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw"]
Nov 03 01:23:19.055297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:19.055274    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be97d124-1c04-42d2-8607-e012bb44f0c7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw\" (UID: \"be97d124-1c04-42d2-8607-e012bb44f0c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw"
Nov 03 01:23:19.055440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:19.055315    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkp9x\" (UniqueName: \"kubernetes.io/projected/be97d124-1c04-42d2-8607-e012bb44f0c7-kube-api-access-wkp9x\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw\" (UID: \"be97d124-1c04-42d2-8607-e012bb44f0c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw"
Nov 03 01:23:19.156627 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:19.156608    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be97d124-1c04-42d2-8607-e012bb44f0c7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw\" (UID: \"be97d124-1c04-42d2-8607-e012bb44f0c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw"
Nov 03 01:23:19.156724 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:19.156643    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkp9x\" (UniqueName: \"kubernetes.io/projected/be97d124-1c04-42d2-8607-e012bb44f0c7-kube-api-access-wkp9x\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw\" (UID: \"be97d124-1c04-42d2-8607-e012bb44f0c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw"
Nov 03 01:23:19.156982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:19.156965    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be97d124-1c04-42d2-8607-e012bb44f0c7-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw\" (UID: \"be97d124-1c04-42d2-8607-e012bb44f0c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw"
Nov 03 01:23:19.217592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:19.217564    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkp9x\" (UniqueName: \"kubernetes.io/projected/be97d124-1c04-42d2-8607-e012bb44f0c7-kube-api-access-wkp9x\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw\" (UID: \"be97d124-1c04-42d2-8607-e012bb44f0c7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw"
Nov 03 01:23:19.508065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:19.508003    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw"
Nov 03 01:23:19.995991 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:23:19.995968    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe97d124_1c04_42d2_8607_e012bb44f0c7.slice/crio-e4d1ef0cef10f4b4721d63b309dc02d23193967a7a5421ef790ea695310ade49 WatchSource:0}: Error finding container e4d1ef0cef10f4b4721d63b309dc02d23193967a7a5421ef790ea695310ade49: Status 404 returned error can't find the container with id e4d1ef0cef10f4b4721d63b309dc02d23193967a7a5421ef790ea695310ade49
Nov 03 01:23:20.034301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:20.034273    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw"]
Nov 03 01:23:20.920634 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:20.920610    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-797bb7c459-nc8cr"
Nov 03 01:23:20.960571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:20.960541    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw" event={"ID":"be97d124-1c04-42d2-8607-e012bb44f0c7","Type":"ContainerStarted","Data":"e4d1ef0cef10f4b4721d63b309dc02d23193967a7a5421ef790ea695310ade49"}
Nov 03 01:23:23.974629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:23.974592    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw" event={"ID":"be97d124-1c04-42d2-8607-e012bb44f0c7","Type":"ContainerStarted","Data":"b9b1d15de2cfbcf66cf2342a58db42f33b93d5146b3b65d7062074e8236c46a9"}
Nov 03 01:23:24.180684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:24.180611    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vdzcw" podStartSLOduration=2.779413923 podStartE2EDuration="6.180593468s" totalImagesPullingTime="3.401179545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:24.132712996 +0000 UTC m=+283.558319478" watchObservedRunningTime="2026-11-03 01:23:24.180593468 +0000 UTC m=+283.606199937"
Nov 03 01:23:27.087208 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.087180    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6"]
Nov 03 01:23:27.093090 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.092227    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6"
Nov 03 01:23:27.210004 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.209981    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-nh555\""
Nov 03 01:23:27.210158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.210091    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 01:23:27.210158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.210151    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 01:23:27.211531 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.211512    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6"]
Nov 03 01:23:27.221493 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.221472    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/802af463-0d24-4f35-99f8-1dd96b6ed704-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qbvm6\" (UID: \"802af463-0d24-4f35-99f8-1dd96b6ed704\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6"
Nov 03 01:23:27.221578 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.221532    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfw6\" (UniqueName: \"kubernetes.io/projected/802af463-0d24-4f35-99f8-1dd96b6ed704-kube-api-access-wmfw6\") pod \"cert-manager-webhook-5dbbbf75cc-qbvm6\" (UID: \"802af463-0d24-4f35-99f8-1dd96b6ed704\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6"
Nov 03 01:23:27.257887 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.257867    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"]
Nov 03 01:23:27.261023 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.261010    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"
Nov 03 01:23:27.310194 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.310172    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 01:23:27.310275 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.310199    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 01:23:27.322094 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.322077    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfw6\" (UniqueName: \"kubernetes.io/projected/802af463-0d24-4f35-99f8-1dd96b6ed704-kube-api-access-wmfw6\") pod \"cert-manager-webhook-5dbbbf75cc-qbvm6\" (UID: \"802af463-0d24-4f35-99f8-1dd96b6ed704\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6"
Nov 03 01:23:27.322179 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.322118    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/802af463-0d24-4f35-99f8-1dd96b6ed704-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qbvm6\" (UID: \"802af463-0d24-4f35-99f8-1dd96b6ed704\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6"
Nov 03 01:23:27.330030 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.330011    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"]
Nov 03 01:23:27.364772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.364709    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pkj7d\""
Nov 03 01:23:27.384796 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.384768    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfw6\" (UniqueName: \"kubernetes.io/projected/802af463-0d24-4f35-99f8-1dd96b6ed704-kube-api-access-wmfw6\") pod \"cert-manager-webhook-5dbbbf75cc-qbvm6\" (UID: \"802af463-0d24-4f35-99f8-1dd96b6ed704\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6"
Nov 03 01:23:27.386083 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.386064    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/802af463-0d24-4f35-99f8-1dd96b6ed704-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-qbvm6\" (UID: \"802af463-0d24-4f35-99f8-1dd96b6ed704\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6"
Nov 03 01:23:27.403283 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.403261    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6"
Nov 03 01:23:27.422865 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.422841    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a99ffc0-6cc1-4188-8878-82a0dce662cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jr6bp\" (UID: \"6a99ffc0-6cc1-4188-8878-82a0dce662cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"
Nov 03 01:23:27.422970 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.422883    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7smm\" (UniqueName: \"kubernetes.io/projected/6a99ffc0-6cc1-4188-8878-82a0dce662cc-kube-api-access-g7smm\") pod \"kuadrant-operator-controller-manager-678d87766f-jr6bp\" (UID: \"6a99ffc0-6cc1-4188-8878-82a0dce662cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"
Nov 03 01:23:27.523735 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.523701    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a99ffc0-6cc1-4188-8878-82a0dce662cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jr6bp\" (UID: \"6a99ffc0-6cc1-4188-8878-82a0dce662cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"
Nov 03 01:23:27.523860 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.523751    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7smm\" (UniqueName: \"kubernetes.io/projected/6a99ffc0-6cc1-4188-8878-82a0dce662cc-kube-api-access-g7smm\") pod \"kuadrant-operator-controller-manager-678d87766f-jr6bp\" (UID: \"6a99ffc0-6cc1-4188-8878-82a0dce662cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"
Nov 03 01:23:27.524041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.524023    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a99ffc0-6cc1-4188-8878-82a0dce662cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jr6bp\" (UID: \"6a99ffc0-6cc1-4188-8878-82a0dce662cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"
Nov 03 01:23:27.580978 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.580954    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7smm\" (UniqueName: \"kubernetes.io/projected/6a99ffc0-6cc1-4188-8878-82a0dce662cc-kube-api-access-g7smm\") pod \"kuadrant-operator-controller-manager-678d87766f-jr6bp\" (UID: \"6a99ffc0-6cc1-4188-8878-82a0dce662cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"
Nov 03 01:23:27.783920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.783894    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6"]
Nov 03 01:23:27.791253 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:23:27.791225    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod802af463_0d24_4f35_99f8_1dd96b6ed704.slice/crio-062c40110d4b51112f8a505ae42d25daced459e6c508190d0431a11eaf6f64d3 WatchSource:0}: Error finding container 062c40110d4b51112f8a505ae42d25daced459e6c508190d0431a11eaf6f64d3: Status 404 returned error can't find the container with id 062c40110d4b51112f8a505ae42d25daced459e6c508190d0431a11eaf6f64d3
Nov 03 01:23:27.873043 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.873017    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"
Nov 03 01:23:27.992921 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:27.992893    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6" event={"ID":"802af463-0d24-4f35-99f8-1dd96b6ed704","Type":"ContainerStarted","Data":"062c40110d4b51112f8a505ae42d25daced459e6c508190d0431a11eaf6f64d3"}
Nov 03 01:23:28.302263 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:28.302236    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"]
Nov 03 01:23:28.310066 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:23:28.310043    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a99ffc0_6cc1_4188_8878_82a0dce662cc.slice/crio-84362349396b3afeeb08a74479b06fd6a15b9496372878c9db86edeb1cf0af96 WatchSource:0}: Error finding container 84362349396b3afeeb08a74479b06fd6a15b9496372878c9db86edeb1cf0af96: Status 404 returned error can't find the container with id 84362349396b3afeeb08a74479b06fd6a15b9496372878c9db86edeb1cf0af96
Nov 03 01:23:29.002021 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:29.001988    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp" event={"ID":"6a99ffc0-6cc1-4188-8878-82a0dce662cc","Type":"ContainerStarted","Data":"84362349396b3afeeb08a74479b06fd6a15b9496372878c9db86edeb1cf0af96"}
Nov 03 01:23:33.026472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:33.026427    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6" event={"ID":"802af463-0d24-4f35-99f8-1dd96b6ed704","Type":"ContainerStarted","Data":"a5084040b8345cf58cd4bf321557a9aecb52320e9d7f3bc942cbfcfe2f25e1ce"}
Nov 03 01:23:33.027150 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:33.026578    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6"
Nov 03 01:23:33.136171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:33.136116    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6" podStartSLOduration=1.571744025 podStartE2EDuration="6.136103719s" totalImagesPullingTime="4.564359694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:33.134781313 +0000 UTC m=+292.560387795" watchObservedRunningTime="2026-11-03 01:23:33.136103719 +0000 UTC m=+292.561710182"
Nov 03 01:23:34.586854 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.586832    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"]
Nov 03 01:23:34.591417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.591401    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"
Nov 03 01:23:34.683162 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.683130    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbkc4\" (UniqueName: \"kubernetes.io/projected/894c4881-96dc-4a44-864b-00d9965f35da-kube-api-access-jbkc4\") pod \"kube-auth-proxy-68dcf9949f-7zlkw\" (UID: \"894c4881-96dc-4a44-864b-00d9965f35da\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"
Nov 03 01:23:34.683302 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.683189    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/894c4881-96dc-4a44-864b-00d9965f35da-tmp\") pod \"kube-auth-proxy-68dcf9949f-7zlkw\" (UID: \"894c4881-96dc-4a44-864b-00d9965f35da\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"
Nov 03 01:23:34.683302 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.683282    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894c4881-96dc-4a44-864b-00d9965f35da-tls-certs\") pod \"kube-auth-proxy-68dcf9949f-7zlkw\" (UID: \"894c4881-96dc-4a44-864b-00d9965f35da\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"
Nov 03 01:23:34.691138 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.691116    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"]
Nov 03 01:23:34.740487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.740461    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 01:23:34.740487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.740484    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-b6fzt\""
Nov 03 01:23:34.740630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.740555    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 01:23:34.784703 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.784675    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/894c4881-96dc-4a44-864b-00d9965f35da-tmp\") pod \"kube-auth-proxy-68dcf9949f-7zlkw\" (UID: \"894c4881-96dc-4a44-864b-00d9965f35da\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"
Nov 03 01:23:34.784831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.784710    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894c4881-96dc-4a44-864b-00d9965f35da-tls-certs\") pod \"kube-auth-proxy-68dcf9949f-7zlkw\" (UID: \"894c4881-96dc-4a44-864b-00d9965f35da\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"
Nov 03 01:23:34.784831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.784777    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbkc4\" (UniqueName: \"kubernetes.io/projected/894c4881-96dc-4a44-864b-00d9965f35da-kube-api-access-jbkc4\") pod \"kube-auth-proxy-68dcf9949f-7zlkw\" (UID: \"894c4881-96dc-4a44-864b-00d9965f35da\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"
Nov 03 01:23:34.786862 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.786843    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/894c4881-96dc-4a44-864b-00d9965f35da-tmp\") pod \"kube-auth-proxy-68dcf9949f-7zlkw\" (UID: \"894c4881-96dc-4a44-864b-00d9965f35da\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"
Nov 03 01:23:34.786999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.786981    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/894c4881-96dc-4a44-864b-00d9965f35da-tls-certs\") pod \"kube-auth-proxy-68dcf9949f-7zlkw\" (UID: \"894c4881-96dc-4a44-864b-00d9965f35da\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"
Nov 03 01:23:34.841846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.841781    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbkc4\" (UniqueName: \"kubernetes.io/projected/894c4881-96dc-4a44-864b-00d9965f35da-kube-api-access-jbkc4\") pod \"kube-auth-proxy-68dcf9949f-7zlkw\" (UID: \"894c4881-96dc-4a44-864b-00d9965f35da\") " pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"
Nov 03 01:23:34.904421 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:34.904396    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"
Nov 03 01:23:35.038311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:35.038279    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp" event={"ID":"6a99ffc0-6cc1-4188-8878-82a0dce662cc","Type":"ContainerStarted","Data":"258e9f52c546ab485f9cf08d5e79a5e64e41f9607d5e03ada49eda63f160ebb3"}
Nov 03 01:23:35.038982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:35.038960    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"
Nov 03 01:23:35.155982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:35.155931    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp" podStartSLOduration=1.903520477 podStartE2EDuration="8.155914555s" totalImagesPullingTime="6.252394078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:35.153968414 +0000 UTC m=+294.579574874" watchObservedRunningTime="2026-11-03 01:23:35.155914555 +0000 UTC m=+294.581521018"
Nov 03 01:23:35.274639 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:35.274610    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw"]
Nov 03 01:23:35.282099 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:23:35.282074    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894c4881_96dc_4a44_864b_00d9965f35da.slice/crio-be4e6d0df75d4cf62453e1f178680498e96f03340cd6ade134279108f2e4fb65 WatchSource:0}: Error finding container be4e6d0df75d4cf62453e1f178680498e96f03340cd6ade134279108f2e4fb65: Status 404 returned error can't find the container with id be4e6d0df75d4cf62453e1f178680498e96f03340cd6ade134279108f2e4fb65
Nov 03 01:23:36.046245 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:36.046208    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw" event={"ID":"894c4881-96dc-4a44-864b-00d9965f35da","Type":"ContainerStarted","Data":"be4e6d0df75d4cf62453e1f178680498e96f03340cd6ade134279108f2e4fb65"}
Nov 03 01:23:39.033851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:39.033820    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6"
Nov 03 01:23:39.063051 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:39.063018    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw" event={"ID":"894c4881-96dc-4a44-864b-00d9965f35da","Type":"ContainerStarted","Data":"acb61f17c0a5aea4bae47ea9dd79ac637689df464b17b2dae46403ddbe2d9d0d"}
Nov 03 01:23:39.254021 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:39.253965    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68dcf9949f-7zlkw" podStartSLOduration=1.960218257 podStartE2EDuration="5.253946406s" totalImagesPullingTime="3.293728149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:39.25355054 +0000 UTC m=+298.679157025" watchObservedRunningTime="2026-11-03 01:23:39.253946406 +0000 UTC m=+298.679552870"
Nov 03 01:23:40.038839 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:40.037829    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nwmk6"]
Nov 03 01:23:40.040607 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:40.040588    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nwmk6"
Nov 03 01:23:40.089498 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:40.089474    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cm5nl\""
Nov 03 01:23:40.090602 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:40.090570    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nwmk6"]
Nov 03 01:23:40.131074 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:40.131052    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4shr7\" (UniqueName: \"kubernetes.io/projected/a4b7d518-d3c0-4202-bb37-14ae448a0f41-kube-api-access-4shr7\") pod \"authorino-57f7f8659f-nwmk6\" (UID: \"a4b7d518-d3c0-4202-bb37-14ae448a0f41\") " pod="kuadrant-system/authorino-57f7f8659f-nwmk6"
Nov 03 01:23:40.231739 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:40.231715    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shr7\" (UniqueName: \"kubernetes.io/projected/a4b7d518-d3c0-4202-bb37-14ae448a0f41-kube-api-access-4shr7\") pod \"authorino-57f7f8659f-nwmk6\" (UID: \"a4b7d518-d3c0-4202-bb37-14ae448a0f41\") " pod="kuadrant-system/authorino-57f7f8659f-nwmk6"
Nov 03 01:23:40.289013 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:40.288959    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4shr7\" (UniqueName: \"kubernetes.io/projected/a4b7d518-d3c0-4202-bb37-14ae448a0f41-kube-api-access-4shr7\") pod \"authorino-57f7f8659f-nwmk6\" (UID: \"a4b7d518-d3c0-4202-bb37-14ae448a0f41\") " pod="kuadrant-system/authorino-57f7f8659f-nwmk6"
Nov 03 01:23:40.351015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:40.350993    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nwmk6"
Nov 03 01:23:40.711762 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:40.711736    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:23:40.712056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:40.712040    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:23:40.715603 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:40.715580    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nwmk6"]
Nov 03 01:23:40.723512 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:23:40.723486    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4b7d518_d3c0_4202_bb37_14ae448a0f41.slice/crio-a54001ada8814ba34c5a3ee3f040ae2defc00672815ab9571f618664c2cf2686 WatchSource:0}: Error finding container a54001ada8814ba34c5a3ee3f040ae2defc00672815ab9571f618664c2cf2686: Status 404 returned error can't find the container with id a54001ada8814ba34c5a3ee3f040ae2defc00672815ab9571f618664c2cf2686
Nov 03 01:23:41.077011 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.076928    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nwmk6" event={"ID":"a4b7d518-d3c0-4202-bb37-14ae448a0f41","Type":"ContainerStarted","Data":"a54001ada8814ba34c5a3ee3f040ae2defc00672815ab9571f618664c2cf2686"}
Nov 03 01:23:41.564927 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.564896    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:23:41.568147 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.568127    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"
Nov 03 01:23:41.643490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.643465    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6904580a-c37d-4058-9f86-dc69f52efdf6-config-file\") pod \"limitador-limitador-7c9b87d574-4pqmh\" (UID: \"6904580a-c37d-4058-9f86-dc69f52efdf6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"
Nov 03 01:23:41.643626 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.643586    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78vn4\" (UniqueName: \"kubernetes.io/projected/6904580a-c37d-4058-9f86-dc69f52efdf6-kube-api-access-78vn4\") pod \"limitador-limitador-7c9b87d574-4pqmh\" (UID: \"6904580a-c37d-4058-9f86-dc69f52efdf6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"
Nov 03 01:23:41.662889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.662870    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 01:23:41.662889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.662871    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8vbrr\""
Nov 03 01:23:41.664720 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.664696    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:23:41.744522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.744498    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78vn4\" (UniqueName: \"kubernetes.io/projected/6904580a-c37d-4058-9f86-dc69f52efdf6-kube-api-access-78vn4\") pod \"limitador-limitador-7c9b87d574-4pqmh\" (UID: \"6904580a-c37d-4058-9f86-dc69f52efdf6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"
Nov 03 01:23:41.744640 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.744560    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6904580a-c37d-4058-9f86-dc69f52efdf6-config-file\") pod \"limitador-limitador-7c9b87d574-4pqmh\" (UID: \"6904580a-c37d-4058-9f86-dc69f52efdf6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"
Nov 03 01:23:41.745400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.745379    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6904580a-c37d-4058-9f86-dc69f52efdf6-config-file\") pod \"limitador-limitador-7c9b87d574-4pqmh\" (UID: \"6904580a-c37d-4058-9f86-dc69f52efdf6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"
Nov 03 01:23:41.802273 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.802246    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78vn4\" (UniqueName: \"kubernetes.io/projected/6904580a-c37d-4058-9f86-dc69f52efdf6-kube-api-access-78vn4\") pod \"limitador-limitador-7c9b87d574-4pqmh\" (UID: \"6904580a-c37d-4058-9f86-dc69f52efdf6\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"
Nov 03 01:23:41.872145 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.872087    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:23:41.878999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:41.878980    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"
Nov 03 01:23:42.243542 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:42.243505    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:23:42.251879 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:23:42.251851    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6904580a_c37d_4058_9f86_dc69f52efdf6.slice/crio-22ef2ed9a40fda3f64ac5c3bcc0e1ad449475cda7e5e494d697eadb9c89cdbae WatchSource:0}: Error finding container 22ef2ed9a40fda3f64ac5c3bcc0e1ad449475cda7e5e494d697eadb9c89cdbae: Status 404 returned error can't find the container with id 22ef2ed9a40fda3f64ac5c3bcc0e1ad449475cda7e5e494d697eadb9c89cdbae
Nov 03 01:23:43.088397 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:43.088365    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pqmh" event={"ID":"6904580a-c37d-4058-9f86-dc69f52efdf6","Type":"ContainerStarted","Data":"22ef2ed9a40fda3f64ac5c3bcc0e1ad449475cda7e5e494d697eadb9c89cdbae"}
Nov 03 01:23:43.817256 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:23:43.817218    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:23:45.099181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:45.099146    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pqmh" event={"ID":"6904580a-c37d-4058-9f86-dc69f52efdf6","Type":"ContainerStarted","Data":"f45bcb95d9a970a9663ef6166b6ef43f6189568bcf85834bffff6d14da00a263"}
Nov 03 01:23:45.099641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:45.099318    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"
Nov 03 01:23:45.100680 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:45.100658    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nwmk6" event={"ID":"a4b7d518-d3c0-4202-bb37-14ae448a0f41","Type":"ContainerStarted","Data":"abd843407104a7198a908ee017d9e3be3b26ad1a71f06916fa13990bbb6e06d6"}
Nov 03 01:23:45.207765 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:45.207707    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pqmh" podStartSLOduration=2.240909459 podStartE2EDuration="4.207688033s" totalImagesPullingTime="1.966778574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:45.207088509 +0000 UTC m=+304.632694975" watchObservedRunningTime="2026-11-03 01:23:45.207688033 +0000 UTC m=+304.633294497"
Nov 03 01:23:45.313870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:45.313813    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-nwmk6" podStartSLOduration=2.761229492 podStartE2EDuration="6.313798726s" totalImagesPullingTime="3.552569234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:45.313050232 +0000 UTC m=+304.738656713" watchObservedRunningTime="2026-11-03 01:23:45.313798726 +0000 UTC m=+304.739405189"
Nov 03 01:23:45.397358 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:45.397317    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nwmk6"]
Nov 03 01:23:47.055321 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:47.055286    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"
Nov 03 01:23:47.109269 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:47.109231    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-nwmk6" podUID="a4b7d518-d3c0-4202-bb37-14ae448a0f41" containerName="authorino" containerID="cri-o://abd843407104a7198a908ee017d9e3be3b26ad1a71f06916fa13990bbb6e06d6" gracePeriod=30
Nov 03 01:23:47.361829 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:47.361805    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nwmk6"
Nov 03 01:23:47.493813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:47.493781    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b7d518-d3c0-4202-bb37-14ae448a0f41-kube-api-access-4shr7\" (UniqueName: \"kubernetes.io/projected/a4b7d518-d3c0-4202-bb37-14ae448a0f41-kube-api-access-4shr7\") pod \"a4b7d518-d3c0-4202-bb37-14ae448a0f41\" (UID: \"a4b7d518-d3c0-4202-bb37-14ae448a0f41\") "
Nov 03 01:23:47.496088 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:47.496061    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b7d518-d3c0-4202-bb37-14ae448a0f41-kube-api-access-4shr7" pod "a4b7d518-d3c0-4202-bb37-14ae448a0f41" (UID: "a4b7d518-d3c0-4202-bb37-14ae448a0f41"). InnerVolumeSpecName "kube-api-access-4shr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:23:47.594696 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:47.594613    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4shr7\" (UniqueName: \"kubernetes.io/projected/a4b7d518-d3c0-4202-bb37-14ae448a0f41-kube-api-access-4shr7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:23:48.120451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:48.120417    5508 generic.go:356] "Generic (PLEG): container finished" podID="a4b7d518-d3c0-4202-bb37-14ae448a0f41" containerID="abd843407104a7198a908ee017d9e3be3b26ad1a71f06916fa13990bbb6e06d6" exitCode=0
Nov 03 01:23:48.120874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:48.120459    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nwmk6"
Nov 03 01:23:48.120874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:48.120480    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nwmk6" event={"ID":"a4b7d518-d3c0-4202-bb37-14ae448a0f41","Type":"ContainerDied","Data":"abd843407104a7198a908ee017d9e3be3b26ad1a71f06916fa13990bbb6e06d6"}
Nov 03 01:23:48.120874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:48.120508    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nwmk6" event={"ID":"a4b7d518-d3c0-4202-bb37-14ae448a0f41","Type":"ContainerDied","Data":"a54001ada8814ba34c5a3ee3f040ae2defc00672815ab9571f618664c2cf2686"}
Nov 03 01:23:48.120874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:48.120524    5508 scope.go:122] "RemoveContainer" containerID="abd843407104a7198a908ee017d9e3be3b26ad1a71f06916fa13990bbb6e06d6"
Nov 03 01:23:48.129109 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:48.129088    5508 scope.go:122] "RemoveContainer" containerID="abd843407104a7198a908ee017d9e3be3b26ad1a71f06916fa13990bbb6e06d6"
Nov 03 01:23:48.129427 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:23:48.129408    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd843407104a7198a908ee017d9e3be3b26ad1a71f06916fa13990bbb6e06d6\": container with ID starting with abd843407104a7198a908ee017d9e3be3b26ad1a71f06916fa13990bbb6e06d6 not found: ID does not exist" containerID="abd843407104a7198a908ee017d9e3be3b26ad1a71f06916fa13990bbb6e06d6"
Nov 03 01:23:48.129483 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:48.129436    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd843407104a7198a908ee017d9e3be3b26ad1a71f06916fa13990bbb6e06d6"} err="failed to get container status \"abd843407104a7198a908ee017d9e3be3b26ad1a71f06916fa13990bbb6e06d6\": rpc error: code = NotFound desc = could not find container \"abd843407104a7198a908ee017d9e3be3b26ad1a71f06916fa13990bbb6e06d6\": container with ID starting with abd843407104a7198a908ee017d9e3be3b26ad1a71f06916fa13990bbb6e06d6 not found: ID does not exist"
Nov 03 01:23:48.285956 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:48.285923    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nwmk6"]
Nov 03 01:23:48.292071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:48.292048    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nwmk6"]
Nov 03 01:23:48.689632 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:48.689604    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b7d518-d3c0-4202-bb37-14ae448a0f41" path="/var/lib/kubelet/pods/a4b7d518-d3c0-4202-bb37-14ae448a0f41/volumes"
Nov 03 01:23:56.106093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:23:56.106064    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"
Nov 03 01:24:13.810533 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:24:13.810494    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:24:34.877751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.877716    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"]
Nov 03 01:24:34.878120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.878061    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a4b7d518-d3c0-4202-bb37-14ae448a0f41" containerName="authorino"
Nov 03 01:24:34.878120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.878071    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b7d518-d3c0-4202-bb37-14ae448a0f41" containerName="authorino"
Nov 03 01:24:34.878188 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.878133    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b7d518-d3c0-4202-bb37-14ae448a0f41" containerName="authorino"
Nov 03 01:24:34.881462 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.881444    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:34.929996 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.929970    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 01:24:34.976341 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.976307    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 01:24:34.976499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.976361    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 01:24:34.976626 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.976611    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 01:24:34.976666 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.976617    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 01:24:34.978294 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.978276    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"]
Nov 03 01:24:34.991792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.991768    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vzvsm\" (UID: \"2bfd7d91-c916-4aa1-882f-a068f42ccfc8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:34.991909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.991804    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vzvsm\" (UID: \"2bfd7d91-c916-4aa1-882f-a068f42ccfc8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:34.991909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.991899    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vzvsm\" (UID: \"2bfd7d91-c916-4aa1-882f-a068f42ccfc8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:34.991978 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:34.991923    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hpfc\" (UniqueName: \"kubernetes.io/projected/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-kube-api-access-4hpfc\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vzvsm\" (UID: \"2bfd7d91-c916-4aa1-882f-a068f42ccfc8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:35.030410 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:35.030383    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-xj2ht\""
Nov 03 01:24:35.093195 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:35.093167    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vzvsm\" (UID: \"2bfd7d91-c916-4aa1-882f-a068f42ccfc8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:35.093326 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:35.093230    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vzvsm\" (UID: \"2bfd7d91-c916-4aa1-882f-a068f42ccfc8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:35.093326 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:35.093264    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpfc\" (UniqueName: \"kubernetes.io/projected/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-kube-api-access-4hpfc\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vzvsm\" (UID: \"2bfd7d91-c916-4aa1-882f-a068f42ccfc8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:35.093425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:35.093335    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vzvsm\" (UID: \"2bfd7d91-c916-4aa1-882f-a068f42ccfc8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:35.093425 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:24:35.093392    5508 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 03 01:24:35.093489 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:24:35.093467    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-proxy-source podName:2bfd7d91-c916-4aa1-882f-a068f42ccfc8 nodeName:}" failed. No retries permitted until 2026-11-03 01:24:35.593448567 +0000 UTC m=+355.019055009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-vzvsm" (UID: "2bfd7d91-c916-4aa1-882f-a068f42ccfc8") : configmap "usage-tenancy-proxy-source" not found
Nov 03 01:24:35.093990 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:35.093972    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vzvsm\" (UID: \"2bfd7d91-c916-4aa1-882f-a068f42ccfc8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:35.095660 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:35.095644    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vzvsm\" (UID: \"2bfd7d91-c916-4aa1-882f-a068f42ccfc8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:35.150995 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:35.150971    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hpfc\" (UniqueName: \"kubernetes.io/projected/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-kube-api-access-4hpfc\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vzvsm\" (UID: \"2bfd7d91-c916-4aa1-882f-a068f42ccfc8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:35.596905 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:35.596830    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vzvsm\" (UID: \"2bfd7d91-c916-4aa1-882f-a068f42ccfc8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:35.597673 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:35.597654    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2bfd7d91-c916-4aa1-882f-a068f42ccfc8-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-vzvsm\" (UID: \"2bfd7d91-c916-4aa1-882f-a068f42ccfc8\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:35.793640 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:35.793589    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:36.162876 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:36.162842    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"]
Nov 03 01:24:36.170963 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:24:36.170935    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfd7d91_c916_4aa1_882f_a068f42ccfc8.slice/crio-725a979b634ed538c0439e1889c22991e8b8c54f6531cd87e08754bc6e962b57 WatchSource:0}: Error finding container 725a979b634ed538c0439e1889c22991e8b8c54f6531cd87e08754bc6e962b57: Status 404 returned error can't find the container with id 725a979b634ed538c0439e1889c22991e8b8c54f6531cd87e08754bc6e962b57
Nov 03 01:24:36.172828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:36.172813    5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:24:36.339202 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:36.339166    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm" event={"ID":"2bfd7d91-c916-4aa1-882f-a068f42ccfc8","Type":"ContainerStarted","Data":"725a979b634ed538c0439e1889c22991e8b8c54f6531cd87e08754bc6e962b57"}
Nov 03 01:24:43.817799 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:24:43.817749    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:24:52.421174 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:52.421138    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm" event={"ID":"2bfd7d91-c916-4aa1-882f-a068f42ccfc8","Type":"ContainerStarted","Data":"b39a1413b1851bd74e650afa1508f49e7d6799119ab595d13bd151e69423fc82"}
Nov 03 01:24:52.421618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:52.421252    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:24:52.532137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:24:52.532091    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm" podStartSLOduration=2.938213358 podStartE2EDuration="18.532075314s" totalImagesPullingTime="15.593861956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:52.530517046 +0000 UTC m=+371.956123511" watchObservedRunningTime="2026-11-03 01:24:52.532075314 +0000 UTC m=+371.957681800"
Nov 03 01:25:03.431907 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:03.431874    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-vzvsm"
Nov 03 01:25:04.523912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.523878    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"]
Nov 03 01:25:04.527168 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.527149    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.575590 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.575561    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 01:25:04.575729 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.575616    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 01:25:04.575729 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.575624    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 01:25:04.575729 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.575664    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h4j6w\""
Nov 03 01:25:04.623779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.623741    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"]
Nov 03 01:25:04.648386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.648339    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e65b09d4-68d0-4223-af89-6423bc81a63a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.648386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.648387    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbbm2\" (UniqueName: \"kubernetes.io/projected/e65b09d4-68d0-4223-af89-6423bc81a63a-kube-api-access-sbbm2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.648577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.648499    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e65b09d4-68d0-4223-af89-6423bc81a63a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.648577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.648537    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65b09d4-68d0-4223-af89-6423bc81a63a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.648577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.648556    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e65b09d4-68d0-4223-af89-6423bc81a63a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.648676 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.648658    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e65b09d4-68d0-4223-af89-6423bc81a63a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.648722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.648711    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65b09d4-68d0-4223-af89-6423bc81a63a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.749101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.749070    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e65b09d4-68d0-4223-af89-6423bc81a63a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.749101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.749101    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbbm2\" (UniqueName: \"kubernetes.io/projected/e65b09d4-68d0-4223-af89-6423bc81a63a-kube-api-access-sbbm2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.749388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.749150    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e65b09d4-68d0-4223-af89-6423bc81a63a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.749388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.749176    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65b09d4-68d0-4223-af89-6423bc81a63a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.749388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.749192    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e65b09d4-68d0-4223-af89-6423bc81a63a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.749388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.749217    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e65b09d4-68d0-4223-af89-6423bc81a63a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.749388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.749264    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65b09d4-68d0-4223-af89-6423bc81a63a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.750058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.750026    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e65b09d4-68d0-4223-af89-6423bc81a63a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.751586 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.751559    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e65b09d4-68d0-4223-af89-6423bc81a63a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.751679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.751611    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65b09d4-68d0-4223-af89-6423bc81a63a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.751718 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.751687    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e65b09d4-68d0-4223-af89-6423bc81a63a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.751801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.751778    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e65b09d4-68d0-4223-af89-6423bc81a63a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.807028 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.806956    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65b09d4-68d0-4223-af89-6423bc81a63a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.807288 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.807261    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbbm2\" (UniqueName: \"kubernetes.io/projected/e65b09d4-68d0-4223-af89-6423bc81a63a-kube-api-access-sbbm2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-44x49\" (UID: \"e65b09d4-68d0-4223-af89-6423bc81a63a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:04.839475 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:04.839449    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:05.262176 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:05.262140    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"]
Nov 03 01:25:05.268436 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:25:05.268402    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65b09d4_68d0_4223_af89_6423bc81a63a.slice/crio-558e0b50ed75126f2eb349bd17ad599592d9265201229968111a54aba6d73c5b WatchSource:0}: Error finding container 558e0b50ed75126f2eb349bd17ad599592d9265201229968111a54aba6d73c5b: Status 404 returned error can't find the container with id 558e0b50ed75126f2eb349bd17ad599592d9265201229968111a54aba6d73c5b
Nov 03 01:25:05.480479 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:05.480448    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49" event={"ID":"e65b09d4-68d0-4223-af89-6423bc81a63a","Type":"ContainerStarted","Data":"558e0b50ed75126f2eb349bd17ad599592d9265201229968111a54aba6d73c5b"}
Nov 03 01:25:09.737968 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:09.737936    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:25:09.738232 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:09.738000    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:25:10.507318 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:10.507275    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49" event={"ID":"e65b09d4-68d0-4223-af89-6423bc81a63a","Type":"ContainerStarted","Data":"70fa83f2d7dc9126228ecc866c3221647aaea7fb8f27ae07dc3a90b2b5b51398"}
Nov 03 01:25:10.507546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:10.507529    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:10.509181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:10.509155    5508 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-44x49 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:25:10.509304 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:10.509204    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49" podUID="e65b09d4-68d0-4223-af89-6423bc81a63a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:25:10.638529 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:10.638470    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49" podStartSLOduration=2.171619646 podStartE2EDuration="6.638457322s" totalImagesPullingTime="4.466837676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:10.636410787 +0000 UTC m=+390.062017270" watchObservedRunningTime="2026-11-03 01:25:10.638457322 +0000 UTC m=+390.064063786"
Nov 03 01:25:11.512677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:11.512650    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-44x49"
Nov 03 01:25:13.810261 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:25:13.810217    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:25:14.900212 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:14.900180    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:25:23.570540 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:23.570463    5508 generic.go:356] "Generic (PLEG): container finished" podID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerID="ad1c888e0d2d24111a812a20b767307a08f7f2f80093c931370712b6334825d6" exitCode=0
Nov 03 01:25:23.570938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:23.570539    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" event={"ID":"d05f8000-1622-4313-89df-5d55ce6b2f81","Type":"ContainerDied","Data":"ad1c888e0d2d24111a812a20b767307a08f7f2f80093c931370712b6334825d6"}
Nov 03 01:25:23.570938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:23.570880    5508 scope.go:122] "RemoveContainer" containerID="ad1c888e0d2d24111a812a20b767307a08f7f2f80093c931370712b6334825d6"
Nov 03 01:25:23.571903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:23.571876    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:25:23.571986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:23.571938    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:25:23.571986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:23.571971    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:25:24.580821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:24.580786    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" event={"ID":"d05f8000-1622-4313-89df-5d55ce6b2f81","Type":"ContainerStarted","Data":"8cd6efdb3124a34d590cbac6058aaad663954c5a1ed68edba25e8fd3e53e0a13"}
Nov 03 01:25:24.989291 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:24.989258    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:25:24.990324 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:24.990299    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-s9q4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Nov 03 01:25:24.990458 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:24.990342    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Nov 03 01:25:25.989572 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:25.989540    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-s9q4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Nov 03 01:25:25.989940 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:25.989589    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Nov 03 01:25:26.681814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:26.681782    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:25:26.989131 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:26.989049    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-s9q4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Nov 03 01:25:26.989131 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:26.989106    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Nov 03 01:25:27.989946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:27.989911    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-s9q4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Nov 03 01:25:27.990327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:27.989974    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Nov 03 01:25:28.989520 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:28.989485    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-s9q4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Nov 03 01:25:28.989702 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:28.989534    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Nov 03 01:25:29.992887 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:29.992860    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:25:29.993739 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:25:29.993722    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:25:43.810963 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:25:43.810925    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:26:13.810936 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:26:13.810889    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:26:43.809429 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:26:43.809394    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:27:13.811357 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:27:13.811301    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:27:43.810691 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:27:43.810648    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:27:46.285975 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:27:46.285893    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:27:56.823275 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:27:56.823239    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:28:04.981518 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:04.981474    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-x2n9z"]
Nov 03 01:28:04.984925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:04.984900    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.033688 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.033655    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 01:28:05.033688 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.033654    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 01:28:05.079731 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.079695    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j8f5n\""
Nov 03 01:28:05.079731 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.079695    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 03 01:28:05.081544 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.081521    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-x2n9z"]
Nov 03 01:28:05.109998 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.109969    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d843061-ce76-4dd2-ad3f-5109467435fb-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.110131 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.110036    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d843061-ce76-4dd2-ad3f-5109467435fb-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.110131 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.110098    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d843061-ce76-4dd2-ad3f-5109467435fb-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.110205 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.110189    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d843061-ce76-4dd2-ad3f-5109467435fb-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.110238 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.110218    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxskz\" (UniqueName: \"kubernetes.io/projected/1d843061-ce76-4dd2-ad3f-5109467435fb-kube-api-access-vxskz\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.110332 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.110318    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d843061-ce76-4dd2-ad3f-5109467435fb-home\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.211272 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.211235    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d843061-ce76-4dd2-ad3f-5109467435fb-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.211272 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.211277    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d843061-ce76-4dd2-ad3f-5109467435fb-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.211535 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.211306    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d843061-ce76-4dd2-ad3f-5109467435fb-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.211535 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.211392    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxskz\" (UniqueName: \"kubernetes.io/projected/1d843061-ce76-4dd2-ad3f-5109467435fb-kube-api-access-vxskz\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.211535 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.211456    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d843061-ce76-4dd2-ad3f-5109467435fb-home\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.211685 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.211549    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d843061-ce76-4dd2-ad3f-5109467435fb-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.211743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.211695    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d843061-ce76-4dd2-ad3f-5109467435fb-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.211849 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.211823    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d843061-ce76-4dd2-ad3f-5109467435fb-home\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.211903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.211883    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d843061-ce76-4dd2-ad3f-5109467435fb-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.213765 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.213739    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d843061-ce76-4dd2-ad3f-5109467435fb-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.213959 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.213939    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d843061-ce76-4dd2-ad3f-5109467435fb-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.270463 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.270395    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxskz\" (UniqueName: \"kubernetes.io/projected/1d843061-ce76-4dd2-ad3f-5109467435fb-kube-api-access-vxskz\") pod \"echo-simulator-kserve-64b6f8f96c-x2n9z\" (UID: \"1d843061-ce76-4dd2-ad3f-5109467435fb\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.299240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.299214    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:05.359244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.359208    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:28:05.669394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:05.669361    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-x2n9z"]
Nov 03 01:28:05.677508 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:28:05.677468    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d843061_ce76_4dd2_ad3f_5109467435fb.slice/crio-b4df8795c25711bd4dd8b4a03af3afdab0c44206852f41572af4dbdfcfd03662 WatchSource:0}: Error finding container b4df8795c25711bd4dd8b4a03af3afdab0c44206852f41572af4dbdfcfd03662: Status 404 returned error can't find the container with id b4df8795c25711bd4dd8b4a03af3afdab0c44206852f41572af4dbdfcfd03662
Nov 03 01:28:06.328423 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:06.328387    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z" event={"ID":"1d843061-ce76-4dd2-ad3f-5109467435fb","Type":"ContainerStarted","Data":"b4df8795c25711bd4dd8b4a03af3afdab0c44206852f41572af4dbdfcfd03662"}
Nov 03 01:28:09.344411 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:09.344376    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z" event={"ID":"1d843061-ce76-4dd2-ad3f-5109467435fb","Type":"ContainerStarted","Data":"7845d2c7a24de58085696083f873dd1e09a22e605ad79ef068e933594cfc9c24"}
Nov 03 01:28:09.455593 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:09.455542    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z" podStartSLOduration=2.806247265 podStartE2EDuration="5.455526442s" totalImagesPullingTime="2.649279177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:09.454361159 +0000 UTC m=+568.879967638" watchObservedRunningTime="2026-11-03 01:28:09.455526442 +0000 UTC m=+568.881132903"
Nov 03 01:28:10.299570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:10.299534    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:13.811404 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:28:13.811372    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:28:15.299437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:15.299400    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:15.312411 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:15.312386    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:15.386223 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:15.386195    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-x2n9z"
Nov 03 01:28:17.383184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.383123    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"]
Nov 03 01:28:17.387061 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.387040    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.422264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.422238    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f26262-d73f-4709-9aba-388786aec407-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.422419 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.422276    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f26262-d73f-4709-9aba-388786aec407-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.422419 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.422298    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.422419 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.422371    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t422n\" (UniqueName: \"kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-kube-api-access-t422n\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.422587 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.422445    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.422587 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.422469    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.422587 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.422500    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.422730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.422618    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.422783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.422734    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.523302 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.523260    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f26262-d73f-4709-9aba-388786aec407-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.523484 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.523318    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.523484 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.523358    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t422n\" (UniqueName: \"kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-kube-api-access-t422n\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.523484 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.523389    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.523484 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.523428    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.523484 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.523458    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.523783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.523503    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.523783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.523593    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.523783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.523641    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f26262-d73f-4709-9aba-388786aec407-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.523883 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.523835    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.524064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.524041    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.524137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.524119    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.524325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.524303    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.524451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.524427    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f26262-d73f-4709-9aba-388786aec407-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.526047 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.526017    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.526169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.526025    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f26262-d73f-4709-9aba-388786aec407-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.580108 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.580071    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.580268 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.580245    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t422n\" (UniqueName: \"kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-kube-api-access-t422n\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4724g\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:17.587498 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.587459    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"]
Nov 03 01:28:17.698903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:17.698880    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:18.122172 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:18.122099    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"]
Nov 03 01:28:18.130007 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:28:18.129974    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f26262_d73f_4709_9aba_388786aec407.slice/crio-79e1ed145808c8ad7f149442cac67869afb66a93297ed652a3f928ec9609a318 WatchSource:0}: Error finding container 79e1ed145808c8ad7f149442cac67869afb66a93297ed652a3f928ec9609a318: Status 404 returned error can't find the container with id 79e1ed145808c8ad7f149442cac67869afb66a93297ed652a3f928ec9609a318
Nov 03 01:28:18.132061 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:18.132034    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:28:18.132135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:18.132097    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:28:18.132171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:18.132142    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:28:18.394323 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:18.394239    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" event={"ID":"15f26262-d73f-4709-9aba-388786aec407","Type":"ContainerStarted","Data":"9514be1043e5ee23d334095804c6dd7272ffe486fc5b1da902312354e8f92db3"}
Nov 03 01:28:18.394323 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:18.394288    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" event={"ID":"15f26262-d73f-4709-9aba-388786aec407","Type":"ContainerStarted","Data":"79e1ed145808c8ad7f149442cac67869afb66a93297ed652a3f928ec9609a318"}
Nov 03 01:28:18.508426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:18.508330    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" podStartSLOduration=1.5082956749999998 podStartE2EDuration="1.508295675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:18.506501595 +0000 UTC m=+577.932108076" watchObservedRunningTime="2026-11-03 01:28:18.508295675 +0000 UTC m=+577.933902140"
Nov 03 01:28:18.699008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:18.698978    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:19.076113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:19.076010    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:28:19.700901 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:19.700868    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 01:28:19.701304 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:19.700924    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" podUID="15f26262-d73f-4709-9aba-388786aec407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 01:28:20.702806 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:20.702612    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 01:28:20.703199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:20.702827    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" podUID="15f26262-d73f-4709-9aba-388786aec407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 01:28:21.699940 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:21.699899    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 01:28:21.700115 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:21.699969    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" podUID="15f26262-d73f-4709-9aba-388786aec407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 01:28:22.699783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:22.699744    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 01:28:22.700183 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:22.699797    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" podUID="15f26262-d73f-4709-9aba-388786aec407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 01:28:23.699699 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:23.699659    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 01:28:23.699894 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:23.699727    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" podUID="15f26262-d73f-4709-9aba-388786aec407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 01:28:24.700300 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:24.700264    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Nov 03 01:28:24.700741 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:24.700323    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" podUID="15f26262-d73f-4709-9aba-388786aec407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Nov 03 01:28:26.700284 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:26.700245    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:28:26.700783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:26.700311    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" podUID="15f26262-d73f-4709-9aba-388786aec407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:28:27.741511 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:27.741471    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4724g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:28:27.741935 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:27.741538    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" podUID="15f26262-d73f-4709-9aba-388786aec407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:28:28.190276 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:28.190240    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:28.190708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:28.190682    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:28.191299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:28.191284    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:28:32.500560 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.500524    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"]
Nov 03 01:28:32.504272 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.504249    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.568507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.568481    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.568653 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.568509    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.568653 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.568537    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.568730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.568643    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e843812-79f6-4c3c-b215-474e4c966d49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.568730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.568674    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jltwf\" (UniqueName: \"kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-kube-api-access-jltwf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.568730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.568716    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.568821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.568768    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.568821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.568793    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e843812-79f6-4c3c-b215-474e4c966d49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.568885 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.568851    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.670227 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.670192    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.670227 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.670226    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e843812-79f6-4c3c-b215-474e4c966d49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.670517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.670244    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.670517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.670271    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.670517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.670286    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.670517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.670414    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.670517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.670475    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e843812-79f6-4c3c-b215-474e4c966d49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.670778 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.670518    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jltwf\" (UniqueName: \"kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-kube-api-access-jltwf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.670778 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.670579    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.670778 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.670702    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.670938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.670910    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.671002 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.670937    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.671137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.671118    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.671276 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.671253    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e843812-79f6-4c3c-b215-474e4c966d49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.672540 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.672522    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.672670 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.672654    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e843812-79f6-4c3c-b215-474e4c966d49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.692100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.692072    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"]
Nov 03 01:28:32.727394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.727367    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.727530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.727483    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jltwf\" (UniqueName: \"kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-kube-api-access-jltwf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-6t5b5\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:32.816663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:32.816589    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:33.242447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:33.242416    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"]
Nov 03 01:28:33.249732 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:28:33.249697    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e843812_79f6_4c3c_b215_474e4c966d49.slice/crio-0d32258d8b34eb4f3f5e5a3dbec9d8344fb59d930c1cde1a7e5d3a97e5129c28 WatchSource:0}: Error finding container 0d32258d8b34eb4f3f5e5a3dbec9d8344fb59d930c1cde1a7e5d3a97e5129c28: Status 404 returned error can't find the container with id 0d32258d8b34eb4f3f5e5a3dbec9d8344fb59d930c1cde1a7e5d3a97e5129c28
Nov 03 01:28:33.251880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:33.251847    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:28:33.251962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:33.251907    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:28:33.251962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:33.251947    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:28:33.473152 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:33.473119    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5" event={"ID":"8e843812-79f6-4c3c-b215-474e4c966d49","Type":"ContainerStarted","Data":"d83545769c89a5d2b225cd5848cc5d2744e4b2f0687dd149822e11c75d41d937"}
Nov 03 01:28:33.473334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:33.473168    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5" event={"ID":"8e843812-79f6-4c3c-b215-474e4c966d49","Type":"ContainerStarted","Data":"0d32258d8b34eb4f3f5e5a3dbec9d8344fb59d930c1cde1a7e5d3a97e5129c28"}
Nov 03 01:28:33.586441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:33.586271    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5" podStartSLOduration=1.5862526890000002 podStartE2EDuration="1.586252689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:33.585370752 +0000 UTC m=+593.010977220" watchObservedRunningTime="2026-11-03 01:28:33.586252689 +0000 UTC m=+593.011859156"
Nov 03 01:28:33.817111 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:33.817076    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:34.819249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:34.819218    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6t5b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 01:28:34.819635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:34.819269    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5" podUID="8e843812-79f6-4c3c-b215-474e4c966d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 01:28:35.817826 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:35.817792    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6t5b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 01:28:35.817978 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:35.817841    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5" podUID="8e843812-79f6-4c3c-b215-474e4c966d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 01:28:36.817529 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:36.817494    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6t5b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 01:28:36.817951 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:36.817550    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5" podUID="8e843812-79f6-4c3c-b215-474e4c966d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 01:28:37.817982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:37.817947    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6t5b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 01:28:37.818407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:37.817998    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5" podUID="8e843812-79f6-4c3c-b215-474e4c966d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 01:28:38.817718 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:38.817683    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6t5b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 01:28:38.817889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:38.817738    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5" podUID="8e843812-79f6-4c3c-b215-474e4c966d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 01:28:39.817390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:39.817336    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-6t5b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Nov 03 01:28:39.817863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:39.817414    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5" podUID="8e843812-79f6-4c3c-b215-474e4c966d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Nov 03 01:28:40.634063 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.634030    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6b9b56d5-vpvgj"]
Nov 03 01:28:40.637721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.637704    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6b9b56d5-vpvgj"
Nov 03 01:28:40.700383 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.700339    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6b9b56d5-vpvgj"]
Nov 03 01:28:40.744818 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.744788    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 01:28:40.745455 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.745309    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cm5nl\""
Nov 03 01:28:40.753509 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.753485    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/43b5424f-45e9-49e4-975c-bc9c0dcda261-tls-cert\") pod \"authorino-b6b9b56d5-vpvgj\" (UID: \"43b5424f-45e9-49e4-975c-bc9c0dcda261\") " pod="kuadrant-system/authorino-b6b9b56d5-vpvgj"
Nov 03 01:28:40.753668 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.753641    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-844f7\" (UniqueName: \"kubernetes.io/projected/43b5424f-45e9-49e4-975c-bc9c0dcda261-kube-api-access-844f7\") pod \"authorino-b6b9b56d5-vpvgj\" (UID: \"43b5424f-45e9-49e4-975c-bc9c0dcda261\") " pod="kuadrant-system/authorino-b6b9b56d5-vpvgj"
Nov 03 01:28:40.763683 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.763661    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:28:40.763978 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.763959    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:28:40.854757 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.854703    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-844f7\" (UniqueName: \"kubernetes.io/projected/43b5424f-45e9-49e4-975c-bc9c0dcda261-kube-api-access-844f7\") pod \"authorino-b6b9b56d5-vpvgj\" (UID: \"43b5424f-45e9-49e4-975c-bc9c0dcda261\") " pod="kuadrant-system/authorino-b6b9b56d5-vpvgj"
Nov 03 01:28:40.855255 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.854876    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/43b5424f-45e9-49e4-975c-bc9c0dcda261-tls-cert\") pod \"authorino-b6b9b56d5-vpvgj\" (UID: \"43b5424f-45e9-49e4-975c-bc9c0dcda261\") " pod="kuadrant-system/authorino-b6b9b56d5-vpvgj"
Nov 03 01:28:40.858119 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.858084    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/43b5424f-45e9-49e4-975c-bc9c0dcda261-tls-cert\") pod \"authorino-b6b9b56d5-vpvgj\" (UID: \"43b5424f-45e9-49e4-975c-bc9c0dcda261\") " pod="kuadrant-system/authorino-b6b9b56d5-vpvgj"
Nov 03 01:28:40.912535 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.912501    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-844f7\" (UniqueName: \"kubernetes.io/projected/43b5424f-45e9-49e4-975c-bc9c0dcda261-kube-api-access-844f7\") pod \"authorino-b6b9b56d5-vpvgj\" (UID: \"43b5424f-45e9-49e4-975c-bc9c0dcda261\") " pod="kuadrant-system/authorino-b6b9b56d5-vpvgj"
Nov 03 01:28:40.951910 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:40.951879    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6b9b56d5-vpvgj"
Nov 03 01:28:41.332729 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:41.332646    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6b9b56d5-vpvgj"]
Nov 03 01:28:41.340271 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:28:41.340237    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43b5424f_45e9_49e4_975c_bc9c0dcda261.slice/crio-cb5ff130d51fffd6a4f78e9d6c7cf3c291cb782e05b2b8941a34dc7ea2729e7b WatchSource:0}: Error finding container cb5ff130d51fffd6a4f78e9d6c7cf3c291cb782e05b2b8941a34dc7ea2729e7b: Status 404 returned error can't find the container with id cb5ff130d51fffd6a4f78e9d6c7cf3c291cb782e05b2b8941a34dc7ea2729e7b
Nov 03 01:28:41.513724 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:41.513693    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6b9b56d5-vpvgj" event={"ID":"43b5424f-45e9-49e4-975c-bc9c0dcda261","Type":"ContainerStarted","Data":"cb5ff130d51fffd6a4f78e9d6c7cf3c291cb782e05b2b8941a34dc7ea2729e7b"}
Nov 03 01:28:41.527851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:41.527822    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:41.528250 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:41.528225    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:41.528945 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:41.528922    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:28:43.525233 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:43.525187    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6b9b56d5-vpvgj" event={"ID":"43b5424f-45e9-49e4-975c-bc9c0dcda261","Type":"ContainerStarted","Data":"2ce766a204df9ac0cc1d2faaf23c97fa27f74fe2f3bef207c3f20aaf0433309c"}
Nov 03 01:28:43.633649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:28:43.633589    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-b6b9b56d5-vpvgj" podStartSLOduration=1.851318385 podStartE2EDuration="3.63357172s" totalImagesPullingTime="1.782253335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:43.632784051 +0000 UTC m=+603.058390534" watchObservedRunningTime="2026-11-03 01:28:43.63357172 +0000 UTC m=+603.059178186"
Nov 03 01:28:43.811475 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:28:43.811388    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:29:13.810313 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:29:13.810275    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:29:18.937380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:18.937276    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"]
Nov 03 01:29:18.942186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:18.942169    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:18.990978 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:18.990956    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-21c25193-openshift-default-dockercfg-m9n2f\""
Nov 03 01:29:18.998443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:18.998409    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-envoy\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:18.998563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:18.998465    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-socket\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:18.998603 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:18.998553    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-certs\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:18.998603 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:18.998580    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a745a282-70f9-46f3-8a35-a64d6807040e-istio-podinfo\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:18.998677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:18.998666    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-data\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:18.998711 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:18.998689    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a745a282-70f9-46f3-8a35-a64d6807040e-istiod-ca-cert\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:18.998711 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:18.998707    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-credential-socket\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:18.998782 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:18.998735    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-istio-token\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:18.998928 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:18.998908    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtvzt\" (UniqueName: \"kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-kube-api-access-qtvzt\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.041323 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.041276    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"]
Nov 03 01:29:19.099898 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.099863    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-istio-token\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.100236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.100193    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtvzt\" (UniqueName: \"kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-kube-api-access-qtvzt\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.100447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.100428    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-envoy\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.100581 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.100566    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-socket\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.100705 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.100690    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-certs\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.100813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.100798    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a745a282-70f9-46f3-8a35-a64d6807040e-istio-podinfo\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.101451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.101425    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-data\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.102570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.102550    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a745a282-70f9-46f3-8a35-a64d6807040e-istiod-ca-cert\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.102692 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.102677    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-credential-socket\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.102838 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.102051    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-data\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.102922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.101536    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-socket\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.103410 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.103391    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a745a282-70f9-46f3-8a35-a64d6807040e-istiod-ca-cert\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.103501 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.101837    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-certs\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.103677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.103644    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-credential-socket\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.106859 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.106840    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-envoy\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.108267 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.108247    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a745a282-70f9-46f3-8a35-a64d6807040e-istio-podinfo\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.162133 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.162103    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-istio-token\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.163098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.163076    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtvzt\" (UniqueName: \"kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-kube-api-access-qtvzt\") pod \"e2e-ait-21c25193-openshift-default-967b44fd9-smz46\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") " pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.220380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.220287    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"]
Nov 03 01:29:19.223724 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.223702    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.254792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.254770    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:19.283066 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.282992    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"]
Nov 03 01:29:19.304437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.304405    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-socket\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.304606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.304473    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsjmc\" (UniqueName: \"kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-kube-api-access-hsjmc\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.304606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.304512    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e3f853-b154-4314-ba90-01a1caaee85a-istio-podinfo\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.304606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.304547    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-envoy\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.304769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.304610    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-credential-socket\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.304769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.304701    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-istio-token\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.304769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.304732    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e3f853-b154-4314-ba90-01a1caaee85a-istiod-ca-cert\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.304769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.304758    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-data\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.304957 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.304794    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-certs\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.343792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.343762    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-98c7cd16-openshift-default-dockercfg-6whl2\""
Nov 03 01:29:19.405939 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.405911    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-socket\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.406104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.405957    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsjmc\" (UniqueName: \"kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-kube-api-access-hsjmc\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.406174 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.406146    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e3f853-b154-4314-ba90-01a1caaee85a-istio-podinfo\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.406231 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.406188    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-envoy\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.406283 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.406262    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-credential-socket\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.406335 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.406311    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-socket\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.406420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.406405    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-istio-token\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.406478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.406439    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e3f853-b154-4314-ba90-01a1caaee85a-istiod-ca-cert\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.406478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.406458    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-data\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.406597 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.406582    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-certs\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.406733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.406683    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-credential-socket\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.406887 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.406865    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-certs\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.406954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.406902    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-data\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.407131 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.407112    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e3f853-b154-4314-ba90-01a1caaee85a-istiod-ca-cert\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.408648 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.408631    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e3f853-b154-4314-ba90-01a1caaee85a-istio-podinfo\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.408771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.408753    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-envoy\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.463270 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.463246    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-istio-token\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.463666 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.463647    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsjmc\" (UniqueName: \"kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-kube-api-access-hsjmc\") pod \"e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") " pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.536233 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.536184    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:19.688756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.688598    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"]
Nov 03 01:29:19.694627 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:29:19.694595    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda745a282_70f9_46f3_8a35_a64d6807040e.slice/crio-2f2620f3dc1635c6ba4751fadab17403083a4122ca00991cd69f42bce70d6256 WatchSource:0}: Error finding container 2f2620f3dc1635c6ba4751fadab17403083a4122ca00991cd69f42bce70d6256: Status 404 returned error can't find the container with id 2f2620f3dc1635c6ba4751fadab17403083a4122ca00991cd69f42bce70d6256
Nov 03 01:29:19.696887 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.696854    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:19.696983 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.696916    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:19.696983 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.696951    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:19.736320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.736296    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" event={"ID":"a745a282-70f9-46f3-8a35-a64d6807040e","Type":"ContainerStarted","Data":"2f2620f3dc1635c6ba4751fadab17403083a4122ca00991cd69f42bce70d6256"}
Nov 03 01:29:19.938806 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.938770    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"]
Nov 03 01:29:19.945679 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:29:19.945650    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e3f853_b154_4314_ba90_01a1caaee85a.slice/crio-07e080ef4c49a1b5fead94ab2ccf0d9a380e3160149d6cee71163c22c7171a31 WatchSource:0}: Error finding container 07e080ef4c49a1b5fead94ab2ccf0d9a380e3160149d6cee71163c22c7171a31: Status 404 returned error can't find the container with id 07e080ef4c49a1b5fead94ab2ccf0d9a380e3160149d6cee71163c22c7171a31
Nov 03 01:29:19.948003 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.947968    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:19.948101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.948055    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:19.948139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:19.948100    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:20.742767 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:20.742725    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" event={"ID":"a745a282-70f9-46f3-8a35-a64d6807040e","Type":"ContainerStarted","Data":"a05644fb4eaaf625384e83fd9bbec5dc50e84a2c986da995411909922440d54e"}
Nov 03 01:29:20.744216 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:20.744193    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62" event={"ID":"73e3f853-b154-4314-ba90-01a1caaee85a","Type":"ContainerStarted","Data":"d3188b5b4e7f3fa553c9da59c087cfa94710bfae4bf96cf710d366b5ad42977e"}
Nov 03 01:29:20.744332 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:20.744230    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62" event={"ID":"73e3f853-b154-4314-ba90-01a1caaee85a","Type":"ContainerStarted","Data":"07e080ef4c49a1b5fead94ab2ccf0d9a380e3160149d6cee71163c22c7171a31"}
Nov 03 01:29:20.857233 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:20.857180    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" podStartSLOduration=2.857166109 podStartE2EDuration="2.857166109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:20.854593366 +0000 UTC m=+640.280199828" watchObservedRunningTime="2026-11-03 01:29:20.857166109 +0000 UTC m=+640.282772573"
Nov 03 01:29:20.995421 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:20.983225    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62" podStartSLOduration=1.9832051339999999 podStartE2EDuration="1.983205134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:20.981939538 +0000 UTC m=+640.407546004" watchObservedRunningTime="2026-11-03 01:29:20.983205134 +0000 UTC m=+640.408811599"
Nov 03 01:29:21.256014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.255924    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:21.260789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.260764    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:21.266683 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.266660    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"]
Nov 03 01:29:21.270201 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.270184    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.319571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.319547    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fc87c747-openshift-default-dockercfg-fggqw\""
Nov 03 01:29:21.327886 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.327856    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"]
Nov 03 01:29:21.335570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.335548    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf782\" (UniqueName: \"kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-kube-api-access-kf782\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.335698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.335588    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-podinfo\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.335698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.335606    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istiod-ca-cert\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.335824 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.335726    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-socket\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.335824 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.335765    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-credential-socket\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.335824 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.335779    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-certs\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.335977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.335875    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-data\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.335977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.335926    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-envoy\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.335977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.335947    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-token\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.436626 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.436593    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-socket\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.436823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.436636    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-credential-socket\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.436823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.436651    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-certs\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.436823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.436702    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-data\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.436823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.436777    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-envoy\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.436823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.436809    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-token\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.437087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.436845    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf782\" (UniqueName: \"kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-kube-api-access-kf782\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.437087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.436889    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-podinfo\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.437087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.436909    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istiod-ca-cert\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.437236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.437102    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-socket\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.437236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.437140    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-certs\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.437236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.437183    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-credential-socket\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.437236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.437226    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-data\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.437650 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.437632    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istiod-ca-cert\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.439142 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.439123    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-envoy\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.439487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.439468    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-podinfo\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.494050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.494012    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-token\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.494218 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.494138    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf782\" (UniqueName: \"kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-kube-api-access-kf782\") pod \"e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") " pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.536976 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.536898    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:21.541852 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.541829    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:21.583487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.583456    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:21.749418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.749386    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:21.749793 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.749448    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:21.750240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.750220    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:21.750376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.750281    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:29:21.985731 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.985690    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"]
Nov 03 01:29:21.991483 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:29:21.991454    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3133b2_0ba6_4231_bd4a_fce0bdd960cd.slice/crio-d61e9af9eb533b4b50b6ea77a554e44e42fbbe64c48d9f8a058c76dd8fb8adf0 WatchSource:0}: Error finding container d61e9af9eb533b4b50b6ea77a554e44e42fbbe64c48d9f8a058c76dd8fb8adf0: Status 404 returned error can't find the container with id d61e9af9eb533b4b50b6ea77a554e44e42fbbe64c48d9f8a058c76dd8fb8adf0
Nov 03 01:29:21.994115 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:21.994076    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:22.011760 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:22.011693    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:22.012274 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:22.011841    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:22.756584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:22.756545    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" event={"ID":"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd","Type":"ContainerStarted","Data":"c47253ff836e6403d40f90c34c07c67bc37ef3dbca02b8f72081a3e130913516"}
Nov 03 01:29:22.756781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:22.756611    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" event={"ID":"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd","Type":"ContainerStarted","Data":"d61e9af9eb533b4b50b6ea77a554e44e42fbbe64c48d9f8a058c76dd8fb8adf0"}
Nov 03 01:29:22.871971 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:22.871924    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" podStartSLOduration=1.87191143 podStartE2EDuration="1.87191143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:22.870490862 +0000 UTC m=+642.296097327" watchObservedRunningTime="2026-11-03 01:29:22.87191143 +0000 UTC m=+642.297517893"
Nov 03 01:29:23.583803 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:23.583770    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:23.588490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:23.588466    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:23.762056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:23.762025    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:23.762894 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:23.762873    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:27.330957 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:27.330926    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"]
Nov 03 01:29:27.334654 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:27.334637    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"
Nov 03 01:29:27.398715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:27.398411    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86nm\" (UniqueName: \"kubernetes.io/projected/a1265bf3-e6a8-4550-b683-becfa74c40cb-kube-api-access-p86nm\") pod \"payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7\" (UID: \"a1265bf3-e6a8-4550-b683-becfa74c40cb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"
Nov 03 01:29:27.398715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:27.398460    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"]
Nov 03 01:29:27.398715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:27.398527    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1265bf3-e6a8-4550-b683-becfa74c40cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7\" (UID: \"a1265bf3-e6a8-4550-b683-becfa74c40cb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"
Nov 03 01:29:27.437881 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:27.437789    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e3f0a557\""
Nov 03 01:29:27.441927 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:27.441877    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e3f0a557-dockercfg-8hc4x\""
Nov 03 01:29:27.499499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:27.499469    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86nm\" (UniqueName: \"kubernetes.io/projected/a1265bf3-e6a8-4550-b683-becfa74c40cb-kube-api-access-p86nm\") pod \"payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7\" (UID: \"a1265bf3-e6a8-4550-b683-becfa74c40cb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"
Nov 03 01:29:27.499654 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:27.499535    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1265bf3-e6a8-4550-b683-becfa74c40cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7\" (UID: \"a1265bf3-e6a8-4550-b683-becfa74c40cb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"
Nov 03 01:29:27.500467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:27.500444    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1265bf3-e6a8-4550-b683-becfa74c40cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7\" (UID: \"a1265bf3-e6a8-4550-b683-becfa74c40cb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"
Nov 03 01:29:27.556873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:27.556846    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86nm\" (UniqueName: \"kubernetes.io/projected/a1265bf3-e6a8-4550-b683-becfa74c40cb-kube-api-access-p86nm\") pod \"payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7\" (UID: \"a1265bf3-e6a8-4550-b683-becfa74c40cb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"
Nov 03 01:29:27.647221 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:27.647186    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"
Nov 03 01:29:28.016273 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:28.016204    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"]
Nov 03 01:29:28.024025 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:29:28.023997    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1265bf3_e6a8_4550_b683_becfa74c40cb.slice/crio-c88709296a48dd0c7c3746906e52b5a3947e21ce26a91d1de7ab18edbadb7baf WatchSource:0}: Error finding container c88709296a48dd0c7c3746906e52b5a3947e21ce26a91d1de7ab18edbadb7baf: Status 404 returned error can't find the container with id c88709296a48dd0c7c3746906e52b5a3947e21ce26a91d1de7ab18edbadb7baf
Nov 03 01:29:28.787771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:28.787734    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7" event={"ID":"a1265bf3-e6a8-4550-b683-becfa74c40cb","Type":"ContainerStarted","Data":"c88709296a48dd0c7c3746906e52b5a3947e21ce26a91d1de7ab18edbadb7baf"}
Nov 03 01:29:31.806768 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:31.806732    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7" event={"ID":"a1265bf3-e6a8-4550-b683-becfa74c40cb","Type":"ContainerStarted","Data":"ccecb3823b411ad2405677da38c36d823d81f71f8da20f2398d4021ce0ce5048"}
Nov 03 01:29:31.807171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:31.806842    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"
Nov 03 01:29:31.915855 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:31.915738    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7" podStartSLOduration=1.934444917 podStartE2EDuration="4.915718954s" totalImagesPullingTime="2.981274037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:31.915111007 +0000 UTC m=+651.340717480" watchObservedRunningTime="2026-11-03 01:29:31.915718954 +0000 UTC m=+651.341325431"
Nov 03 01:29:31.990259 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:31.990220    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"]
Nov 03 01:29:31.994649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:31.994626    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"
Nov 03 01:29:32.049386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.045742    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-21c25193-dockercfg-hmc5h\""
Nov 03 01:29:32.049386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.048386    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"]
Nov 03 01:29:32.049386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.048941    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-21c25193\""
Nov 03 01:29:32.091298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.091015    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t6g7\" (UniqueName: \"kubernetes.io/projected/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-kube-api-access-6t6g7\") pod \"payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv\" (UID: \"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"
Nov 03 01:29:32.091298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.091075    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv\" (UID: \"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"
Nov 03 01:29:32.191854 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.191824    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t6g7\" (UniqueName: \"kubernetes.io/projected/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-kube-api-access-6t6g7\") pod \"payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv\" (UID: \"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"
Nov 03 01:29:32.192036 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.191869    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv\" (UID: \"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"
Nov 03 01:29:32.192631 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.192611    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv\" (UID: \"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"
Nov 03 01:29:32.249865 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.249840    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t6g7\" (UniqueName: \"kubernetes.io/projected/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-kube-api-access-6t6g7\") pod \"payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv\" (UID: \"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"
Nov 03 01:29:32.322317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.322259    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"
Nov 03 01:29:32.717367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.717328    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"]
Nov 03 01:29:32.724528 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:29:32.724502    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a9ccb3_2e3d_4599_a491_8167a2bd1e40.slice/crio-a8d0453e80219171c2dc12e3d176cf5f53aa2b49f76a0f2d665adea4db48ad3d WatchSource:0}: Error finding container a8d0453e80219171c2dc12e3d176cf5f53aa2b49f76a0f2d665adea4db48ad3d: Status 404 returned error can't find the container with id a8d0453e80219171c2dc12e3d176cf5f53aa2b49f76a0f2d665adea4db48ad3d
Nov 03 01:29:32.813099 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.813068    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv" event={"ID":"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40","Type":"ContainerStarted","Data":"feff271e020e1e85dbc8115de4cd40e294b45a9cd03f729665fa36a1f91c7402"}
Nov 03 01:29:32.813464 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.813127    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv" event={"ID":"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40","Type":"ContainerStarted","Data":"a8d0453e80219171c2dc12e3d176cf5f53aa2b49f76a0f2d665adea4db48ad3d"}
Nov 03 01:29:32.813464 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.813356    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"
Nov 03 01:29:32.922394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:32.922331    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv" podStartSLOduration=1.9223175700000001 podStartE2EDuration="1.92231757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:32.92045908 +0000 UTC m=+652.346065545" watchObservedRunningTime="2026-11-03 01:29:32.92231757 +0000 UTC m=+652.347924035"
Nov 03 01:29:36.863411 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:36.863375    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"]
Nov 03 01:29:36.867200 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:36.867179    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"
Nov 03 01:29:36.920700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:36.920666    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"]
Nov 03 01:29:36.924541 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:36.924519    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"
Nov 03 01:29:36.953219 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:36.953179    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704ddecf-1477-4861-adaf-261c02fac634-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8\" (UID: \"704ddecf-1477-4861-adaf-261c02fac634\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"
Nov 03 01:29:36.953477 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:36.953238    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n4lr\" (UniqueName: \"kubernetes.io/projected/704ddecf-1477-4861-adaf-261c02fac634-kube-api-access-8n4lr\") pod \"payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8\" (UID: \"704ddecf-1477-4861-adaf-261c02fac634\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"
Nov 03 01:29:36.964626 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:36.964568    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fc87c747-dockercfg-5ck6l\""
Nov 03 01:29:36.964785 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:36.964595    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fc87c747\""
Nov 03 01:29:36.967237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:36.967215    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"]
Nov 03 01:29:37.013442 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.013407    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"]
Nov 03 01:29:37.053934 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.053906    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n4lr\" (UniqueName: \"kubernetes.io/projected/704ddecf-1477-4861-adaf-261c02fac634-kube-api-access-8n4lr\") pod \"payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8\" (UID: \"704ddecf-1477-4861-adaf-261c02fac634\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"
Nov 03 01:29:37.054096 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.053947    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95532f38-0c60-4cd3-be34-27f692a72b50-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq\" (UID: \"95532f38-0c60-4cd3-be34-27f692a72b50\") " pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"
Nov 03 01:29:37.054096 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.053984    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq226\" (UniqueName: \"kubernetes.io/projected/95532f38-0c60-4cd3-be34-27f692a72b50-kube-api-access-tq226\") pod \"payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq\" (UID: \"95532f38-0c60-4cd3-be34-27f692a72b50\") " pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"
Nov 03 01:29:37.054202 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.054179    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704ddecf-1477-4861-adaf-261c02fac634-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8\" (UID: \"704ddecf-1477-4861-adaf-261c02fac634\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"
Nov 03 01:29:37.054894 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.054870    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704ddecf-1477-4861-adaf-261c02fac634-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8\" (UID: \"704ddecf-1477-4861-adaf-261c02fac634\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"
Nov 03 01:29:37.111323 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.111297    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n4lr\" (UniqueName: \"kubernetes.io/projected/704ddecf-1477-4861-adaf-261c02fac634-kube-api-access-8n4lr\") pod \"payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8\" (UID: \"704ddecf-1477-4861-adaf-261c02fac634\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"
Nov 03 01:29:37.155658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.155633    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95532f38-0c60-4cd3-be34-27f692a72b50-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq\" (UID: \"95532f38-0c60-4cd3-be34-27f692a72b50\") " pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"
Nov 03 01:29:37.155792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.155675    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq226\" (UniqueName: \"kubernetes.io/projected/95532f38-0c60-4cd3-be34-27f692a72b50-kube-api-access-tq226\") pod \"payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq\" (UID: \"95532f38-0c60-4cd3-be34-27f692a72b50\") " pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"
Nov 03 01:29:37.156293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.156271    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95532f38-0c60-4cd3-be34-27f692a72b50-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq\" (UID: \"95532f38-0c60-4cd3-be34-27f692a72b50\") " pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"
Nov 03 01:29:37.179884 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.179854    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"
Nov 03 01:29:37.213750 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.213723    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq226\" (UniqueName: \"kubernetes.io/projected/95532f38-0c60-4cd3-be34-27f692a72b50-kube-api-access-tq226\") pod \"payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq\" (UID: \"95532f38-0c60-4cd3-be34-27f692a72b50\") " pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"
Nov 03 01:29:37.240262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.240235    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"
Nov 03 01:29:37.553545 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.553473    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"]
Nov 03 01:29:37.560488 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:29:37.560462    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704ddecf_1477_4861_adaf_261c02fac634.slice/crio-e01a2c4aeb527dd946140e7bbb13ffcd0fbbec2e1ff6f463d70712c44ba7b806 WatchSource:0}: Error finding container e01a2c4aeb527dd946140e7bbb13ffcd0fbbec2e1ff6f463d70712c44ba7b806: Status 404 returned error can't find the container with id e01a2c4aeb527dd946140e7bbb13ffcd0fbbec2e1ff6f463d70712c44ba7b806
Nov 03 01:29:37.657620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.657592    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"]
Nov 03 01:29:37.664524 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:29:37.664498    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95532f38_0c60_4cd3_be34_27f692a72b50.slice/crio-2d7da5c539733ab111e87dce7833cc9ab07070aaae66651b38663e24dd3d9057 WatchSource:0}: Error finding container 2d7da5c539733ab111e87dce7833cc9ab07070aaae66651b38663e24dd3d9057: Status 404 returned error can't find the container with id 2d7da5c539733ab111e87dce7833cc9ab07070aaae66651b38663e24dd3d9057
Nov 03 01:29:37.854441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.854338    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8" event={"ID":"704ddecf-1477-4861-adaf-261c02fac634","Type":"ContainerStarted","Data":"11216f9c72a04e2444430cb80a86190e328dc2c00a4cc37cdde24efa23aec4df"}
Nov 03 01:29:37.854441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.854419    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"
Nov 03 01:29:37.854650 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.854446    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8" event={"ID":"704ddecf-1477-4861-adaf-261c02fac634","Type":"ContainerStarted","Data":"e01a2c4aeb527dd946140e7bbb13ffcd0fbbec2e1ff6f463d70712c44ba7b806"}
Nov 03 01:29:37.855919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.855892    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq" event={"ID":"95532f38-0c60-4cd3-be34-27f692a72b50","Type":"ContainerStarted","Data":"88c01349ebb9f24509f05c507168658823a041d19bfd054b90df26b4278d289f"}
Nov 03 01:29:37.856030 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.855940    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq" event={"ID":"95532f38-0c60-4cd3-be34-27f692a72b50","Type":"ContainerStarted","Data":"2d7da5c539733ab111e87dce7833cc9ab07070aaae66651b38663e24dd3d9057"}
Nov 03 01:29:37.856030 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.855968    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"
Nov 03 01:29:37.965094 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:37.965041    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8" podStartSLOduration=1.9650264480000001 podStartE2EDuration="1.965026448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:37.962791442 +0000 UTC m=+657.388397931" watchObservedRunningTime="2026-11-03 01:29:37.965026448 +0000 UTC m=+657.390633120"
Nov 03 01:29:38.092815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:38.089977    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq" podStartSLOduration=2.089954539 podStartE2EDuration="2.089954539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:38.071612061 +0000 UTC m=+657.497218546" watchObservedRunningTime="2026-11-03 01:29:38.089954539 +0000 UTC m=+657.515561006"
Nov 03 01:29:40.387185 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:40.387148    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:29:42.816258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:42.816226    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"
Nov 03 01:29:43.811817 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:29:43.811784    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:29:43.822509 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:43.822486    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"
Nov 03 01:29:48.883382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:48.883333    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"
Nov 03 01:29:48.884863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:48.884826    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"
Nov 03 01:29:49.970251 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:49.968310    5508 generic.go:356] "Generic (PLEG): container finished" podID="73e3f853-b154-4314-ba90-01a1caaee85a" containerID="d3188b5b4e7f3fa553c9da59c087cfa94710bfae4bf96cf710d366b5ad42977e" exitCode=0
Nov 03 01:29:49.970251 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:49.968528    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62" event={"ID":"73e3f853-b154-4314-ba90-01a1caaee85a","Type":"ContainerDied","Data":"d3188b5b4e7f3fa553c9da59c087cfa94710bfae4bf96cf710d366b5ad42977e"}
Nov 03 01:29:49.970251 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:49.968968    5508 scope.go:122] "RemoveContainer" containerID="d3188b5b4e7f3fa553c9da59c087cfa94710bfae4bf96cf710d366b5ad42977e"
Nov 03 01:29:49.972329 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:49.971929    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:49.972329 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:49.972020    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:49.972329 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:49.972101    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:51.002086 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:51.001295    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62" event={"ID":"73e3f853-b154-4314-ba90-01a1caaee85a","Type":"ContainerStarted","Data":"22207b46516a2de38037aca1d3c492378839e56be5a95d4f43fce3ce07f446fe"}
Nov 03 01:29:51.536527 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:51.536495    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:51.537871 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:51.537844    5508 patch_prober.go:29] interesting pod/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 01:29:51.537964 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:51.537911    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62" podUID="73e3f853-b154-4314-ba90-01a1caaee85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 01:29:51.756372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:51.752430    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:52.537007 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:52.536974    5508 patch_prober.go:29] interesting pod/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 01:29:52.537516 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:52.537031    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62" podUID="73e3f853-b154-4314-ba90-01a1caaee85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 01:29:53.537253 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:53.537217    5508 patch_prober.go:29] interesting pod/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 01:29:53.537671 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:53.537265    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62" podUID="73e3f853-b154-4314-ba90-01a1caaee85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 01:29:54.536843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:54.536808    5508 patch_prober.go:29] interesting pod/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 01:29:54.537043 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:54.536876    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62" podUID="73e3f853-b154-4314-ba90-01a1caaee85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 01:29:55.537041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:55.537002    5508 patch_prober.go:29] interesting pod/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Nov 03 01:29:55.537570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:55.537081    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62" podUID="73e3f853-b154-4314-ba90-01a1caaee85a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Nov 03 01:29:56.468120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:56.468076    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:29:56.541529 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:56.541498    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:56.542534 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:56.542500    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:29:58.081580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:58.081492    5508 generic.go:356] "Generic (PLEG): container finished" podID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerID="c47253ff836e6403d40f90c34c07c67bc37ef3dbca02b8f72081a3e130913516" exitCode=0
Nov 03 01:29:58.081580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:58.081561    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" event={"ID":"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd","Type":"ContainerDied","Data":"c47253ff836e6403d40f90c34c07c67bc37ef3dbca02b8f72081a3e130913516"}
Nov 03 01:29:58.082166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:58.082022    5508 scope.go:122] "RemoveContainer" containerID="c47253ff836e6403d40f90c34c07c67bc37ef3dbca02b8f72081a3e130913516"
Nov 03 01:29:58.083486 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:58.083452    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:58.083598 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:58.083537    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:58.083598 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:58.083587    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:29:59.104125 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:59.104087    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" event={"ID":"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd","Type":"ContainerStarted","Data":"025df2bf378fcd03832463fc9e23e7f08e21410cecb47e3528ffd22ed6592138"}
Nov 03 01:29:59.589368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:59.588733    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:29:59.589368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:59.588857    5508 patch_prober.go:29] interesting pod/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 01:29:59.589368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:29:59.588897    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 01:30:00.186757 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.184579    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"]
Nov 03 01:30:00.192614 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.191583    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"
Nov 03 01:30:00.193460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.193438    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"]
Nov 03 01:30:00.201564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.201542    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"]
Nov 03 01:30:00.201754 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.201730    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"
Nov 03 01:30:00.205655 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.205640    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"
Nov 03 01:30:00.213780 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.213761    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"]
Nov 03 01:30:00.217257 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.217242    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"
Nov 03 01:30:00.232689 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.232670    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b610877-52f4-4d89-b7ba-b65d4b7e4312-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx\" (UID: \"1b610877-52f4-4d89-b7ba-b65d4b7e4312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"
Nov 03 01:30:00.232756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.232714    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5g9c\" (UniqueName: \"kubernetes.io/projected/4f03fce0-0280-4d45-82ae-59c40bbad92f-kube-api-access-k5g9c\") pod \"maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675\" (UID: \"4f03fce0-0280-4d45-82ae-59c40bbad92f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"
Nov 03 01:30:00.232817 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.232803    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f03fce0-0280-4d45-82ae-59c40bbad92f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675\" (UID: \"4f03fce0-0280-4d45-82ae-59c40bbad92f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"
Nov 03 01:30:00.232853 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.232837    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgvhv\" (UniqueName: \"kubernetes.io/projected/1b610877-52f4-4d89-b7ba-b65d4b7e4312-kube-api-access-sgvhv\") pod \"maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx\" (UID: \"1b610877-52f4-4d89-b7ba-b65d4b7e4312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"
Nov 03 01:30:00.232889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.232854    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79520c6d-ffee-41d0-a8f6-0801d98196ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894490-qgwrm\" (UID: \"79520c6d-ffee-41d0-a8f6-0801d98196ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"
Nov 03 01:30:00.232936 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.232919    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxb9\" (UniqueName: \"kubernetes.io/projected/79520c6d-ffee-41d0-a8f6-0801d98196ba-kube-api-access-9xxb9\") pod \"maas-api-key-cleanup-29894490-qgwrm\" (UID: \"79520c6d-ffee-41d0-a8f6-0801d98196ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"
Nov 03 01:30:00.240599 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.240583    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:30:00.240677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.240612    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:30:00.240677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.240645    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:30:00.287498 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.287469    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"]
Nov 03 01:30:00.308499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.303014    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"]
Nov 03 01:30:00.308847 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.308762    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"]
Nov 03 01:30:00.334266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.334233    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm29k\" (UniqueName: \"kubernetes.io/projected/969c5a71-7c35-466c-8b1d-2a864343c062-kube-api-access-gm29k\") pod \"maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp\" (UID: \"969c5a71-7c35-466c-8b1d-2a864343c062\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"
Nov 03 01:30:00.334405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.334284    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c5a71-7c35-466c-8b1d-2a864343c062-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp\" (UID: \"969c5a71-7c35-466c-8b1d-2a864343c062\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"
Nov 03 01:30:00.334405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.334383    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f03fce0-0280-4d45-82ae-59c40bbad92f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675\" (UID: \"4f03fce0-0280-4d45-82ae-59c40bbad92f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"
Nov 03 01:30:00.334488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.334430    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvhv\" (UniqueName: \"kubernetes.io/projected/1b610877-52f4-4d89-b7ba-b65d4b7e4312-kube-api-access-sgvhv\") pod \"maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx\" (UID: \"1b610877-52f4-4d89-b7ba-b65d4b7e4312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"
Nov 03 01:30:00.334488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.334451    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79520c6d-ffee-41d0-a8f6-0801d98196ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894490-qgwrm\" (UID: \"79520c6d-ffee-41d0-a8f6-0801d98196ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"
Nov 03 01:30:00.334563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.334498    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxb9\" (UniqueName: \"kubernetes.io/projected/79520c6d-ffee-41d0-a8f6-0801d98196ba-kube-api-access-9xxb9\") pod \"maas-api-key-cleanup-29894490-qgwrm\" (UID: \"79520c6d-ffee-41d0-a8f6-0801d98196ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"
Nov 03 01:30:00.334563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.334541    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b610877-52f4-4d89-b7ba-b65d4b7e4312-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx\" (UID: \"1b610877-52f4-4d89-b7ba-b65d4b7e4312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"
Nov 03 01:30:00.334622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.334579    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5g9c\" (UniqueName: \"kubernetes.io/projected/4f03fce0-0280-4d45-82ae-59c40bbad92f-kube-api-access-k5g9c\") pod \"maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675\" (UID: \"4f03fce0-0280-4d45-82ae-59c40bbad92f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"
Nov 03 01:30:00.336364 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.335329    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f03fce0-0280-4d45-82ae-59c40bbad92f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675\" (UID: \"4f03fce0-0280-4d45-82ae-59c40bbad92f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"
Nov 03 01:30:00.336364 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.335932    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79520c6d-ffee-41d0-a8f6-0801d98196ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894490-qgwrm\" (UID: \"79520c6d-ffee-41d0-a8f6-0801d98196ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"
Nov 03 01:30:00.336364 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.336322    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b610877-52f4-4d89-b7ba-b65d4b7e4312-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx\" (UID: \"1b610877-52f4-4d89-b7ba-b65d4b7e4312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"
Nov 03 01:30:00.347922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.347899    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"]
Nov 03 01:30:00.411770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.411077    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvhv\" (UniqueName: \"kubernetes.io/projected/1b610877-52f4-4d89-b7ba-b65d4b7e4312-kube-api-access-sgvhv\") pod \"maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx\" (UID: \"1b610877-52f4-4d89-b7ba-b65d4b7e4312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"
Nov 03 01:30:00.414862 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.414834    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5g9c\" (UniqueName: \"kubernetes.io/projected/4f03fce0-0280-4d45-82ae-59c40bbad92f-kube-api-access-k5g9c\") pod \"maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675\" (UID: \"4f03fce0-0280-4d45-82ae-59c40bbad92f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"
Nov 03 01:30:00.415105 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.415090    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxb9\" (UniqueName: \"kubernetes.io/projected/79520c6d-ffee-41d0-a8f6-0801d98196ba-kube-api-access-9xxb9\") pod \"maas-api-key-cleanup-29894490-qgwrm\" (UID: \"79520c6d-ffee-41d0-a8f6-0801d98196ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"
Nov 03 01:30:00.435905 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.435884    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm29k\" (UniqueName: \"kubernetes.io/projected/969c5a71-7c35-466c-8b1d-2a864343c062-kube-api-access-gm29k\") pod \"maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp\" (UID: \"969c5a71-7c35-466c-8b1d-2a864343c062\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"
Nov 03 01:30:00.436043 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.436030    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c5a71-7c35-466c-8b1d-2a864343c062-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp\" (UID: \"969c5a71-7c35-466c-8b1d-2a864343c062\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"
Nov 03 01:30:00.436756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.436739    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c5a71-7c35-466c-8b1d-2a864343c062-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp\" (UID: \"969c5a71-7c35-466c-8b1d-2a864343c062\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"
Nov 03 01:30:00.510158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.510070    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm29k\" (UniqueName: \"kubernetes.io/projected/969c5a71-7c35-466c-8b1d-2a864343c062-kube-api-access-gm29k\") pod \"maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp\" (UID: \"969c5a71-7c35-466c-8b1d-2a864343c062\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"
Nov 03 01:30:00.516365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.516326    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"
Nov 03 01:30:00.523706 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.523688    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"
Nov 03 01:30:00.530946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.530924    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"
Nov 03 01:30:00.537025 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.537004    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"
Nov 03 01:30:00.610395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.589473    5508 patch_prober.go:29] interesting pod/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 01:30:00.610395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.589532    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 01:30:00.965585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:00.965506    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"]
Nov 03 01:30:01.011244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:01.011218    5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:30:01.115592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:01.115520    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm" event={"ID":"79520c6d-ffee-41d0-a8f6-0801d98196ba","Type":"ContainerStarted","Data":"738020c2958231c8fbd7647a3204febf02ac667c4cb9afef4c80dd663f1a5aba"}
Nov 03 01:30:01.203606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:01.203572    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"]
Nov 03 01:30:01.206899 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:30:01.206867    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b610877_52f4_4d89_b7ba_b65d4b7e4312.slice/crio-36f063023a820333a51e4e65f496c13e7c51c432e1ebb591cbfc2fb97e9b515f WatchSource:0}: Error finding container 36f063023a820333a51e4e65f496c13e7c51c432e1ebb591cbfc2fb97e9b515f: Status 404 returned error can't find the container with id 36f063023a820333a51e4e65f496c13e7c51c432e1ebb591cbfc2fb97e9b515f
Nov 03 01:30:01.259381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:01.251141    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"]
Nov 03 01:30:01.267908 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:01.267881    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"]
Nov 03 01:30:01.585371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:01.585117    5508 patch_prober.go:29] interesting pod/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 01:30:01.585371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:01.585197    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 01:30:02.122004 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:02.121965    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675" event={"ID":"4f03fce0-0280-4d45-82ae-59c40bbad92f","Type":"ContainerStarted","Data":"f2c4cae09b13c1c7f28c82895d2d388a5f0fadf7beacd4a88132555a52a36d32"}
Nov 03 01:30:02.123593 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:02.123559    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp" event={"ID":"969c5a71-7c35-466c-8b1d-2a864343c062","Type":"ContainerStarted","Data":"8a8da252876b90e7a72ff8788034e6d9b1c4edeaa3e9fe5c7b02345846fe1f0b"}
Nov 03 01:30:02.125077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:02.125047    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx" event={"ID":"1b610877-52f4-4d89-b7ba-b65d4b7e4312","Type":"ContainerStarted","Data":"36f063023a820333a51e4e65f496c13e7c51c432e1ebb591cbfc2fb97e9b515f"}
Nov 03 01:30:02.584253 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:02.584214    5508 patch_prober.go:29] interesting pod/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 01:30:02.584657 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:02.584280    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 01:30:03.583929 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:03.583892    5508 patch_prober.go:29] interesting pod/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 01:30:03.584108 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:03.583950    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 01:30:04.589141 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:04.589109    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:30:04.590284 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:04.590260    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:30:04.591067 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:04.591044    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:30:05.542185 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:05.542146    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:30:06.762051 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:06.762025    5508 patch_prober.go:29] interesting pod/e2e-ait-21c25193-openshift-default-967b44fd9-smz46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 01:30:06.762473 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:06.762089    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 01:30:07.183495 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:07.183465    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp" event={"ID":"969c5a71-7c35-466c-8b1d-2a864343c062","Type":"ContainerStarted","Data":"9d385fc2f96f5cddc42f6a3672cc93b41020ef43a6af2aaea76cd81fe81a744e"}
Nov 03 01:30:07.185479 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:07.185455    5508 generic.go:356] "Generic (PLEG): container finished" podID="a745a282-70f9-46f3-8a35-a64d6807040e" containerID="a05644fb4eaaf625384e83fd9bbec5dc50e84a2c986da995411909922440d54e" exitCode=0
Nov 03 01:30:07.185572 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:07.185515    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" event={"ID":"a745a282-70f9-46f3-8a35-a64d6807040e","Type":"ContainerDied","Data":"a05644fb4eaaf625384e83fd9bbec5dc50e84a2c986da995411909922440d54e"}
Nov 03 01:30:07.185853 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:07.185837    5508 scope.go:122] "RemoveContainer" containerID="a05644fb4eaaf625384e83fd9bbec5dc50e84a2c986da995411909922440d54e"
Nov 03 01:30:07.191327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:07.191298    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:30:07.191429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:07.191382    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:30:07.191429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:07.191415    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:30:07.336512 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:07.336411    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp" podStartSLOduration=1.901503184 podStartE2EDuration="7.336392675s" totalImagesPullingTime="5.434889491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:07.335411508 +0000 UTC m=+686.761017975" watchObservedRunningTime="2026-11-03 01:30:07.336392675 +0000 UTC m=+686.761999140"
Nov 03 01:30:08.191902 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:08.191867    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675" event={"ID":"4f03fce0-0280-4d45-82ae-59c40bbad92f","Type":"ContainerStarted","Data":"5192325592c225780c84d303de3752f2bdc192f7d399ee9f3326762b5ab61015"}
Nov 03 01:30:08.193466 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:08.193441    5508 generic.go:356] "Generic (PLEG): container finished" podID="969c5a71-7c35-466c-8b1d-2a864343c062" containerID="9d385fc2f96f5cddc42f6a3672cc93b41020ef43a6af2aaea76cd81fe81a744e" exitCode=60
Nov 03 01:30:08.193589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:08.193525    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp" event={"ID":"969c5a71-7c35-466c-8b1d-2a864343c062","Type":"ContainerDied","Data":"9d385fc2f96f5cddc42f6a3672cc93b41020ef43a6af2aaea76cd81fe81a744e"}
Nov 03 01:30:08.193709 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:08.193696    5508 scope.go:122] "RemoveContainer" containerID="9d385fc2f96f5cddc42f6a3672cc93b41020ef43a6af2aaea76cd81fe81a744e"
Nov 03 01:30:08.195125 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:08.195101    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx" event={"ID":"1b610877-52f4-4d89-b7ba-b65d4b7e4312","Type":"ContainerStarted","Data":"d5173bccbafb60880a8707d826427239c9a414a0646dd814cd005429971fd861"}
Nov 03 01:30:08.196662 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:08.196640    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm" event={"ID":"79520c6d-ffee-41d0-a8f6-0801d98196ba","Type":"ContainerStarted","Data":"571c5b16b4daeb69f18f214bbec5b5bc4e746713783b1b3de2239ab2bd030ee1"}
Nov 03 01:30:08.206382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:08.206362    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" event={"ID":"a745a282-70f9-46f3-8a35-a64d6807040e","Type":"ContainerStarted","Data":"33778f13edfa11483a8955014dfb9eb95f2d44e0005dcbbad55eac2154b74146"}
Nov 03 01:30:08.276376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:08.275873    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:30:08.276376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:08.275928    5508 patch_prober.go:29] interesting pod/e2e-ait-21c25193-openshift-default-967b44fd9-smz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 01:30:08.276376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:08.276007    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 01:30:08.299939 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:08.299878    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675" podStartSLOduration=2.384447763 podStartE2EDuration="8.299864588s" totalImagesPullingTime="5.915416825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:08.298453064 +0000 UTC m=+687.724059546" watchObservedRunningTime="2026-11-03 01:30:08.299864588 +0000 UTC m=+687.725471076"
Nov 03 01:30:08.407925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:08.407878    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx" podStartSLOduration=2.432726902 podStartE2EDuration="8.40786263s" totalImagesPullingTime="5.975135728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:08.405596557 +0000 UTC m=+687.831203022" watchObservedRunningTime="2026-11-03 01:30:08.40786263 +0000 UTC m=+687.833469090"
Nov 03 01:30:08.781092 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:08.780972    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm" podStartSLOduration=2.57961943 podStartE2EDuration="8.780955424s" totalImagesPullingTime="6.201335994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:08.779028928 +0000 UTC m=+688.204635411" watchObservedRunningTime="2026-11-03 01:30:08.780955424 +0000 UTC m=+688.206561887"
Nov 03 01:30:09.212465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.212439    5508 generic.go:356] "Generic (PLEG): container finished" podID="4f03fce0-0280-4d45-82ae-59c40bbad92f" containerID="5192325592c225780c84d303de3752f2bdc192f7d399ee9f3326762b5ab61015" exitCode=60
Nov 03 01:30:09.212830 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.212517    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675" event={"ID":"4f03fce0-0280-4d45-82ae-59c40bbad92f","Type":"ContainerDied","Data":"5192325592c225780c84d303de3752f2bdc192f7d399ee9f3326762b5ab61015"}
Nov 03 01:30:09.212830 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.212776    5508 scope.go:122] "RemoveContainer" containerID="5192325592c225780c84d303de3752f2bdc192f7d399ee9f3326762b5ab61015"
Nov 03 01:30:09.214733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.214709    5508 generic.go:356] "Generic (PLEG): container finished" podID="969c5a71-7c35-466c-8b1d-2a864343c062" containerID="e6a945e3e69b50b9e9b86193a6bafda5504035a2b3e5add83b67537eebcf3c87" exitCode=60
Nov 03 01:30:09.214838 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.214775    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp" event={"ID":"969c5a71-7c35-466c-8b1d-2a864343c062","Type":"ContainerDied","Data":"e6a945e3e69b50b9e9b86193a6bafda5504035a2b3e5add83b67537eebcf3c87"}
Nov 03 01:30:09.214838 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.214820    5508 scope.go:122] "RemoveContainer" containerID="9d385fc2f96f5cddc42f6a3672cc93b41020ef43a6af2aaea76cd81fe81a744e"
Nov 03 01:30:09.215026 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.214995    5508 scope.go:122] "RemoveContainer" containerID="e6a945e3e69b50b9e9b86193a6bafda5504035a2b3e5add83b67537eebcf3c87"
Nov 03 01:30:09.215225 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:09.215207    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp_odh-ai-gateway-infra(969c5a71-7c35-466c-8b1d-2a864343c062)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp" podUID="969c5a71-7c35-466c-8b1d-2a864343c062"
Nov 03 01:30:09.216862 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.216842    5508 generic.go:356] "Generic (PLEG): container finished" podID="1b610877-52f4-4d89-b7ba-b65d4b7e4312" containerID="d5173bccbafb60880a8707d826427239c9a414a0646dd814cd005429971fd861" exitCode=60
Nov 03 01:30:09.216977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.216944    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx" event={"ID":"1b610877-52f4-4d89-b7ba-b65d4b7e4312","Type":"ContainerDied","Data":"d5173bccbafb60880a8707d826427239c9a414a0646dd814cd005429971fd861"}
Nov 03 01:30:09.217156 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.217140    5508 scope.go:122] "RemoveContainer" containerID="d5173bccbafb60880a8707d826427239c9a414a0646dd814cd005429971fd861"
Nov 03 01:30:09.218547 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.218511    5508 generic.go:356] "Generic (PLEG): container finished" podID="79520c6d-ffee-41d0-a8f6-0801d98196ba" containerID="571c5b16b4daeb69f18f214bbec5b5bc4e746713783b1b3de2239ab2bd030ee1" exitCode=60
Nov 03 01:30:09.218648 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.218575    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm" event={"ID":"79520c6d-ffee-41d0-a8f6-0801d98196ba","Type":"ContainerDied","Data":"571c5b16b4daeb69f18f214bbec5b5bc4e746713783b1b3de2239ab2bd030ee1"}
Nov 03 01:30:09.218762 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.218749    5508 scope.go:122] "RemoveContainer" containerID="571c5b16b4daeb69f18f214bbec5b5bc4e746713783b1b3de2239ab2bd030ee1"
Nov 03 01:30:09.263787 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.263689    5508 patch_prober.go:29] interesting pod/e2e-ait-21c25193-openshift-default-967b44fd9-smz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 01:30:09.263787 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.263738    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 01:30:09.885676 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.885600    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"]
Nov 03 01:30:09.889483 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.889465    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"
Nov 03 01:30:09.939630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.939599    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn\" (UID: \"96b1640b-8375-46dc-aa36-038823af4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"
Nov 03 01:30:09.939758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.939636    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn\" (UID: \"96b1640b-8375-46dc-aa36-038823af4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"
Nov 03 01:30:09.939830 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.939759    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxqxf\" (UniqueName: \"kubernetes.io/projected/96b1640b-8375-46dc-aa36-038823af4c71-kube-api-access-jxqxf\") pod \"maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn\" (UID: \"96b1640b-8375-46dc-aa36-038823af4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"
Nov 03 01:30:09.946583 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.946563    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:30:09.952378 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:09.949414    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"]
Nov 03 01:30:10.040645 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.040609    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxqxf\" (UniqueName: \"kubernetes.io/projected/96b1640b-8375-46dc-aa36-038823af4c71-kube-api-access-jxqxf\") pod \"maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn\" (UID: \"96b1640b-8375-46dc-aa36-038823af4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"
Nov 03 01:30:10.040796 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.040706    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn\" (UID: \"96b1640b-8375-46dc-aa36-038823af4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"
Nov 03 01:30:10.040796 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.040728    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn\" (UID: \"96b1640b-8375-46dc-aa36-038823af4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"
Nov 03 01:30:10.041429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.041405    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn\" (UID: \"96b1640b-8375-46dc-aa36-038823af4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"
Nov 03 01:30:10.042976 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.042954    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn\" (UID: \"96b1640b-8375-46dc-aa36-038823af4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"
Nov 03 01:30:10.097616 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.097588    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxqxf\" (UniqueName: \"kubernetes.io/projected/96b1640b-8375-46dc-aa36-038823af4c71-kube-api-access-jxqxf\") pod \"maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn\" (UID: \"96b1640b-8375-46dc-aa36-038823af4c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"
Nov 03 01:30:10.201791 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.201759    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"
Nov 03 01:30:10.227284 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.227255    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675" event={"ID":"4f03fce0-0280-4d45-82ae-59c40bbad92f","Type":"ContainerStarted","Data":"33c77559061b7f9cf93a1fffbd55a5704ce4d33e6d98778fee37fbac80f64821"}
Nov 03 01:30:10.229219 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.229194    5508 scope.go:122] "RemoveContainer" containerID="e6a945e3e69b50b9e9b86193a6bafda5504035a2b3e5add83b67537eebcf3c87"
Nov 03 01:30:10.229603 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:10.229581    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp_odh-ai-gateway-infra(969c5a71-7c35-466c-8b1d-2a864343c062)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp" podUID="969c5a71-7c35-466c-8b1d-2a864343c062"
Nov 03 01:30:10.230647 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.230618    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx" event={"ID":"1b610877-52f4-4d89-b7ba-b65d4b7e4312","Type":"ContainerStarted","Data":"65ae4f4fe969c603f14c961cf7b983753c2e47ae914c164a01d2e0e746dc3584"}
Nov 03 01:30:10.232814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.232784    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm" event={"ID":"79520c6d-ffee-41d0-a8f6-0801d98196ba","Type":"ContainerStarted","Data":"384a2f96e9bac03249683a6ba6dc676d68431beeafaa0173d036de6e701872c3"}
Nov 03 01:30:10.262723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.255722    5508 patch_prober.go:29] interesting pod/e2e-ait-21c25193-openshift-default-967b44fd9-smz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 01:30:10.262723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.255779    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 01:30:10.599762 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:10.599684    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"]
Nov 03 01:30:10.606891 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:30:10.606860    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b1640b_8375_46dc_aa36_038823af4c71.slice/crio-d17888a4ace8d6365ac5e9882cb8d588bb68621fbef4a342de476e130345c13c WatchSource:0}: Error finding container d17888a4ace8d6365ac5e9882cb8d588bb68621fbef4a342de476e130345c13c: Status 404 returned error can't find the container with id d17888a4ace8d6365ac5e9882cb8d588bb68621fbef4a342de476e130345c13c
Nov 03 01:30:11.240039 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.240006    5508 generic.go:356] "Generic (PLEG): container finished" podID="1b610877-52f4-4d89-b7ba-b65d4b7e4312" containerID="65ae4f4fe969c603f14c961cf7b983753c2e47ae914c164a01d2e0e746dc3584" exitCode=60
Nov 03 01:30:11.240559 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.240092    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx" event={"ID":"1b610877-52f4-4d89-b7ba-b65d4b7e4312","Type":"ContainerDied","Data":"65ae4f4fe969c603f14c961cf7b983753c2e47ae914c164a01d2e0e746dc3584"}
Nov 03 01:30:11.240559 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.240158    5508 scope.go:122] "RemoveContainer" containerID="d5173bccbafb60880a8707d826427239c9a414a0646dd814cd005429971fd861"
Nov 03 01:30:11.240559 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.240395    5508 scope.go:122] "RemoveContainer" containerID="65ae4f4fe969c603f14c961cf7b983753c2e47ae914c164a01d2e0e746dc3584"
Nov 03 01:30:11.240726 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:11.240701    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx_odh-ai-gateway-infra(1b610877-52f4-4d89-b7ba-b65d4b7e4312)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx" podUID="1b610877-52f4-4d89-b7ba-b65d4b7e4312"
Nov 03 01:30:11.245548 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.245232    5508 generic.go:356] "Generic (PLEG): container finished" podID="79520c6d-ffee-41d0-a8f6-0801d98196ba" containerID="384a2f96e9bac03249683a6ba6dc676d68431beeafaa0173d036de6e701872c3" exitCode=60
Nov 03 01:30:11.245548 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.245376    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm" event={"ID":"79520c6d-ffee-41d0-a8f6-0801d98196ba","Type":"ContainerDied","Data":"384a2f96e9bac03249683a6ba6dc676d68431beeafaa0173d036de6e701872c3"}
Nov 03 01:30:11.245763 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.245746    5508 scope.go:122] "RemoveContainer" containerID="384a2f96e9bac03249683a6ba6dc676d68431beeafaa0173d036de6e701872c3"
Nov 03 01:30:11.246200 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:11.245994    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894490-qgwrm_odh-ai-gateway-infra(79520c6d-ffee-41d0-a8f6-0801d98196ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm" podUID="79520c6d-ffee-41d0-a8f6-0801d98196ba"
Nov 03 01:30:11.247645 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.247627    5508 generic.go:356] "Generic (PLEG): container finished" podID="4f03fce0-0280-4d45-82ae-59c40bbad92f" containerID="33c77559061b7f9cf93a1fffbd55a5704ce4d33e6d98778fee37fbac80f64821" exitCode=60
Nov 03 01:30:11.247726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.247697    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675" event={"ID":"4f03fce0-0280-4d45-82ae-59c40bbad92f","Type":"ContainerDied","Data":"33c77559061b7f9cf93a1fffbd55a5704ce4d33e6d98778fee37fbac80f64821"}
Nov 03 01:30:11.247942 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.247929    5508 scope.go:122] "RemoveContainer" containerID="33c77559061b7f9cf93a1fffbd55a5704ce4d33e6d98778fee37fbac80f64821"
Nov 03 01:30:11.248142 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:11.248120    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675_odh-ai-gateway-infra(4f03fce0-0280-4d45-82ae-59c40bbad92f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675" podUID="4f03fce0-0280-4d45-82ae-59c40bbad92f"
Nov 03 01:30:11.264753 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.264711    5508 patch_prober.go:29] interesting pod/e2e-ait-21c25193-openshift-default-967b44fd9-smz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 01:30:11.264927 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.264779    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 01:30:11.268377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.266773    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn" event={"ID":"96b1640b-8375-46dc-aa36-038823af4c71","Type":"ContainerStarted","Data":"176dbd36ecfaba153afe0f62edf87aa605118d9ed521ab50dd5cbe470099dcea"}
Nov 03 01:30:11.268377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.266827    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn" event={"ID":"96b1640b-8375-46dc-aa36-038823af4c71","Type":"ContainerStarted","Data":"d17888a4ace8d6365ac5e9882cb8d588bb68621fbef4a342de476e130345c13c"}
Nov 03 01:30:11.283373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.282518    5508 scope.go:122] "RemoveContainer" containerID="571c5b16b4daeb69f18f214bbec5b5bc4e746713783b1b3de2239ab2bd030ee1"
Nov 03 01:30:11.295425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.295336    5508 scope.go:122] "RemoveContainer" containerID="5192325592c225780c84d303de3752f2bdc192f7d399ee9f3326762b5ab61015"
Nov 03 01:30:11.470207 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:11.461247    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn" podStartSLOduration=2.461228084 podStartE2EDuration="2.461228084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:11.459537202 +0000 UTC m=+690.885143668" watchObservedRunningTime="2026-11-03 01:30:11.461228084 +0000 UTC m=+690.886834551"
Nov 03 01:30:12.262375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:12.255695    5508 patch_prober.go:29] interesting pod/e2e-ait-21c25193-openshift-default-967b44fd9-smz46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Nov 03 01:30:12.262375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:12.255753    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Nov 03 01:30:12.280234 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:12.280210    5508 scope.go:122] "RemoveContainer" containerID="65ae4f4fe969c603f14c961cf7b983753c2e47ae914c164a01d2e0e746dc3584"
Nov 03 01:30:12.280504 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:12.280483    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx_odh-ai-gateway-infra(1b610877-52f4-4d89-b7ba-b65d4b7e4312)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx" podUID="1b610877-52f4-4d89-b7ba-b65d4b7e4312"
Nov 03 01:30:12.281667 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:12.281648    5508 scope.go:122] "RemoveContainer" containerID="384a2f96e9bac03249683a6ba6dc676d68431beeafaa0173d036de6e701872c3"
Nov 03 01:30:12.281884 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:12.281867    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894490-qgwrm_odh-ai-gateway-infra(79520c6d-ffee-41d0-a8f6-0801d98196ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm" podUID="79520c6d-ffee-41d0-a8f6-0801d98196ba"
Nov 03 01:30:12.283039 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:12.283023    5508 scope.go:122] "RemoveContainer" containerID="33c77559061b7f9cf93a1fffbd55a5704ce4d33e6d98778fee37fbac80f64821"
Nov 03 01:30:12.283211 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:12.283192    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675_odh-ai-gateway-infra(4f03fce0-0280-4d45-82ae-59c40bbad92f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675" podUID="4f03fce0-0280-4d45-82ae-59c40bbad92f"
Nov 03 01:30:12.284308 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:12.284287    5508 generic.go:356] "Generic (PLEG): container finished" podID="96b1640b-8375-46dc-aa36-038823af4c71" containerID="176dbd36ecfaba153afe0f62edf87aa605118d9ed521ab50dd5cbe470099dcea" exitCode=0
Nov 03 01:30:12.284403 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:12.284329    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn" event={"ID":"96b1640b-8375-46dc-aa36-038823af4c71","Type":"ContainerDied","Data":"176dbd36ecfaba153afe0f62edf87aa605118d9ed521ab50dd5cbe470099dcea"}
Nov 03 01:30:13.259966 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:13.259936    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:30:13.260221 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:13.260203    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:30:13.261028 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:13.261007    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:30:13.433695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:13.433672    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"
Nov 03 01:30:13.497152 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:13.497121    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-openshift-ca\") pod \"96b1640b-8375-46dc-aa36-038823af4c71\" (UID: \"96b1640b-8375-46dc-aa36-038823af4c71\") "
Nov 03 01:30:13.497152 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:13.497160    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b1640b-8375-46dc-aa36-038823af4c71-kube-api-access-jxqxf\" (UniqueName: \"kubernetes.io/projected/96b1640b-8375-46dc-aa36-038823af4c71-kube-api-access-jxqxf\") pod \"96b1640b-8375-46dc-aa36-038823af4c71\" (UID: \"96b1640b-8375-46dc-aa36-038823af4c71\") "
Nov 03 01:30:13.497396 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:13.497314    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-xks-ca\") pod \"96b1640b-8375-46dc-aa36-038823af4c71\" (UID: \"96b1640b-8375-46dc-aa36-038823af4c71\") "
Nov 03 01:30:13.497607 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:13.497581    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-openshift-ca" pod "96b1640b-8375-46dc-aa36-038823af4c71" (UID: "96b1640b-8375-46dc-aa36-038823af4c71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:13.499453 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:13.499425    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b1640b-8375-46dc-aa36-038823af4c71-kube-api-access-jxqxf" pod "96b1640b-8375-46dc-aa36-038823af4c71" (UID: "96b1640b-8375-46dc-aa36-038823af4c71"). InnerVolumeSpecName "kube-api-access-jxqxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:13.499574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:13.499470    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-xks-ca" pod "96b1640b-8375-46dc-aa36-038823af4c71" (UID: "96b1640b-8375-46dc-aa36-038823af4c71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:30:13.598338 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:13.598258    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:13.598338 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:13.598289    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96b1640b-8375-46dc-aa36-038823af4c71-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:13.598338 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:13.598299    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxqxf\" (UniqueName: \"kubernetes.io/projected/96b1640b-8375-46dc-aa36-038823af4c71-kube-api-access-jxqxf\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:13.812876 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:13.812838    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:30:14.298836 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.298808    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"
Nov 03 01:30:14.299013 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.298845    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn" event={"ID":"96b1640b-8375-46dc-aa36-038823af4c71","Type":"ContainerDied","Data":"d17888a4ace8d6365ac5e9882cb8d588bb68621fbef4a342de476e130345c13c"}
Nov 03 01:30:14.299013 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.298899    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d17888a4ace8d6365ac5e9882cb8d588bb68621fbef4a342de476e130345c13c"
Nov 03 01:30:14.762779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.762745    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"]
Nov 03 01:30:14.763186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.763174    5508 cpu_manager.go:414] "containerMap: removing container" podUID="96b1640b-8375-46dc-aa36-038823af4c71" containerName="revoke-keys"
Nov 03 01:30:14.763228 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.763187    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b1640b-8375-46dc-aa36-038823af4c71" containerName="revoke-keys"
Nov 03 01:30:14.763275 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.763267    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b1640b-8375-46dc-aa36-038823af4c71" containerName="revoke-keys"
Nov 03 01:30:14.766655 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.766639    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"
Nov 03 01:30:14.817343 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.817274    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:30:14.820519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.820491    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"]
Nov 03 01:30:14.822276 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.822249    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f\" (UID: \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"
Nov 03 01:30:14.822395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.822306    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghvmz\" (UniqueName: \"kubernetes.io/projected/4e4980a1-1514-4f0c-b370-bf7222ab84a3-kube-api-access-ghvmz\") pod \"maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f\" (UID: \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"
Nov 03 01:30:14.822473 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.822447    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f\" (UID: \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"
Nov 03 01:30:14.923180 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.923153    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f\" (UID: \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"
Nov 03 01:30:14.923322 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.923196    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghvmz\" (UniqueName: \"kubernetes.io/projected/4e4980a1-1514-4f0c-b370-bf7222ab84a3-kube-api-access-ghvmz\") pod \"maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f\" (UID: \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"
Nov 03 01:30:14.923322 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.923233    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f\" (UID: \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"
Nov 03 01:30:14.923872 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.923848    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f\" (UID: \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"
Nov 03 01:30:14.925520 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.925496    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f\" (UID: \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"
Nov 03 01:30:14.983507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:14.983476    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghvmz\" (UniqueName: \"kubernetes.io/projected/4e4980a1-1514-4f0c-b370-bf7222ab84a3-kube-api-access-ghvmz\") pod \"maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f\" (UID: \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"
Nov 03 01:30:15.078763 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:15.078669    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"
Nov 03 01:30:15.450614 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:15.450581    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"]
Nov 03 01:30:15.459791 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:30:15.459761    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4980a1_1514_4f0c_b370_bf7222ab84a3.slice/crio-c8094b976fbd188c22b405eb404519d9c95fc7a3e05a62e2e600379cb9088828 WatchSource:0}: Error finding container c8094b976fbd188c22b405eb404519d9c95fc7a3e05a62e2e600379cb9088828: Status 404 returned error can't find the container with id c8094b976fbd188c22b405eb404519d9c95fc7a3e05a62e2e600379cb9088828
Nov 03 01:30:16.312465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:16.312432    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f" event={"ID":"4e4980a1-1514-4f0c-b370-bf7222ab84a3","Type":"ContainerStarted","Data":"4aca470736cb38737e446dce3467c076c34ea6f1f95b1771e1e3ddd1ef67987b"}
Nov 03 01:30:16.312822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:16.312479    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f" event={"ID":"4e4980a1-1514-4f0c-b370-bf7222ab84a3","Type":"ContainerStarted","Data":"c8094b976fbd188c22b405eb404519d9c95fc7a3e05a62e2e600379cb9088828"}
Nov 03 01:30:16.422253 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:16.422198    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f" podStartSLOduration=2.422185035 podStartE2EDuration="2.422185035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:16.420408959 +0000 UTC m=+695.846015433" watchObservedRunningTime="2026-11-03 01:30:16.422185035 +0000 UTC m=+695.847791569"
Nov 03 01:30:17.319178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:17.319144    5508 generic.go:356] "Generic (PLEG): container finished" podID="4e4980a1-1514-4f0c-b370-bf7222ab84a3" containerID="4aca470736cb38737e446dce3467c076c34ea6f1f95b1771e1e3ddd1ef67987b" exitCode=0
Nov 03 01:30:17.319659 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:17.319222    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f" event={"ID":"4e4980a1-1514-4f0c-b370-bf7222ab84a3","Type":"ContainerDied","Data":"4aca470736cb38737e446dce3467c076c34ea6f1f95b1771e1e3ddd1ef67987b"}
Nov 03 01:30:18.450127 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:18.450100    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"
Nov 03 01:30:18.563332 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:18.563300    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-openshift-ca\") pod \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\" (UID: \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\") "
Nov 03 01:30:18.563511 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:18.563418    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-xks-ca\") pod \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\" (UID: \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\") "
Nov 03 01:30:18.563511 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:18.563462    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4980a1-1514-4f0c-b370-bf7222ab84a3-kube-api-access-ghvmz\" (UniqueName: \"kubernetes.io/projected/4e4980a1-1514-4f0c-b370-bf7222ab84a3-kube-api-access-ghvmz\") pod \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\" (UID: \"4e4980a1-1514-4f0c-b370-bf7222ab84a3\") "
Nov 03 01:30:18.563698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:18.563671    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-openshift-ca" pod "4e4980a1-1514-4f0c-b370-bf7222ab84a3" (UID: "4e4980a1-1514-4f0c-b370-bf7222ab84a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:18.563794 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:18.563777    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:18.566611 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:18.566584    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-xks-ca" pod "4e4980a1-1514-4f0c-b370-bf7222ab84a3" (UID: "4e4980a1-1514-4f0c-b370-bf7222ab84a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:30:18.569129 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:18.569095    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4980a1-1514-4f0c-b370-bf7222ab84a3-kube-api-access-ghvmz" pod "4e4980a1-1514-4f0c-b370-bf7222ab84a3" (UID: "4e4980a1-1514-4f0c-b370-bf7222ab84a3"). InnerVolumeSpecName "kube-api-access-ghvmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:18.665098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:18.665071    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e4980a1-1514-4f0c-b370-bf7222ab84a3-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:18.665098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:18.665098    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghvmz\" (UniqueName: \"kubernetes.io/projected/4e4980a1-1514-4f0c-b370-bf7222ab84a3-kube-api-access-ghvmz\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:19.331262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:19.331232    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"
Nov 03 01:30:19.331464 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:19.331234    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f" event={"ID":"4e4980a1-1514-4f0c-b370-bf7222ab84a3","Type":"ContainerDied","Data":"c8094b976fbd188c22b405eb404519d9c95fc7a3e05a62e2e600379cb9088828"}
Nov 03 01:30:19.331464 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:19.331375    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8094b976fbd188c22b405eb404519d9c95fc7a3e05a62e2e600379cb9088828"
Nov 03 01:30:21.684880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:21.684846    5508 scope.go:122] "RemoveContainer" containerID="384a2f96e9bac03249683a6ba6dc676d68431beeafaa0173d036de6e701872c3"
Nov 03 01:30:22.349910 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:22.349832    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm" event={"ID":"79520c6d-ffee-41d0-a8f6-0801d98196ba","Type":"ContainerStarted","Data":"04f1dc0621418c7a6fad549df93013045dc0c8b552d6730aabdbcba6172761d5"}
Nov 03 01:30:22.691734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:22.691603    5508 scope.go:122] "RemoveContainer" containerID="65ae4f4fe969c603f14c961cf7b983753c2e47ae914c164a01d2e0e746dc3584"
Nov 03 01:30:23.358084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:23.357998    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx" event={"ID":"1b610877-52f4-4d89-b7ba-b65d4b7e4312","Type":"ContainerStarted","Data":"e7f50fa89a248c78da63d4c61a52590ec051448dd4fbe7a439cdee50f8ddd2c5"}
Nov 03 01:30:23.359869 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:23.359847    5508 generic.go:356] "Generic (PLEG): container finished" podID="79520c6d-ffee-41d0-a8f6-0801d98196ba" containerID="04f1dc0621418c7a6fad549df93013045dc0c8b552d6730aabdbcba6172761d5" exitCode=60
Nov 03 01:30:23.360001 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:23.359910    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm" event={"ID":"79520c6d-ffee-41d0-a8f6-0801d98196ba","Type":"ContainerDied","Data":"04f1dc0621418c7a6fad549df93013045dc0c8b552d6730aabdbcba6172761d5"}
Nov 03 01:30:23.360001 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:23.359952    5508 scope.go:122] "RemoveContainer" containerID="384a2f96e9bac03249683a6ba6dc676d68431beeafaa0173d036de6e701872c3"
Nov 03 01:30:23.360223 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:23.360198    5508 scope.go:122] "RemoveContainer" containerID="04f1dc0621418c7a6fad549df93013045dc0c8b552d6730aabdbcba6172761d5"
Nov 03 01:30:23.360465 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:23.360441    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894490-qgwrm_odh-ai-gateway-infra(79520c6d-ffee-41d0-a8f6-0801d98196ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm" podUID="79520c6d-ffee-41d0-a8f6-0801d98196ba"
Nov 03 01:30:23.467432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:23.467400    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"]
Nov 03 01:30:23.684549 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:23.684519    5508 scope.go:122] "RemoveContainer" containerID="e6a945e3e69b50b9e9b86193a6bafda5504035a2b3e5add83b67537eebcf3c87"
Nov 03 01:30:24.369554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.369527    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp" event={"ID":"969c5a71-7c35-466c-8b1d-2a864343c062","Type":"ContainerStarted","Data":"de122457bf8651e01fbb8c494b0fc1408b34313a3f113202aec0aad6b187018c"}
Nov 03 01:30:24.369885 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.369873    5508 scope.go:122] "RemoveContainer" containerID="de122457bf8651e01fbb8c494b0fc1408b34313a3f113202aec0aad6b187018c"
Nov 03 01:30:24.370140 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:24.370118    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp_odh-ai-gateway-infra(969c5a71-7c35-466c-8b1d-2a864343c062)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp" podUID="969c5a71-7c35-466c-8b1d-2a864343c062"
Nov 03 01:30:24.371536 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.371515    5508 generic.go:356] "Generic (PLEG): container finished" podID="1b610877-52f4-4d89-b7ba-b65d4b7e4312" containerID="e7f50fa89a248c78da63d4c61a52590ec051448dd4fbe7a439cdee50f8ddd2c5" exitCode=60
Nov 03 01:30:24.371624 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.371561    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx" event={"ID":"1b610877-52f4-4d89-b7ba-b65d4b7e4312","Type":"ContainerDied","Data":"e7f50fa89a248c78da63d4c61a52590ec051448dd4fbe7a439cdee50f8ddd2c5"}
Nov 03 01:30:24.371624 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.371598    5508 scope.go:122] "RemoveContainer" containerID="65ae4f4fe969c603f14c961cf7b983753c2e47ae914c164a01d2e0e746dc3584"
Nov 03 01:30:24.371879 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.371861    5508 scope.go:122] "RemoveContainer" containerID="e7f50fa89a248c78da63d4c61a52590ec051448dd4fbe7a439cdee50f8ddd2c5"
Nov 03 01:30:24.372102 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:24.372084    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx_odh-ai-gateway-infra(1b610877-52f4-4d89-b7ba-b65d4b7e4312)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx" podUID="1b610877-52f4-4d89-b7ba-b65d4b7e4312"
Nov 03 01:30:24.481412 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.481383    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"]
Nov 03 01:30:24.495781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.495758    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"
Nov 03 01:30:24.640860 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.640785    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79520c6d-ffee-41d0-a8f6-0801d98196ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79520c6d-ffee-41d0-a8f6-0801d98196ba-maas-api-service-ca\") pod \"79520c6d-ffee-41d0-a8f6-0801d98196ba\" (UID: \"79520c6d-ffee-41d0-a8f6-0801d98196ba\") "
Nov 03 01:30:24.641028 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.640918    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79520c6d-ffee-41d0-a8f6-0801d98196ba-kube-api-access-9xxb9\" (UniqueName: \"kubernetes.io/projected/79520c6d-ffee-41d0-a8f6-0801d98196ba-kube-api-access-9xxb9\") pod \"79520c6d-ffee-41d0-a8f6-0801d98196ba\" (UID: \"79520c6d-ffee-41d0-a8f6-0801d98196ba\") "
Nov 03 01:30:24.641170 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.641148    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79520c6d-ffee-41d0-a8f6-0801d98196ba-maas-api-service-ca" pod "79520c6d-ffee-41d0-a8f6-0801d98196ba" (UID: "79520c6d-ffee-41d0-a8f6-0801d98196ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:24.642961 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.642944    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79520c6d-ffee-41d0-a8f6-0801d98196ba-kube-api-access-9xxb9" pod "79520c6d-ffee-41d0-a8f6-0801d98196ba" (UID: "79520c6d-ffee-41d0-a8f6-0801d98196ba"). InnerVolumeSpecName "kube-api-access-9xxb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:24.685295 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.685260    5508 scope.go:122] "RemoveContainer" containerID="33c77559061b7f9cf93a1fffbd55a5704ce4d33e6d98778fee37fbac80f64821"
Nov 03 01:30:24.742327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.742305    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xxb9\" (UniqueName: \"kubernetes.io/projected/79520c6d-ffee-41d0-a8f6-0801d98196ba-kube-api-access-9xxb9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:24.742327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:24.742327    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79520c6d-ffee-41d0-a8f6-0801d98196ba-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:25.378147 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.378124    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"
Nov 03 01:30:25.378573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.378148    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm" event={"ID":"79520c6d-ffee-41d0-a8f6-0801d98196ba","Type":"ContainerDied","Data":"738020c2958231c8fbd7647a3204febf02ac667c4cb9afef4c80dd663f1a5aba"}
Nov 03 01:30:25.378573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.378209    5508 scope.go:122] "RemoveContainer" containerID="04f1dc0621418c7a6fad549df93013045dc0c8b552d6730aabdbcba6172761d5"
Nov 03 01:30:25.380095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.380072    5508 generic.go:356] "Generic (PLEG): container finished" podID="4f03fce0-0280-4d45-82ae-59c40bbad92f" containerID="71712129136faf7e95a3030c068ac8ae83e82bbe299ebf699a13a5dcd0bfe9a2" exitCode=60
Nov 03 01:30:25.380206 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.380140    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675" event={"ID":"4f03fce0-0280-4d45-82ae-59c40bbad92f","Type":"ContainerDied","Data":"71712129136faf7e95a3030c068ac8ae83e82bbe299ebf699a13a5dcd0bfe9a2"}
Nov 03 01:30:25.380497 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.380481    5508 scope.go:122] "RemoveContainer" containerID="71712129136faf7e95a3030c068ac8ae83e82bbe299ebf699a13a5dcd0bfe9a2"
Nov 03 01:30:25.380747 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:25.380725    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675_odh-ai-gateway-infra(4f03fce0-0280-4d45-82ae-59c40bbad92f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675" podUID="4f03fce0-0280-4d45-82ae-59c40bbad92f"
Nov 03 01:30:25.382282 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.382264    5508 generic.go:356] "Generic (PLEG): container finished" podID="969c5a71-7c35-466c-8b1d-2a864343c062" containerID="de122457bf8651e01fbb8c494b0fc1408b34313a3f113202aec0aad6b187018c" exitCode=60
Nov 03 01:30:25.382384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.382335    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp" event={"ID":"969c5a71-7c35-466c-8b1d-2a864343c062","Type":"ContainerDied","Data":"de122457bf8651e01fbb8c494b0fc1408b34313a3f113202aec0aad6b187018c"}
Nov 03 01:30:25.382710 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.382692    5508 scope.go:122] "RemoveContainer" containerID="de122457bf8651e01fbb8c494b0fc1408b34313a3f113202aec0aad6b187018c"
Nov 03 01:30:25.382936 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:25.382914    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp_odh-ai-gateway-infra(969c5a71-7c35-466c-8b1d-2a864343c062)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp" podUID="969c5a71-7c35-466c-8b1d-2a864343c062"
Nov 03 01:30:25.417234 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.417211    5508 scope.go:122] "RemoveContainer" containerID="33c77559061b7f9cf93a1fffbd55a5704ce4d33e6d98778fee37fbac80f64821"
Nov 03 01:30:25.429873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.429851    5508 scope.go:122] "RemoveContainer" containerID="e6a945e3e69b50b9e9b86193a6bafda5504035a2b3e5add83b67537eebcf3c87"
Nov 03 01:30:25.509459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.509440    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"
Nov 03 01:30:25.649666 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.649636    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b610877-52f4-4d89-b7ba-b65d4b7e4312-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b610877-52f4-4d89-b7ba-b65d4b7e4312-maas-api-service-ca\") pod \"1b610877-52f4-4d89-b7ba-b65d4b7e4312\" (UID: \"1b610877-52f4-4d89-b7ba-b65d4b7e4312\") "
Nov 03 01:30:25.649833 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.649700    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b610877-52f4-4d89-b7ba-b65d4b7e4312-kube-api-access-sgvhv\" (UniqueName: \"kubernetes.io/projected/1b610877-52f4-4d89-b7ba-b65d4b7e4312-kube-api-access-sgvhv\") pod \"1b610877-52f4-4d89-b7ba-b65d4b7e4312\" (UID: \"1b610877-52f4-4d89-b7ba-b65d4b7e4312\") "
Nov 03 01:30:25.650049 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.650022    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b610877-52f4-4d89-b7ba-b65d4b7e4312-maas-api-service-ca" pod "1b610877-52f4-4d89-b7ba-b65d4b7e4312" (UID: "1b610877-52f4-4d89-b7ba-b65d4b7e4312"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:25.651757 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.651741    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b610877-52f4-4d89-b7ba-b65d4b7e4312-kube-api-access-sgvhv" pod "1b610877-52f4-4d89-b7ba-b65d4b7e4312" (UID: "1b610877-52f4-4d89-b7ba-b65d4b7e4312"). InnerVolumeSpecName "kube-api-access-sgvhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:25.701761 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.701722    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"]
Nov 03 01:30:25.750777 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.750740    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b610877-52f4-4d89-b7ba-b65d4b7e4312-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:25.750777 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.750769    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgvhv\" (UniqueName: \"kubernetes.io/projected/1b610877-52f4-4d89-b7ba-b65d4b7e4312-kube-api-access-sgvhv\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:25.857920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.857887    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"]
Nov 03 01:30:25.863582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:25.863554    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-qgwrm"]
Nov 03 01:30:26.392904 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.392878    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"
Nov 03 01:30:26.393311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.392921    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx" event={"ID":"1b610877-52f4-4d89-b7ba-b65d4b7e4312","Type":"ContainerDied","Data":"36f063023a820333a51e4e65f496c13e7c51c432e1ebb591cbfc2fb97e9b515f"}
Nov 03 01:30:26.393311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.393028    5508 scope.go:122] "RemoveContainer" containerID="e7f50fa89a248c78da63d4c61a52590ec051448dd4fbe7a439cdee50f8ddd2c5"
Nov 03 01:30:26.537700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.537677    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"
Nov 03 01:30:26.541294 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.541263    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"]
Nov 03 01:30:26.604385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.604358    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"]
Nov 03 01:30:26.608864 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.608841    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-fc87c747-29894490-pfwrx"]
Nov 03 01:30:26.658994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.658971    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969c5a71-7c35-466c-8b1d-2a864343c062-kube-api-access-gm29k\" (UniqueName: \"kubernetes.io/projected/969c5a71-7c35-466c-8b1d-2a864343c062-kube-api-access-gm29k\") pod \"969c5a71-7c35-466c-8b1d-2a864343c062\" (UID: \"969c5a71-7c35-466c-8b1d-2a864343c062\") "
Nov 03 01:30:26.659143 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.659014    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/969c5a71-7c35-466c-8b1d-2a864343c062-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c5a71-7c35-466c-8b1d-2a864343c062-maas-api-service-ca\") pod \"969c5a71-7c35-466c-8b1d-2a864343c062\" (UID: \"969c5a71-7c35-466c-8b1d-2a864343c062\") "
Nov 03 01:30:26.659420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.659401    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969c5a71-7c35-466c-8b1d-2a864343c062-maas-api-service-ca" pod "969c5a71-7c35-466c-8b1d-2a864343c062" (UID: "969c5a71-7c35-466c-8b1d-2a864343c062"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:26.660981 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.660962    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969c5a71-7c35-466c-8b1d-2a864343c062-kube-api-access-gm29k" pod "969c5a71-7c35-466c-8b1d-2a864343c062" (UID: "969c5a71-7c35-466c-8b1d-2a864343c062"). InnerVolumeSpecName "kube-api-access-gm29k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:26.691119 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.691092    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b610877-52f4-4d89-b7ba-b65d4b7e4312" path="/var/lib/kubelet/pods/1b610877-52f4-4d89-b7ba-b65d4b7e4312/volumes"
Nov 03 01:30:26.691576 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.691556    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79520c6d-ffee-41d0-a8f6-0801d98196ba" path="/var/lib/kubelet/pods/79520c6d-ffee-41d0-a8f6-0801d98196ba/volumes"
Nov 03 01:30:26.760446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.760417    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm29k\" (UniqueName: \"kubernetes.io/projected/969c5a71-7c35-466c-8b1d-2a864343c062-kube-api-access-gm29k\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:26.760446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.760440    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/969c5a71-7c35-466c-8b1d-2a864343c062-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:26.764716 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.764691    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"]
Nov 03 01:30:26.966795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:26.966468    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"
Nov 03 01:30:27.072681 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.072652    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f03fce0-0280-4d45-82ae-59c40bbad92f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f03fce0-0280-4d45-82ae-59c40bbad92f-maas-api-service-ca\") pod \"4f03fce0-0280-4d45-82ae-59c40bbad92f\" (UID: \"4f03fce0-0280-4d45-82ae-59c40bbad92f\") "
Nov 03 01:30:27.072866 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.072721    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f03fce0-0280-4d45-82ae-59c40bbad92f-kube-api-access-k5g9c\" (UniqueName: \"kubernetes.io/projected/4f03fce0-0280-4d45-82ae-59c40bbad92f-kube-api-access-k5g9c\") pod \"4f03fce0-0280-4d45-82ae-59c40bbad92f\" (UID: \"4f03fce0-0280-4d45-82ae-59c40bbad92f\") "
Nov 03 01:30:27.073057 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.073035    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f03fce0-0280-4d45-82ae-59c40bbad92f-maas-api-service-ca" pod "4f03fce0-0280-4d45-82ae-59c40bbad92f" (UID: "4f03fce0-0280-4d45-82ae-59c40bbad92f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:27.073126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.073112    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f03fce0-0280-4d45-82ae-59c40bbad92f-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:27.075225 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.075198    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f03fce0-0280-4d45-82ae-59c40bbad92f-kube-api-access-k5g9c" pod "4f03fce0-0280-4d45-82ae-59c40bbad92f" (UID: "4f03fce0-0280-4d45-82ae-59c40bbad92f"). InnerVolumeSpecName "kube-api-access-k5g9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:27.174298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.174258    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5g9c\" (UniqueName: \"kubernetes.io/projected/4f03fce0-0280-4d45-82ae-59c40bbad92f-kube-api-access-k5g9c\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:27.402507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.402480    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"
Nov 03 01:30:27.402979 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.402476    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp" event={"ID":"969c5a71-7c35-466c-8b1d-2a864343c062","Type":"ContainerDied","Data":"8a8da252876b90e7a72ff8788034e6d9b1c4edeaa3e9fe5c7b02345846fe1f0b"}
Nov 03 01:30:27.402979 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.402607    5508 scope.go:122] "RemoveContainer" containerID="de122457bf8651e01fbb8c494b0fc1408b34313a3f113202aec0aad6b187018c"
Nov 03 01:30:27.405621 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.405592    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675" event={"ID":"4f03fce0-0280-4d45-82ae-59c40bbad92f","Type":"ContainerDied","Data":"f2c4cae09b13c1c7f28c82895d2d388a5f0fadf7beacd4a88132555a52a36d32"}
Nov 03 01:30:27.405719 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.405671    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"
Nov 03 01:30:27.411700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.411679    5508 scope.go:122] "RemoveContainer" containerID="71712129136faf7e95a3030c068ac8ae83e82bbe299ebf699a13a5dcd0bfe9a2"
Nov 03 01:30:27.562157 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.562129    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"]
Nov 03 01:30:27.566098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.566074    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-e3f0a557-29894490-7qnfp"]
Nov 03 01:30:27.670236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.670174    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"]
Nov 03 01:30:27.676412 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:27.676393    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-98c7cd16-29894490-rc675"]
Nov 03 01:30:28.690264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:28.690232    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f03fce0-0280-4d45-82ae-59c40bbad92f" path="/var/lib/kubelet/pods/4f03fce0-0280-4d45-82ae-59c40bbad92f/volumes"
Nov 03 01:30:28.690698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:28.690682    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="969c5a71-7c35-466c-8b1d-2a864343c062" path="/var/lib/kubelet/pods/969c5a71-7c35-466c-8b1d-2a864343c062/volumes"
Nov 03 01:30:28.779923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:28.779893    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"]
Nov 03 01:30:28.879244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:28.879219    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"]
Nov 03 01:30:30.253012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.252975    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"]
Nov 03 01:30:30.253430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253413    5508 cpu_manager.go:414] "containerMap: removing container" podUID="79520c6d-ffee-41d0-a8f6-0801d98196ba" containerName="cleanup"
Nov 03 01:30:30.253430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253425    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79520c6d-ffee-41d0-a8f6-0801d98196ba" containerName="cleanup"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253433    5508 cpu_manager.go:414] "containerMap: removing container" podUID="1b610877-52f4-4d89-b7ba-b65d4b7e4312" containerName="cleanup"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253439    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b610877-52f4-4d89-b7ba-b65d4b7e4312" containerName="cleanup"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253446    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4f03fce0-0280-4d45-82ae-59c40bbad92f" containerName="cleanup"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253452    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f03fce0-0280-4d45-82ae-59c40bbad92f" containerName="cleanup"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253464    5508 cpu_manager.go:414] "containerMap: removing container" podUID="969c5a71-7c35-466c-8b1d-2a864343c062" containerName="cleanup"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253468    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969c5a71-7c35-466c-8b1d-2a864343c062" containerName="cleanup"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253480    5508 cpu_manager.go:414] "containerMap: removing container" podUID="969c5a71-7c35-466c-8b1d-2a864343c062" containerName="cleanup"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253484    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969c5a71-7c35-466c-8b1d-2a864343c062" containerName="cleanup"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253490    5508 cpu_manager.go:414] "containerMap: removing container" podUID="79520c6d-ffee-41d0-a8f6-0801d98196ba" containerName="cleanup"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253494    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79520c6d-ffee-41d0-a8f6-0801d98196ba" containerName="cleanup"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253503    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4f03fce0-0280-4d45-82ae-59c40bbad92f" containerName="cleanup"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253507    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f03fce0-0280-4d45-82ae-59c40bbad92f" containerName="cleanup"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253514    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4e4980a1-1514-4f0c-b370-bf7222ab84a3" containerName="revoke-keys"
Nov 03 01:30:30.253528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253518    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4980a1-1514-4f0c-b370-bf7222ab84a3" containerName="revoke-keys"
Nov 03 01:30:30.253891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253539    5508 cpu_manager.go:414] "containerMap: removing container" podUID="1b610877-52f4-4d89-b7ba-b65d4b7e4312" containerName="cleanup"
Nov 03 01:30:30.253891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253544    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b610877-52f4-4d89-b7ba-b65d4b7e4312" containerName="cleanup"
Nov 03 01:30:30.253891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253611    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4980a1-1514-4f0c-b370-bf7222ab84a3" containerName="revoke-keys"
Nov 03 01:30:30.253891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253617    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="79520c6d-ffee-41d0-a8f6-0801d98196ba" containerName="cleanup"
Nov 03 01:30:30.253891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253622    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b610877-52f4-4d89-b7ba-b65d4b7e4312" containerName="cleanup"
Nov 03 01:30:30.253891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253627    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f03fce0-0280-4d45-82ae-59c40bbad92f" containerName="cleanup"
Nov 03 01:30:30.253891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253633    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="969c5a71-7c35-466c-8b1d-2a864343c062" containerName="cleanup"
Nov 03 01:30:30.253891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253641    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="79520c6d-ffee-41d0-a8f6-0801d98196ba" containerName="cleanup"
Nov 03 01:30:30.253891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253645    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b610877-52f4-4d89-b7ba-b65d4b7e4312" containerName="cleanup"
Nov 03 01:30:30.253891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253650    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f03fce0-0280-4d45-82ae-59c40bbad92f" containerName="cleanup"
Nov 03 01:30:30.253891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253655    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="969c5a71-7c35-466c-8b1d-2a864343c062" containerName="cleanup"
Nov 03 01:30:30.253891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.253660    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="969c5a71-7c35-466c-8b1d-2a864343c062" containerName="cleanup"
Nov 03 01:30:30.258274 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.258257    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"
Nov 03 01:30:30.308015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.307974    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:30:30.308015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.308010    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:30:30.308015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.308015    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:30:30.310720 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.310695    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-21c25193-7nfs7\" (UID: \"92b2241a-30ba-489b-bc0d-15cd4e44313d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"
Nov 03 01:30:30.310854 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.310740    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnmfd\" (UniqueName: \"kubernetes.io/projected/92b2241a-30ba-489b-bc0d-15cd4e44313d-kube-api-access-cnmfd\") pod \"maas-api-revoke-keys-e2e-ait-21c25193-7nfs7\" (UID: \"92b2241a-30ba-489b-bc0d-15cd4e44313d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"
Nov 03 01:30:30.310958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.310938    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-21c25193-7nfs7\" (UID: \"92b2241a-30ba-489b-bc0d-15cd4e44313d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"
Nov 03 01:30:30.311756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.311734    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"]
Nov 03 01:30:30.351237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.351210    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:30:30.411642 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.411608    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-21c25193-7nfs7\" (UID: \"92b2241a-30ba-489b-bc0d-15cd4e44313d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"
Nov 03 01:30:30.411858 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.411662    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnmfd\" (UniqueName: \"kubernetes.io/projected/92b2241a-30ba-489b-bc0d-15cd4e44313d-kube-api-access-cnmfd\") pod \"maas-api-revoke-keys-e2e-ait-21c25193-7nfs7\" (UID: \"92b2241a-30ba-489b-bc0d-15cd4e44313d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"
Nov 03 01:30:30.411858 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.411744    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-21c25193-7nfs7\" (UID: \"92b2241a-30ba-489b-bc0d-15cd4e44313d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"
Nov 03 01:30:30.412388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.412341    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-21c25193-7nfs7\" (UID: \"92b2241a-30ba-489b-bc0d-15cd4e44313d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"
Nov 03 01:30:30.414087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.414058    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-21c25193-7nfs7\" (UID: \"92b2241a-30ba-489b-bc0d-15cd4e44313d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"
Nov 03 01:30:30.486129 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.486099    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnmfd\" (UniqueName: \"kubernetes.io/projected/92b2241a-30ba-489b-bc0d-15cd4e44313d-kube-api-access-cnmfd\") pod \"maas-api-revoke-keys-e2e-ait-21c25193-7nfs7\" (UID: \"92b2241a-30ba-489b-bc0d-15cd4e44313d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"
Nov 03 01:30:30.575867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.571525    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"
Nov 03 01:30:30.948550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:30.948522    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"]
Nov 03 01:30:30.955368 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:30:30.955320    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b2241a_30ba_489b_bc0d_15cd4e44313d.slice/crio-c2c67cfc0602420be959dd91a08b45dd43320aa76f0d87ca18f20f82caa98816 WatchSource:0}: Error finding container c2c67cfc0602420be959dd91a08b45dd43320aa76f0d87ca18f20f82caa98816: Status 404 returned error can't find the container with id c2c67cfc0602420be959dd91a08b45dd43320aa76f0d87ca18f20f82caa98816
Nov 03 01:30:31.277493 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:31.277466    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"]
Nov 03 01:30:31.280559 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:31.280535    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-98c7cd16-gvnzn"]
Nov 03 01:30:31.431856 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:31.431818    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7" event={"ID":"92b2241a-30ba-489b-bc0d-15cd4e44313d","Type":"ContainerStarted","Data":"e6508f97438696e96b2ff47d093b28d59a31c7faf1d0d904744f90d6e74b7af9"}
Nov 03 01:30:31.431992 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:31.431881    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7" event={"ID":"92b2241a-30ba-489b-bc0d-15cd4e44313d","Type":"ContainerStarted","Data":"c2c67cfc0602420be959dd91a08b45dd43320aa76f0d87ca18f20f82caa98816"}
Nov 03 01:30:31.542295 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:31.542209    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7" podUID="a1265bf3-e6a8-4550-b683-becfa74c40cb" containerName="payload-pre-processing" containerID="cri-o://ccecb3823b411ad2405677da38c36d823d81f71f8da20f2398d4021ce0ce5048" gracePeriod=25
Nov 03 01:30:31.542937 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:31.542899    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7" podStartSLOduration=1.542886329 podStartE2EDuration="1.542886329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:31.540659144 +0000 UTC m=+710.966265608" watchObservedRunningTime="2026-11-03 01:30:31.542886329 +0000 UTC m=+710.968492792"
Nov 03 01:30:31.809301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:31.809269    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"
Nov 03 01:30:31.832098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:31.831825    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1265bf3-e6a8-4550-b683-becfa74c40cb-kube-api-access-p86nm\" (UniqueName: \"kubernetes.io/projected/a1265bf3-e6a8-4550-b683-becfa74c40cb-kube-api-access-p86nm\") pod \"a1265bf3-e6a8-4550-b683-becfa74c40cb\" (UID: \"a1265bf3-e6a8-4550-b683-becfa74c40cb\") "
Nov 03 01:30:31.832098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:31.831922    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1265bf3-e6a8-4550-b683-becfa74c40cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1265bf3-e6a8-4550-b683-becfa74c40cb-plugins-config-volume\") pod \"a1265bf3-e6a8-4550-b683-becfa74c40cb\" (UID: \"a1265bf3-e6a8-4550-b683-becfa74c40cb\") "
Nov 03 01:30:31.832440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:31.832411    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1265bf3-e6a8-4550-b683-becfa74c40cb-plugins-config-volume" pod "a1265bf3-e6a8-4550-b683-becfa74c40cb" (UID: "a1265bf3-e6a8-4550-b683-becfa74c40cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:31.834110 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:31.834086    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1265bf3-e6a8-4550-b683-becfa74c40cb-kube-api-access-p86nm" pod "a1265bf3-e6a8-4550-b683-becfa74c40cb" (UID: "a1265bf3-e6a8-4550-b683-becfa74c40cb"). InnerVolumeSpecName "kube-api-access-p86nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:31.942847 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:31.942812    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p86nm\" (UniqueName: \"kubernetes.io/projected/a1265bf3-e6a8-4550-b683-becfa74c40cb-kube-api-access-p86nm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:31.942847 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:31.942846    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1265bf3-e6a8-4550-b683-becfa74c40cb-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:32.438539 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.438510    5508 generic.go:356] "Generic (PLEG): container finished" podID="a1265bf3-e6a8-4550-b683-becfa74c40cb" containerID="ccecb3823b411ad2405677da38c36d823d81f71f8da20f2398d4021ce0ce5048" exitCode=0
Nov 03 01:30:32.438960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.438556    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"
Nov 03 01:30:32.438960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.438579    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7" event={"ID":"a1265bf3-e6a8-4550-b683-becfa74c40cb","Type":"ContainerDied","Data":"ccecb3823b411ad2405677da38c36d823d81f71f8da20f2398d4021ce0ce5048"}
Nov 03 01:30:32.438960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.438643    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7" event={"ID":"a1265bf3-e6a8-4550-b683-becfa74c40cb","Type":"ContainerDied","Data":"c88709296a48dd0c7c3746906e52b5a3947e21ce26a91d1de7ab18edbadb7baf"}
Nov 03 01:30:32.438960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.438671    5508 scope.go:122] "RemoveContainer" containerID="ccecb3823b411ad2405677da38c36d823d81f71f8da20f2398d4021ce0ce5048"
Nov 03 01:30:32.440172 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.440148    5508 generic.go:356] "Generic (PLEG): container finished" podID="92b2241a-30ba-489b-bc0d-15cd4e44313d" containerID="e6508f97438696e96b2ff47d093b28d59a31c7faf1d0d904744f90d6e74b7af9" exitCode=0
Nov 03 01:30:32.440268 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.440187    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7" event={"ID":"92b2241a-30ba-489b-bc0d-15cd4e44313d","Type":"ContainerDied","Data":"e6508f97438696e96b2ff47d093b28d59a31c7faf1d0d904744f90d6e74b7af9"}
Nov 03 01:30:32.447420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.447403    5508 scope.go:122] "RemoveContainer" containerID="ccecb3823b411ad2405677da38c36d823d81f71f8da20f2398d4021ce0ce5048"
Nov 03 01:30:32.447707 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:32.447691    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccecb3823b411ad2405677da38c36d823d81f71f8da20f2398d4021ce0ce5048\": container with ID starting with ccecb3823b411ad2405677da38c36d823d81f71f8da20f2398d4021ce0ce5048 not found: ID does not exist" containerID="ccecb3823b411ad2405677da38c36d823d81f71f8da20f2398d4021ce0ce5048"
Nov 03 01:30:32.447769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.447714    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccecb3823b411ad2405677da38c36d823d81f71f8da20f2398d4021ce0ce5048"} err="failed to get container status \"ccecb3823b411ad2405677da38c36d823d81f71f8da20f2398d4021ce0ce5048\": rpc error: code = NotFound desc = could not find container \"ccecb3823b411ad2405677da38c36d823d81f71f8da20f2398d4021ce0ce5048\": container with ID starting with ccecb3823b411ad2405677da38c36d823d81f71f8da20f2398d4021ce0ce5048 not found: ID does not exist"
Nov 03 01:30:32.607373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.606412    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"]
Nov 03 01:30:32.612375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.609632    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e3f0a557-8476df7599-zfwx7"]
Nov 03 01:30:32.690444 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.690339    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b1640b-8375-46dc-aa36-038823af4c71" path="/var/lib/kubelet/pods/96b1640b-8375-46dc-aa36-038823af4c71/volumes"
Nov 03 01:30:32.690771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.690757    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1265bf3-e6a8-4550-b683-becfa74c40cb" path="/var/lib/kubelet/pods/a1265bf3-e6a8-4550-b683-becfa74c40cb/volumes"
Nov 03 01:30:32.747280 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.747244    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"]
Nov 03 01:30:32.751538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.751509    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"]
Nov 03 01:30:32.751863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.751827    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62" podUID="73e3f853-b154-4314-ba90-01a1caaee85a" containerName="istio-proxy" containerID="cri-o://22207b46516a2de38037aca1d3c492378839e56be5a95d4f43fce3ce07f446fe" gracePeriod=30
Nov 03 01:30:32.753472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:32.753427    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e3f0a557-x5d6f"]
Nov 03 01:30:33.593991 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:33.593966    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"
Nov 03 01:30:33.655825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:33.655791    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-openshift-ca\") pod \"92b2241a-30ba-489b-bc0d-15cd4e44313d\" (UID: \"92b2241a-30ba-489b-bc0d-15cd4e44313d\") "
Nov 03 01:30:33.656016 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:33.655890    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b2241a-30ba-489b-bc0d-15cd4e44313d-kube-api-access-cnmfd\" (UniqueName: \"kubernetes.io/projected/92b2241a-30ba-489b-bc0d-15cd4e44313d-kube-api-access-cnmfd\") pod \"92b2241a-30ba-489b-bc0d-15cd4e44313d\" (UID: \"92b2241a-30ba-489b-bc0d-15cd4e44313d\") "
Nov 03 01:30:33.656016 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:33.655945    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-xks-ca\") pod \"92b2241a-30ba-489b-bc0d-15cd4e44313d\" (UID: \"92b2241a-30ba-489b-bc0d-15cd4e44313d\") "
Nov 03 01:30:33.656182 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:33.656154    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-openshift-ca" pod "92b2241a-30ba-489b-bc0d-15cd4e44313d" (UID: "92b2241a-30ba-489b-bc0d-15cd4e44313d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:33.656320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:33.656302    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:33.658001 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:33.657972    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-xks-ca" pod "92b2241a-30ba-489b-bc0d-15cd4e44313d" (UID: "92b2241a-30ba-489b-bc0d-15cd4e44313d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:30:33.658175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:33.658156    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b2241a-30ba-489b-bc0d-15cd4e44313d-kube-api-access-cnmfd" pod "92b2241a-30ba-489b-bc0d-15cd4e44313d" (UID: "92b2241a-30ba-489b-bc0d-15cd4e44313d"). InnerVolumeSpecName "kube-api-access-cnmfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:33.757404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:33.757328    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnmfd\" (UniqueName: \"kubernetes.io/projected/92b2241a-30ba-489b-bc0d-15cd4e44313d-kube-api-access-cnmfd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:33.757404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:33.757372    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b2241a-30ba-489b-bc0d-15cd4e44313d-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:33.780744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:33.780709    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq" podUID="95532f38-0c60-4cd3-be34-27f692a72b50" containerName="payload-processing" containerID="cri-o://88c01349ebb9f24509f05c507168658823a041d19bfd054b90df26b4278d289f" gracePeriod=25
Nov 03 01:30:33.879922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:33.879878    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8" podUID="704ddecf-1477-4861-adaf-261c02fac634" containerName="payload-pre-processing" containerID="cri-o://11216f9c72a04e2444430cb80a86190e328dc2c00a4cc37cdde24efa23aec4df" gracePeriod=25
Nov 03 01:30:34.023620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.023596    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"
Nov 03 01:30:34.060937 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.060912    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95532f38-0c60-4cd3-be34-27f692a72b50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95532f38-0c60-4cd3-be34-27f692a72b50-plugins-config-volume\") pod \"95532f38-0c60-4cd3-be34-27f692a72b50\" (UID: \"95532f38-0c60-4cd3-be34-27f692a72b50\") "
Nov 03 01:30:34.061054 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.061041    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95532f38-0c60-4cd3-be34-27f692a72b50-kube-api-access-tq226\" (UniqueName: \"kubernetes.io/projected/95532f38-0c60-4cd3-be34-27f692a72b50-kube-api-access-tq226\") pod \"95532f38-0c60-4cd3-be34-27f692a72b50\" (UID: \"95532f38-0c60-4cd3-be34-27f692a72b50\") "
Nov 03 01:30:34.061251 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.061228    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95532f38-0c60-4cd3-be34-27f692a72b50-plugins-config-volume" pod "95532f38-0c60-4cd3-be34-27f692a72b50" (UID: "95532f38-0c60-4cd3-be34-27f692a72b50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:34.061365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.061338    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95532f38-0c60-4cd3-be34-27f692a72b50-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:34.063828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.063807    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95532f38-0c60-4cd3-be34-27f692a72b50-kube-api-access-tq226" pod "95532f38-0c60-4cd3-be34-27f692a72b50" (UID: "95532f38-0c60-4cd3-be34-27f692a72b50"). InnerVolumeSpecName "kube-api-access-tq226". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:34.154333 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.154305    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"
Nov 03 01:30:34.163394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.163337    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704ddecf-1477-4861-adaf-261c02fac634-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704ddecf-1477-4861-adaf-261c02fac634-plugins-config-volume\") pod \"704ddecf-1477-4861-adaf-261c02fac634\" (UID: \"704ddecf-1477-4861-adaf-261c02fac634\") "
Nov 03 01:30:34.163585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.163437    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704ddecf-1477-4861-adaf-261c02fac634-kube-api-access-8n4lr\" (UniqueName: \"kubernetes.io/projected/704ddecf-1477-4861-adaf-261c02fac634-kube-api-access-8n4lr\") pod \"704ddecf-1477-4861-adaf-261c02fac634\" (UID: \"704ddecf-1477-4861-adaf-261c02fac634\") "
Nov 03 01:30:34.163692 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.163667    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704ddecf-1477-4861-adaf-261c02fac634-plugins-config-volume" pod "704ddecf-1477-4861-adaf-261c02fac634" (UID: "704ddecf-1477-4861-adaf-261c02fac634"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:34.163692 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.163690    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq226\" (UniqueName: \"kubernetes.io/projected/95532f38-0c60-4cd3-be34-27f692a72b50-kube-api-access-tq226\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:34.165529 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.165501    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704ddecf-1477-4861-adaf-261c02fac634-kube-api-access-8n4lr" pod "704ddecf-1477-4861-adaf-261c02fac634" (UID: "704ddecf-1477-4861-adaf-261c02fac634"). InnerVolumeSpecName "kube-api-access-8n4lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:34.265654 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.264840    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/704ddecf-1477-4861-adaf-261c02fac634-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:34.265654 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.264980    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n4lr\" (UniqueName: \"kubernetes.io/projected/704ddecf-1477-4861-adaf-261c02fac634-kube-api-access-8n4lr\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:34.329448 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.329419    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"]
Nov 03 01:30:34.329659 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.329622    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerName="istio-proxy" containerID="cri-o://025df2bf378fcd03832463fc9e23e7f08e21410cecb47e3528ffd22ed6592138" gracePeriod=30
Nov 03 01:30:34.453395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.453364    5508 generic.go:356] "Generic (PLEG): container finished" podID="95532f38-0c60-4cd3-be34-27f692a72b50" containerID="88c01349ebb9f24509f05c507168658823a041d19bfd054b90df26b4278d289f" exitCode=0
Nov 03 01:30:34.453573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.453415    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"
Nov 03 01:30:34.453573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.453441    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq" event={"ID":"95532f38-0c60-4cd3-be34-27f692a72b50","Type":"ContainerDied","Data":"88c01349ebb9f24509f05c507168658823a041d19bfd054b90df26b4278d289f"}
Nov 03 01:30:34.453573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.453497    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq" event={"ID":"95532f38-0c60-4cd3-be34-27f692a72b50","Type":"ContainerDied","Data":"2d7da5c539733ab111e87dce7833cc9ab07070aaae66651b38663e24dd3d9057"}
Nov 03 01:30:34.453573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.453526    5508 scope.go:122] "RemoveContainer" containerID="88c01349ebb9f24509f05c507168658823a041d19bfd054b90df26b4278d289f"
Nov 03 01:30:34.454820 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.454802    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"
Nov 03 01:30:34.454954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.454848    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7" event={"ID":"92b2241a-30ba-489b-bc0d-15cd4e44313d","Type":"ContainerDied","Data":"c2c67cfc0602420be959dd91a08b45dd43320aa76f0d87ca18f20f82caa98816"}
Nov 03 01:30:34.454954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.454894    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c67cfc0602420be959dd91a08b45dd43320aa76f0d87ca18f20f82caa98816"
Nov 03 01:30:34.456514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.456486    5508 generic.go:356] "Generic (PLEG): container finished" podID="704ddecf-1477-4861-adaf-261c02fac634" containerID="11216f9c72a04e2444430cb80a86190e328dc2c00a4cc37cdde24efa23aec4df" exitCode=0
Nov 03 01:30:34.456514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.456522    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"
Nov 03 01:30:34.456686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.456535    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8" event={"ID":"704ddecf-1477-4861-adaf-261c02fac634","Type":"ContainerDied","Data":"11216f9c72a04e2444430cb80a86190e328dc2c00a4cc37cdde24efa23aec4df"}
Nov 03 01:30:34.456686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.456580    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8" event={"ID":"704ddecf-1477-4861-adaf-261c02fac634","Type":"ContainerDied","Data":"e01a2c4aeb527dd946140e7bbb13ffcd0fbbec2e1ff6f463d70712c44ba7b806"}
Nov 03 01:30:34.464741 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.464722    5508 scope.go:122] "RemoveContainer" containerID="88c01349ebb9f24509f05c507168658823a041d19bfd054b90df26b4278d289f"
Nov 03 01:30:34.464969 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:34.464953    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c01349ebb9f24509f05c507168658823a041d19bfd054b90df26b4278d289f\": container with ID starting with 88c01349ebb9f24509f05c507168658823a041d19bfd054b90df26b4278d289f not found: ID does not exist" containerID="88c01349ebb9f24509f05c507168658823a041d19bfd054b90df26b4278d289f"
Nov 03 01:30:34.465023 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.464977    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c01349ebb9f24509f05c507168658823a041d19bfd054b90df26b4278d289f"} err="failed to get container status \"88c01349ebb9f24509f05c507168658823a041d19bfd054b90df26b4278d289f\": rpc error: code = NotFound desc = could not find container \"88c01349ebb9f24509f05c507168658823a041d19bfd054b90df26b4278d289f\": container with ID starting with 88c01349ebb9f24509f05c507168658823a041d19bfd054b90df26b4278d289f not found: ID does not exist"
Nov 03 01:30:34.465023 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.464991    5508 scope.go:122] "RemoveContainer" containerID="11216f9c72a04e2444430cb80a86190e328dc2c00a4cc37cdde24efa23aec4df"
Nov 03 01:30:34.473823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.473802    5508 scope.go:122] "RemoveContainer" containerID="11216f9c72a04e2444430cb80a86190e328dc2c00a4cc37cdde24efa23aec4df"
Nov 03 01:30:34.474085 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:34.474063    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11216f9c72a04e2444430cb80a86190e328dc2c00a4cc37cdde24efa23aec4df\": container with ID starting with 11216f9c72a04e2444430cb80a86190e328dc2c00a4cc37cdde24efa23aec4df not found: ID does not exist" containerID="11216f9c72a04e2444430cb80a86190e328dc2c00a4cc37cdde24efa23aec4df"
Nov 03 01:30:34.474137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.474094    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11216f9c72a04e2444430cb80a86190e328dc2c00a4cc37cdde24efa23aec4df"} err="failed to get container status \"11216f9c72a04e2444430cb80a86190e328dc2c00a4cc37cdde24efa23aec4df\": rpc error: code = NotFound desc = could not find container \"11216f9c72a04e2444430cb80a86190e328dc2c00a4cc37cdde24efa23aec4df\": container with ID starting with 11216f9c72a04e2444430cb80a86190e328dc2c00a4cc37cdde24efa23aec4df not found: ID does not exist"
Nov 03 01:30:34.482618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.482597    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"]
Nov 03 01:30:34.482957 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.482944    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a1265bf3-e6a8-4550-b683-becfa74c40cb" containerName="payload-pre-processing"
Nov 03 01:30:34.483015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.482958    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1265bf3-e6a8-4550-b683-becfa74c40cb" containerName="payload-pre-processing"
Nov 03 01:30:34.483015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.482981    5508 cpu_manager.go:414] "containerMap: removing container" podUID="1b610877-52f4-4d89-b7ba-b65d4b7e4312" containerName="cleanup"
Nov 03 01:30:34.483015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.482986    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b610877-52f4-4d89-b7ba-b65d4b7e4312" containerName="cleanup"
Nov 03 01:30:34.483015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.482996    5508 cpu_manager.go:414] "containerMap: removing container" podUID="704ddecf-1477-4861-adaf-261c02fac634" containerName="payload-pre-processing"
Nov 03 01:30:34.483015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483001    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704ddecf-1477-4861-adaf-261c02fac634" containerName="payload-pre-processing"
Nov 03 01:30:34.483015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483011    5508 cpu_manager.go:414] "containerMap: removing container" podUID="95532f38-0c60-4cd3-be34-27f692a72b50" containerName="payload-processing"
Nov 03 01:30:34.483015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483016    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95532f38-0c60-4cd3-be34-27f692a72b50" containerName="payload-processing"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483023    5508 cpu_manager.go:414] "containerMap: removing container" podUID="969c5a71-7c35-466c-8b1d-2a864343c062" containerName="cleanup"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483028    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969c5a71-7c35-466c-8b1d-2a864343c062" containerName="cleanup"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483034    5508 cpu_manager.go:414] "containerMap: removing container" podUID="79520c6d-ffee-41d0-a8f6-0801d98196ba" containerName="cleanup"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483038    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79520c6d-ffee-41d0-a8f6-0801d98196ba" containerName="cleanup"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483047    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4f03fce0-0280-4d45-82ae-59c40bbad92f" containerName="cleanup"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483050    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f03fce0-0280-4d45-82ae-59c40bbad92f" containerName="cleanup"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483057    5508 cpu_manager.go:414] "containerMap: removing container" podUID="92b2241a-30ba-489b-bc0d-15cd4e44313d" containerName="revoke-keys"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483062    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b2241a-30ba-489b-bc0d-15cd4e44313d" containerName="revoke-keys"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483117    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1265bf3-e6a8-4550-b683-becfa74c40cb" containerName="payload-pre-processing"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483123    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f03fce0-0280-4d45-82ae-59c40bbad92f" containerName="cleanup"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483133    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="704ddecf-1477-4861-adaf-261c02fac634" containerName="payload-pre-processing"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483139    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="95532f38-0c60-4cd3-be34-27f692a72b50" containerName="payload-processing"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483144    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b2241a-30ba-489b-bc0d-15cd4e44313d" containerName="revoke-keys"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483149    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="79520c6d-ffee-41d0-a8f6-0801d98196ba" containerName="cleanup"
Nov 03 01:30:34.483247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.483154    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b610877-52f4-4d89-b7ba-b65d4b7e4312" containerName="cleanup"
Nov 03 01:30:34.487664 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.487649    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.536264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.536200    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-41aa5f-openshift-default-dockercfg-g26tf\""
Nov 03 01:30:34.543600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.543573    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"]
Nov 03 01:30:34.566504 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.566479    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.566621 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.566510    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.566621 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.566527    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da42a7e-c55b-451e-b887-ea9d8461f23b-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.566621 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.566543    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.566755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.566712    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.566755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.566744    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.566822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.566798    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.566856 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.566849    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mn46\" (UniqueName: \"kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-kube-api-access-2mn46\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.566909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.566877    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.592924 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.592894    5508 patch_prober.go:29] interesting pod/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:30:34.593035 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.592940    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:30:34.606757 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.606728    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"]
Nov 03 01:30:34.610647 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.610624    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc87c747-78d4fb7f97-4q7rq"]
Nov 03 01:30:34.667966 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.667933    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mn46\" (UniqueName: \"kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-kube-api-access-2mn46\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.668142 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.667982    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.668142 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.668036    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.668142 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.668057    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.668142 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.668072    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da42a7e-c55b-451e-b887-ea9d8461f23b-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.668142 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.668095    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.668365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.668161    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.668365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.668186    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.668365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.668201    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.668560 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.668539    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.668639 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.668617    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.668847 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.668829    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.668891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.668861    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da42a7e-c55b-451e-b887-ea9d8461f23b-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.668948 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.668927    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.670954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.670926    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.671112 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.671095    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.690999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.690977    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4980a1-1514-4f0c-b370-bf7222ab84a3" path="/var/lib/kubelet/pods/4e4980a1-1514-4f0c-b370-bf7222ab84a3/volumes"
Nov 03 01:30:34.691317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.691305    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95532f38-0c60-4cd3-be34-27f692a72b50" path="/var/lib/kubelet/pods/95532f38-0c60-4cd3-be34-27f692a72b50/volumes"
Nov 03 01:30:34.726154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.726133    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.726243 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.726208    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mn46\" (UniqueName: \"kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-kube-api-access-2mn46\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.799515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.799433    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:34.874610 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.873730    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"]
Nov 03 01:30:34.877579 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:34.877536    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc87c747-98b57c799-9cdj8"]
Nov 03 01:30:35.246909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.246880    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"]
Nov 03 01:30:35.254882 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:30:35.254858    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da42a7e_c55b_451e_b887_ea9d8461f23b.slice/crio-798c1af3b8a070362122af39cea15db70ce5cd83cca7f4d09f5440dbf9e9a153 WatchSource:0}: Error finding container 798c1af3b8a070362122af39cea15db70ce5cd83cca7f4d09f5440dbf9e9a153: Status 404 returned error can't find the container with id 798c1af3b8a070362122af39cea15db70ce5cd83cca7f4d09f5440dbf9e9a153
Nov 03 01:30:35.257443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.257312    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:30:35.257443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.257422    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:30:35.257562 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.257465    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:30:35.467938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.467895    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99" event={"ID":"5da42a7e-c55b-451e-b887-ea9d8461f23b","Type":"ContainerStarted","Data":"b948d0ea76b602a2a348cb871d767262befb20cf54791f26704861f3560c8fd0"}
Nov 03 01:30:35.468105 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.467946    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99" event={"ID":"5da42a7e-c55b-451e-b887-ea9d8461f23b","Type":"ContainerStarted","Data":"798c1af3b8a070362122af39cea15db70ce5cd83cca7f4d09f5440dbf9e9a153"}
Nov 03 01:30:35.583563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.583466    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99" podStartSLOduration=1.5834472929999999 podStartE2EDuration="1.583447293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:35.58104872 +0000 UTC m=+715.006655209" watchObservedRunningTime="2026-11-03 01:30:35.583447293 +0000 UTC m=+715.009053759"
Nov 03 01:30:35.800228 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.800195    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:35.930883 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.930854    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:30:35.970842 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.970815    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"]
Nov 03 01:30:35.974497 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.974480    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:35.988378 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.988338    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:35.988514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.988418    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:35.988514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.988479    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:35.988633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.988552    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:35.988689 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.988629    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:35.988689 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.988661    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:35.988798 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.988691    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:35.989631 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.988962    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfkzm\" (UniqueName: \"kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-kube-api-access-tfkzm\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:35.989631 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:35.989160    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.023575 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.023549    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b6fb9b-openshift-default-dockercfg-997hh\""
Nov 03 01:30:36.032658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.032633    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"]
Nov 03 01:30:36.090174 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090147    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.090277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090192    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.090277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090225    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.090277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090243    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.090277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090257    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.090277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090271    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfkzm\" (UniqueName: \"kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-kube-api-access-tfkzm\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.090542 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090375    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.090542 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090478    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.090542 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090509    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.090695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090580    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.090695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090621    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.090836 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090814    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.090891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090848    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.090982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.090951    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.092514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.092491    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.092854 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.092837    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.150794 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.150763    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.152052 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.152031    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfkzm\" (UniqueName: \"kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-kube-api-access-tfkzm\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.296857 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.296781    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:36.690456 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.690421    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704ddecf-1477-4861-adaf-261c02fac634" path="/var/lib/kubelet/pods/704ddecf-1477-4861-adaf-261c02fac634/volumes"
Nov 03 01:30:36.721116 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.721087    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"]
Nov 03 01:30:36.727727 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:30:36.727701    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e99b241_cf23_4d04_9087_87e5b89aaf9b.slice/crio-bf6c1c895ef16429a876d7c4efcca3275784aa1555cfb579fffb9ef500441473 WatchSource:0}: Error finding container bf6c1c895ef16429a876d7c4efcca3275784aa1555cfb579fffb9ef500441473: Status 404 returned error can't find the container with id bf6c1c895ef16429a876d7c4efcca3275784aa1555cfb579fffb9ef500441473
Nov 03 01:30:36.730147 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.730110    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:30:36.730225 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.730196    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:30:36.730265 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.730243    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:30:36.804491 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:36.804468    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:37.480515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:37.480477    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl" event={"ID":"8e99b241-cf23-4d04-9087-87e5b89aaf9b","Type":"ContainerStarted","Data":"8a5d0b4639a3103d1d71a767196b77d62f4c9de25693cafbf543dd0a1a8414cd"}
Nov 03 01:30:37.480724 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:37.480541    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl" event={"ID":"8e99b241-cf23-4d04-9087-87e5b89aaf9b","Type":"ContainerStarted","Data":"bf6c1c895ef16429a876d7c4efcca3275784aa1555cfb579fffb9ef500441473"}
Nov 03 01:30:37.480789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:37.480779    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:37.481931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:37.481910    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:30:37.594187 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:37.594084    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl" podStartSLOduration=2.594065528 podStartE2EDuration="2.594065528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:37.593132918 +0000 UTC m=+717.018739382" watchObservedRunningTime="2026-11-03 01:30:37.594065528 +0000 UTC m=+717.019671994"
Nov 03 01:30:38.016145 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.016118    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:30:38.115804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.115723    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-credential-socket\") pod \"73e3f853-b154-4314-ba90-01a1caaee85a\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") "
Nov 03 01:30:38.115804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.115778    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-certs\") pod \"73e3f853-b154-4314-ba90-01a1caaee85a\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") "
Nov 03 01:30:38.116034 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.115813    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-kube-api-access-hsjmc\" (UniqueName: \"kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-kube-api-access-hsjmc\") pod \"73e3f853-b154-4314-ba90-01a1caaee85a\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") "
Nov 03 01:30:38.116034 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.115840    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-envoy\") pod \"73e3f853-b154-4314-ba90-01a1caaee85a\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") "
Nov 03 01:30:38.116034 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.115856    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e3f853-b154-4314-ba90-01a1caaee85a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e3f853-b154-4314-ba90-01a1caaee85a-istiod-ca-cert\") pod \"73e3f853-b154-4314-ba90-01a1caaee85a\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") "
Nov 03 01:30:38.116034 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.115885    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-socket\") pod \"73e3f853-b154-4314-ba90-01a1caaee85a\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") "
Nov 03 01:30:38.116034 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.115939    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-istio-token\" (UniqueName: \"kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-istio-token\") pod \"73e3f853-b154-4314-ba90-01a1caaee85a\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") "
Nov 03 01:30:38.116034 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.115971    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-data\") pod \"73e3f853-b154-4314-ba90-01a1caaee85a\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") "
Nov 03 01:30:38.116276 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.116035    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-credential-socket" pod "73e3f853-b154-4314-ba90-01a1caaee85a" (UID: "73e3f853-b154-4314-ba90-01a1caaee85a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:38.116276 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.116057    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73e3f853-b154-4314-ba90-01a1caaee85a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e3f853-b154-4314-ba90-01a1caaee85a-istio-podinfo\") pod \"73e3f853-b154-4314-ba90-01a1caaee85a\" (UID: \"73e3f853-b154-4314-ba90-01a1caaee85a\") "
Nov 03 01:30:38.116276 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.116143    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-certs" pod "73e3f853-b154-4314-ba90-01a1caaee85a" (UID: "73e3f853-b154-4314-ba90-01a1caaee85a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:38.116469 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.116298    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e3f853-b154-4314-ba90-01a1caaee85a-istiod-ca-cert" pod "73e3f853-b154-4314-ba90-01a1caaee85a" (UID: "73e3f853-b154-4314-ba90-01a1caaee85a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:38.116469 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.116442    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-socket" pod "73e3f853-b154-4314-ba90-01a1caaee85a" (UID: "73e3f853-b154-4314-ba90-01a1caaee85a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:38.116571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.116554    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-data" pod "73e3f853-b154-4314-ba90-01a1caaee85a" (UID: "73e3f853-b154-4314-ba90-01a1caaee85a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:38.116697 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.116683    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:38.116758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.116699    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e3f853-b154-4314-ba90-01a1caaee85a-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:38.116758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.116708    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:38.116758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.116720    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:38.116758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.116732    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:38.118295 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.118270    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73e3f853-b154-4314-ba90-01a1caaee85a-istio-podinfo" pod "73e3f853-b154-4314-ba90-01a1caaee85a" (UID: "73e3f853-b154-4314-ba90-01a1caaee85a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:30:38.118295 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.118288    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-kube-api-access-hsjmc" pod "73e3f853-b154-4314-ba90-01a1caaee85a" (UID: "73e3f853-b154-4314-ba90-01a1caaee85a"). InnerVolumeSpecName "kube-api-access-hsjmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:38.118575 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.118559    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-istio-token" pod "73e3f853-b154-4314-ba90-01a1caaee85a" (UID: "73e3f853-b154-4314-ba90-01a1caaee85a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:38.118668 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.118567    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-envoy" pod "73e3f853-b154-4314-ba90-01a1caaee85a" (UID: "73e3f853-b154-4314-ba90-01a1caaee85a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:38.217956 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.217921    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:38.217956 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.217951    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e3f853-b154-4314-ba90-01a1caaee85a-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:38.217956 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.217960    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsjmc\" (UniqueName: \"kubernetes.io/projected/73e3f853-b154-4314-ba90-01a1caaee85a-kube-api-access-hsjmc\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:38.218192 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.217970    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e3f853-b154-4314-ba90-01a1caaee85a-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:38.297814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.297780    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:38.302719 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.302699    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:38.487646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.487615    5508 generic.go:356] "Generic (PLEG): container finished" podID="73e3f853-b154-4314-ba90-01a1caaee85a" containerID="22207b46516a2de38037aca1d3c492378839e56be5a95d4f43fce3ce07f446fe" exitCode=0
Nov 03 01:30:38.487816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.487662    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"
Nov 03 01:30:38.487816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.487701    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62" event={"ID":"73e3f853-b154-4314-ba90-01a1caaee85a","Type":"ContainerDied","Data":"22207b46516a2de38037aca1d3c492378839e56be5a95d4f43fce3ce07f446fe"}
Nov 03 01:30:38.487816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.487761    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62" event={"ID":"73e3f853-b154-4314-ba90-01a1caaee85a","Type":"ContainerDied","Data":"07e080ef4c49a1b5fead94ab2ccf0d9a380e3160149d6cee71163c22c7171a31"}
Nov 03 01:30:38.487816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.487790    5508 scope.go:122] "RemoveContainer" containerID="22207b46516a2de38037aca1d3c492378839e56be5a95d4f43fce3ce07f446fe"
Nov 03 01:30:38.488645 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.488314    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:38.489042 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.489024    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:30:38.497101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.497088    5508 scope.go:122] "RemoveContainer" containerID="d3188b5b4e7f3fa553c9da59c087cfa94710bfae4bf96cf710d366b5ad42977e"
Nov 03 01:30:38.505308 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.505287    5508 scope.go:122] "RemoveContainer" containerID="22207b46516a2de38037aca1d3c492378839e56be5a95d4f43fce3ce07f446fe"
Nov 03 01:30:38.505558 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:38.505542    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22207b46516a2de38037aca1d3c492378839e56be5a95d4f43fce3ce07f446fe\": container with ID starting with 22207b46516a2de38037aca1d3c492378839e56be5a95d4f43fce3ce07f446fe not found: ID does not exist" containerID="22207b46516a2de38037aca1d3c492378839e56be5a95d4f43fce3ce07f446fe"
Nov 03 01:30:38.505600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.505567    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22207b46516a2de38037aca1d3c492378839e56be5a95d4f43fce3ce07f446fe"} err="failed to get container status \"22207b46516a2de38037aca1d3c492378839e56be5a95d4f43fce3ce07f446fe\": rpc error: code = NotFound desc = could not find container \"22207b46516a2de38037aca1d3c492378839e56be5a95d4f43fce3ce07f446fe\": container with ID starting with 22207b46516a2de38037aca1d3c492378839e56be5a95d4f43fce3ce07f446fe not found: ID does not exist"
Nov 03 01:30:38.505600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.505584    5508 scope.go:122] "RemoveContainer" containerID="d3188b5b4e7f3fa553c9da59c087cfa94710bfae4bf96cf710d366b5ad42977e"
Nov 03 01:30:38.505780 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:38.505761    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3188b5b4e7f3fa553c9da59c087cfa94710bfae4bf96cf710d366b5ad42977e\": container with ID starting with d3188b5b4e7f3fa553c9da59c087cfa94710bfae4bf96cf710d366b5ad42977e not found: ID does not exist" containerID="d3188b5b4e7f3fa553c9da59c087cfa94710bfae4bf96cf710d366b5ad42977e"
Nov 03 01:30:38.505821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.505786    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3188b5b4e7f3fa553c9da59c087cfa94710bfae4bf96cf710d366b5ad42977e"} err="failed to get container status \"d3188b5b4e7f3fa553c9da59c087cfa94710bfae4bf96cf710d366b5ad42977e\": rpc error: code = NotFound desc = could not find container \"d3188b5b4e7f3fa553c9da59c087cfa94710bfae4bf96cf710d366b5ad42977e\": container with ID starting with d3188b5b4e7f3fa553c9da59c087cfa94710bfae4bf96cf710d366b5ad42977e not found: ID does not exist"
Nov 03 01:30:38.765112 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.765039    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"]
Nov 03 01:30:38.772131 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:38.772103    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-98c7cd16-openshift-default-575c7769f9-6gf62"]
Nov 03 01:30:39.496148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.496114    5508 generic.go:356] "Generic (PLEG): container finished" podID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerID="025df2bf378fcd03832463fc9e23e7f08e21410cecb47e3528ffd22ed6592138" exitCode=0
Nov 03 01:30:39.496620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.496188    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" event={"ID":"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd","Type":"ContainerDied","Data":"025df2bf378fcd03832463fc9e23e7f08e21410cecb47e3528ffd22ed6592138"}
Nov 03 01:30:39.496620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.496268    5508 scope.go:122] "RemoveContainer" containerID="c47253ff836e6403d40f90c34c07c67bc37ef3dbca02b8f72081a3e130913516"
Nov 03 01:30:39.584722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.584689    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:30:39.632629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.632542    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-data\") pod \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") "
Nov 03 01:30:39.632629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.632585    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-podinfo\") pod \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") "
Nov 03 01:30:39.632629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.632615    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-certs\") pod \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") "
Nov 03 01:30:39.632912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.632643    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-socket\") pod \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") "
Nov 03 01:30:39.632912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.632691    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-envoy\") pod \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") "
Nov 03 01:30:39.632912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.632720    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-token\" (UniqueName: \"kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-token\") pod \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") "
Nov 03 01:30:39.632912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.632758    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istiod-ca-cert\") pod \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") "
Nov 03 01:30:39.632912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.632789    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-credential-socket\") pod \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") "
Nov 03 01:30:39.632912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.632852    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-kube-api-access-kf782\" (UniqueName: \"kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-kube-api-access-kf782\") pod \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\" (UID: \"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd\") "
Nov 03 01:30:39.632912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.632858    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-data" pod "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" (UID: "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:39.633259 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.633122    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-certs" pod "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" (UID: "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:39.633259 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.633149    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-socket" pod "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" (UID: "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:39.633259 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.633197    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-credential-socket" pod "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" (UID: "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:39.633447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.633267    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istiod-ca-cert" pod "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" (UID: "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:39.633447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.633270    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.633447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.633313    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.633447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.633334    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.635031 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.635006    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-envoy" pod "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" (UID: "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:30:39.635031 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.635014    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-token" pod "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" (UID: "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:39.635169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.635054    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-kube-api-access-kf782" pod "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" (UID: "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd"). InnerVolumeSpecName "kube-api-access-kf782". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:39.635169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.635125    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-podinfo" pod "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" (UID: "7c3133b2-0ba6-4231-bd4a-fce0bdd960cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:30:39.734256 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.734225    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf782\" (UniqueName: \"kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-kube-api-access-kf782\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.734256 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.734252    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.734256 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.734263    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.734492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.734272    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.734492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.734280    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:39.734492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:39.734290    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:40.511144 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:40.511115    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"
Nov 03 01:30:40.511652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:40.511144    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r" event={"ID":"7c3133b2-0ba6-4231-bd4a-fce0bdd960cd","Type":"ContainerDied","Data":"d61e9af9eb533b4b50b6ea77a554e44e42fbbe64c48d9f8a058c76dd8fb8adf0"}
Nov 03 01:30:40.511652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:40.511207    5508 scope.go:122] "RemoveContainer" containerID="025df2bf378fcd03832463fc9e23e7f08e21410cecb47e3528ffd22ed6592138"
Nov 03 01:30:40.690204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:40.690177    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e3f853-b154-4314-ba90-01a1caaee85a" path="/var/lib/kubelet/pods/73e3f853-b154-4314-ba90-01a1caaee85a/volumes"
Nov 03 01:30:40.710827 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:40.710806    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"]
Nov 03 01:30:40.757755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:40.757734    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fc87c747-openshift-default-56d449669f-dsh4r"]
Nov 03 01:30:41.967783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:41.967753    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"]
Nov 03 01:30:41.968154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:41.968115    5508 cpu_manager.go:414] "containerMap: removing container" podUID="73e3f853-b154-4314-ba90-01a1caaee85a" containerName="istio-proxy"
Nov 03 01:30:41.968154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:41.968125    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e3f853-b154-4314-ba90-01a1caaee85a" containerName="istio-proxy"
Nov 03 01:30:41.968154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:41.968135    5508 cpu_manager.go:414] "containerMap: removing container" podUID="73e3f853-b154-4314-ba90-01a1caaee85a" containerName="istio-proxy"
Nov 03 01:30:41.968154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:41.968141    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e3f853-b154-4314-ba90-01a1caaee85a" containerName="istio-proxy"
Nov 03 01:30:41.968281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:41.968159    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerName="istio-proxy"
Nov 03 01:30:41.968281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:41.968164    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerName="istio-proxy"
Nov 03 01:30:41.968281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:41.968181    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerName="istio-proxy"
Nov 03 01:30:41.968281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:41.968186    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerName="istio-proxy"
Nov 03 01:30:41.968281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:41.968237    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerName="istio-proxy"
Nov 03 01:30:41.968281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:41.968247    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e3f853-b154-4314-ba90-01a1caaee85a" containerName="istio-proxy"
Nov 03 01:30:41.968281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:41.968253    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e3f853-b154-4314-ba90-01a1caaee85a" containerName="istio-proxy"
Nov 03 01:30:41.972891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:41.972872    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"
Nov 03 01:30:42.023303 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.023262    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-41aa5f\""
Nov 03 01:30:42.060471 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.060420    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngq4l\" (UniqueName: \"kubernetes.io/projected/3b2deab8-243a-4592-a30c-c521fe20db90-kube-api-access-ngq4l\") pod \"payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4\" (UID: \"3b2deab8-243a-4592-a30c-c521fe20db90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"
Nov 03 01:30:42.060889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.060867    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2deab8-243a-4592-a30c-c521fe20db90-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4\" (UID: \"3b2deab8-243a-4592-a30c-c521fe20db90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"
Nov 03 01:30:42.085580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.085540    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-41aa5f-dockercfg-5bpzg\""
Nov 03 01:30:42.090444 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.090413    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"]
Nov 03 01:30:42.132845 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.132818    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"]
Nov 03 01:30:42.133272 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.133260    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" containerName="istio-proxy"
Nov 03 01:30:42.136388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.136372    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"
Nov 03 01:30:42.161751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.161732    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xmtg\" (UniqueName: \"kubernetes.io/projected/3588e919-ccba-4fe9-8e34-30efb454c3b2-kube-api-access-9xmtg\") pod \"payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7\" (UID: \"3588e919-ccba-4fe9-8e34-30efb454c3b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"
Nov 03 01:30:42.161864 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.161765    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3588e919-ccba-4fe9-8e34-30efb454c3b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7\" (UID: \"3588e919-ccba-4fe9-8e34-30efb454c3b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"
Nov 03 01:30:42.161864 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.161790    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngq4l\" (UniqueName: \"kubernetes.io/projected/3b2deab8-243a-4592-a30c-c521fe20db90-kube-api-access-ngq4l\") pod \"payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4\" (UID: \"3b2deab8-243a-4592-a30c-c521fe20db90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"
Nov 03 01:30:42.162006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.161985    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2deab8-243a-4592-a30c-c521fe20db90-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4\" (UID: \"3b2deab8-243a-4592-a30c-c521fe20db90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"
Nov 03 01:30:42.162596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.162579    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2deab8-243a-4592-a30c-c521fe20db90-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4\" (UID: \"3b2deab8-243a-4592-a30c-c521fe20db90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"
Nov 03 01:30:42.184107 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.184085    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"]
Nov 03 01:30:42.218888 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.218826    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngq4l\" (UniqueName: \"kubernetes.io/projected/3b2deab8-243a-4592-a30c-c521fe20db90-kube-api-access-ngq4l\") pod \"payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4\" (UID: \"3b2deab8-243a-4592-a30c-c521fe20db90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"
Nov 03 01:30:42.262959 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.262928    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xmtg\" (UniqueName: \"kubernetes.io/projected/3588e919-ccba-4fe9-8e34-30efb454c3b2-kube-api-access-9xmtg\") pod \"payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7\" (UID: \"3588e919-ccba-4fe9-8e34-30efb454c3b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"
Nov 03 01:30:42.263082 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.262975    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3588e919-ccba-4fe9-8e34-30efb454c3b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7\" (UID: \"3588e919-ccba-4fe9-8e34-30efb454c3b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"
Nov 03 01:30:42.263683 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.263667    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3588e919-ccba-4fe9-8e34-30efb454c3b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7\" (UID: \"3588e919-ccba-4fe9-8e34-30efb454c3b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"
Nov 03 01:30:42.285817 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.285794    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"
Nov 03 01:30:42.320605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.320584    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xmtg\" (UniqueName: \"kubernetes.io/projected/3588e919-ccba-4fe9-8e34-30efb454c3b2-kube-api-access-9xmtg\") pod \"payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7\" (UID: \"3588e919-ccba-4fe9-8e34-30efb454c3b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"
Nov 03 01:30:42.448890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.448862    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"
Nov 03 01:30:42.654157 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.654120    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"]
Nov 03 01:30:42.661401 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:30:42.661372    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2deab8_243a_4592_a30c_c521fe20db90.slice/crio-24bfdce41df33b857fc7436fdcbd3e49da786ad7eb7c64f94a2b88e6c3a8f74c WatchSource:0}: Error finding container 24bfdce41df33b857fc7436fdcbd3e49da786ad7eb7c64f94a2b88e6c3a8f74c: Status 404 returned error can't find the container with id 24bfdce41df33b857fc7436fdcbd3e49da786ad7eb7c64f94a2b88e6c3a8f74c
Nov 03 01:30:42.690856 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.690833    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c3133b2-0ba6-4231-bd4a-fce0bdd960cd" path="/var/lib/kubelet/pods/7c3133b2-0ba6-4231-bd4a-fce0bdd960cd/volumes"
Nov 03 01:30:42.730246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.730223    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"]
Nov 03 01:30:42.819444 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:42.819376    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"]
Nov 03 01:30:42.826822 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:30:42.826795    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3588e919_ccba_4fe9_8e34_30efb454c3b2.slice/crio-ce36d5f3f8913cb7fadc48d9248dc39a4dc376b365f75eb1fd9b803a4104b63a WatchSource:0}: Error finding container ce36d5f3f8913cb7fadc48d9248dc39a4dc376b365f75eb1fd9b803a4104b63a: Status 404 returned error can't find the container with id ce36d5f3f8913cb7fadc48d9248dc39a4dc376b365f75eb1fd9b803a4104b63a
Nov 03 01:30:43.528925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:43.528888    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4" event={"ID":"3b2deab8-243a-4592-a30c-c521fe20db90","Type":"ContainerStarted","Data":"0871fe04083f43fa50625bb8921e0bf570683638867d4b1bb6fc95bc8fd1db04"}
Nov 03 01:30:43.529418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:43.528949    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4" event={"ID":"3b2deab8-243a-4592-a30c-c521fe20db90","Type":"ContainerStarted","Data":"24bfdce41df33b857fc7436fdcbd3e49da786ad7eb7c64f94a2b88e6c3a8f74c"}
Nov 03 01:30:43.529418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:43.528993    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"
Nov 03 01:30:43.530525 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:43.530500    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7" event={"ID":"3588e919-ccba-4fe9-8e34-30efb454c3b2","Type":"ContainerStarted","Data":"1d689822fb8f00ace1a046f1759b407c2c7f9b9935b17a50d739d43868d6617c"}
Nov 03 01:30:43.530630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:43.530550    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"
Nov 03 01:30:43.530630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:43.530568    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7" event={"ID":"3588e919-ccba-4fe9-8e34-30efb454c3b2","Type":"ContainerStarted","Data":"ce36d5f3f8913cb7fadc48d9248dc39a4dc376b365f75eb1fd9b803a4104b63a"}
Nov 03 01:30:43.640998 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:43.640945    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4" podStartSLOduration=2.640930062 podStartE2EDuration="2.640930062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:43.638941823 +0000 UTC m=+723.064548306" watchObservedRunningTime="2026-11-03 01:30:43.640930062 +0000 UTC m=+723.066536525"
Nov 03 01:30:43.761744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:43.761661    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7" podStartSLOduration=1.761641165 podStartE2EDuration="1.761641165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:43.746425856 +0000 UTC m=+723.172032321" watchObservedRunningTime="2026-11-03 01:30:43.761641165 +0000 UTC m=+723.187247640"
Nov 03 01:30:43.814815 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:43.814735    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:30:47.730912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:47.730795    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv" podUID="d2a9ccb3-2e3d-4599-a491-8167a2bd1e40" containerName="payload-pre-processing" containerID="cri-o://feff271e020e1e85dbc8115de4cd40e294b45a9cd03f729665fa36a1f91c7402" gracePeriod=25
Nov 03 01:30:47.983156 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:47.983097    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"
Nov 03 01:30:48.023332 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.023304    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-kube-api-access-6t6g7\" (UniqueName: \"kubernetes.io/projected/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-kube-api-access-6t6g7\") pod \"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40\" (UID: \"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40\") "
Nov 03 01:30:48.023507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.023386    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-plugins-config-volume\") pod \"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40\" (UID: \"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40\") "
Nov 03 01:30:48.023690 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.023672    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-plugins-config-volume" pod "d2a9ccb3-2e3d-4599-a491-8167a2bd1e40" (UID: "d2a9ccb3-2e3d-4599-a491-8167a2bd1e40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:48.023830 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.023816    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:48.025277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.025254    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-kube-api-access-6t6g7" pod "d2a9ccb3-2e3d-4599-a491-8167a2bd1e40" (UID: "d2a9ccb3-2e3d-4599-a491-8167a2bd1e40"). InnerVolumeSpecName "kube-api-access-6t6g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:48.124895 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.124864    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t6g7\" (UniqueName: \"kubernetes.io/projected/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40-kube-api-access-6t6g7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:30:48.558555 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.558517    5508 generic.go:356] "Generic (PLEG): container finished" podID="d2a9ccb3-2e3d-4599-a491-8167a2bd1e40" containerID="feff271e020e1e85dbc8115de4cd40e294b45a9cd03f729665fa36a1f91c7402" exitCode=0
Nov 03 01:30:48.558765 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.558563    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"
Nov 03 01:30:48.558765 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.558598    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv" event={"ID":"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40","Type":"ContainerDied","Data":"feff271e020e1e85dbc8115de4cd40e294b45a9cd03f729665fa36a1f91c7402"}
Nov 03 01:30:48.558765 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.558691    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv" event={"ID":"d2a9ccb3-2e3d-4599-a491-8167a2bd1e40","Type":"ContainerDied","Data":"a8d0453e80219171c2dc12e3d176cf5f53aa2b49f76a0f2d665adea4db48ad3d"}
Nov 03 01:30:48.558765 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.558724    5508 scope.go:122] "RemoveContainer" containerID="feff271e020e1e85dbc8115de4cd40e294b45a9cd03f729665fa36a1f91c7402"
Nov 03 01:30:48.568151 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.568129    5508 scope.go:122] "RemoveContainer" containerID="feff271e020e1e85dbc8115de4cd40e294b45a9cd03f729665fa36a1f91c7402"
Nov 03 01:30:48.568495 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:30:48.568476    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feff271e020e1e85dbc8115de4cd40e294b45a9cd03f729665fa36a1f91c7402\": container with ID starting with feff271e020e1e85dbc8115de4cd40e294b45a9cd03f729665fa36a1f91c7402 not found: ID does not exist" containerID="feff271e020e1e85dbc8115de4cd40e294b45a9cd03f729665fa36a1f91c7402"
Nov 03 01:30:48.568578 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.568505    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feff271e020e1e85dbc8115de4cd40e294b45a9cd03f729665fa36a1f91c7402"} err="failed to get container status \"feff271e020e1e85dbc8115de4cd40e294b45a9cd03f729665fa36a1f91c7402\": rpc error: code = NotFound desc = could not find container \"feff271e020e1e85dbc8115de4cd40e294b45a9cd03f729665fa36a1f91c7402\": container with ID starting with feff271e020e1e85dbc8115de4cd40e294b45a9cd03f729665fa36a1f91c7402 not found: ID does not exist"
Nov 03 01:30:48.715423 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.715391    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"]
Nov 03 01:30:48.761517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:48.761486    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-21c25193-55f98f767-4w6hv"]
Nov 03 01:30:49.073930 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.073901    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"]
Nov 03 01:30:49.074306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.074294    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d2a9ccb3-2e3d-4599-a491-8167a2bd1e40" containerName="payload-pre-processing"
Nov 03 01:30:49.074362 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.074307    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a9ccb3-2e3d-4599-a491-8167a2bd1e40" containerName="payload-pre-processing"
Nov 03 01:30:49.074400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.074384    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a9ccb3-2e3d-4599-a491-8167a2bd1e40" containerName="payload-pre-processing"
Nov 03 01:30:49.078889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.078873    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"
Nov 03 01:30:49.128722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.128698    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-0bf34a\""
Nov 03 01:30:49.128920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.128814    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-0bf34a-dockercfg-gfn46\""
Nov 03 01:30:49.131686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.131660    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"]
Nov 03 01:30:49.137553 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.137522    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crvt6\" (UniqueName: \"kubernetes.io/projected/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-kube-api-access-crvt6\") pod \"payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j\" (UID: \"6684afe1-182b-4c8b-b2ed-d9491cfd68a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"
Nov 03 01:30:49.137720 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.137700    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j\" (UID: \"6684afe1-182b-4c8b-b2ed-d9491cfd68a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"
Nov 03 01:30:49.238981 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.238944    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j\" (UID: \"6684afe1-182b-4c8b-b2ed-d9491cfd68a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"
Nov 03 01:30:49.239197 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.239013    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crvt6\" (UniqueName: \"kubernetes.io/projected/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-kube-api-access-crvt6\") pod \"payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j\" (UID: \"6684afe1-182b-4c8b-b2ed-d9491cfd68a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"
Nov 03 01:30:49.239751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.239722    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j\" (UID: \"6684afe1-182b-4c8b-b2ed-d9491cfd68a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"
Nov 03 01:30:49.296015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.295978    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crvt6\" (UniqueName: \"kubernetes.io/projected/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-kube-api-access-crvt6\") pod \"payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j\" (UID: \"6684afe1-182b-4c8b-b2ed-d9491cfd68a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"
Nov 03 01:30:49.391742 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.391659    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"
Nov 03 01:30:49.761969 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:49.761933    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"]
Nov 03 01:30:49.769590 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:30:49.769562    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6684afe1_182b_4c8b_b2ed_d9491cfd68a8.slice/crio-3abbbdb3f0809d35f9ff446902e9758ef4b23d182bfa9a0109194ea8aa2492c7 WatchSource:0}: Error finding container 3abbbdb3f0809d35f9ff446902e9758ef4b23d182bfa9a0109194ea8aa2492c7: Status 404 returned error can't find the container with id 3abbbdb3f0809d35f9ff446902e9758ef4b23d182bfa9a0109194ea8aa2492c7
Nov 03 01:30:50.571245 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:50.571207    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j" event={"ID":"6684afe1-182b-4c8b-b2ed-d9491cfd68a8","Type":"ContainerStarted","Data":"20aa4477277de3f3f8fbd2bdfbb88aeab2dae84247695eed5d5a1111cebf6e92"}
Nov 03 01:30:50.571440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:50.571271    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j" event={"ID":"6684afe1-182b-4c8b-b2ed-d9491cfd68a8","Type":"ContainerStarted","Data":"3abbbdb3f0809d35f9ff446902e9758ef4b23d182bfa9a0109194ea8aa2492c7"}
Nov 03 01:30:50.571440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:50.571304    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"
Nov 03 01:30:50.681405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:50.681358    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j" podStartSLOduration=1.681334642 podStartE2EDuration="1.681334642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:50.678699089 +0000 UTC m=+730.104305564" watchObservedRunningTime="2026-11-03 01:30:50.681334642 +0000 UTC m=+730.106941105"
Nov 03 01:30:50.690104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:50.690080    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a9ccb3-2e3d-4599-a491-8167a2bd1e40" path="/var/lib/kubelet/pods/d2a9ccb3-2e3d-4599-a491-8167a2bd1e40/volumes"
Nov 03 01:30:54.538396 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:54.538368    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"
Nov 03 01:30:54.538803 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:30:54.538644    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"
Nov 03 01:31:01.579828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:01.579794    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"
Nov 03 01:31:02.307751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:02.307712    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:31:11.521587 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:11.521510    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"]
Nov 03 01:31:11.568759 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:11.568712    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-21c25193-7nfs7"]
Nov 03 01:31:11.612492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:11.612418    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"]
Nov 03 01:31:11.612648 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:11.612627    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy" containerID="cri-o://33778f13edfa11483a8955014dfb9eb95f2d44e0005dcbbad55eac2154b74146" gracePeriod=30
Nov 03 01:31:11.751571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:11.751530    5508 generic.go:356] "Generic (PLEG): container finished" podID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerID="b948d0ea76b602a2a348cb871d767262befb20cf54791f26704861f3560c8fd0" exitCode=0
Nov 03 01:31:11.751765 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:11.751596    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99" event={"ID":"5da42a7e-c55b-451e-b887-ea9d8461f23b","Type":"ContainerDied","Data":"b948d0ea76b602a2a348cb871d767262befb20cf54791f26704861f3560c8fd0"}
Nov 03 01:31:11.751992 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:11.751973    5508 scope.go:122] "RemoveContainer" containerID="b948d0ea76b602a2a348cb871d767262befb20cf54791f26704861f3560c8fd0"
Nov 03 01:31:11.753178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:11.753139    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:31:11.753298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:11.753202    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:31:11.753298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:11.753233    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:31:11.891205 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:11.891124    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:31:12.705912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:12.705873    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b2241a-30ba-489b-bc0d-15cd4e44313d" path="/var/lib/kubelet/pods/92b2241a-30ba-489b-bc0d-15cd4e44313d/volumes"
Nov 03 01:31:12.829367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:12.824615    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99" event={"ID":"5da42a7e-c55b-451e-b887-ea9d8461f23b","Type":"ContainerStarted","Data":"5e91e1f2cbf4db6236ff6b554283818e007ca6ebdd27cf723e04a4296fc1c5b9"}
Nov 03 01:31:13.023723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.023645    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"]
Nov 03 01:31:13.027819 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.027799    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.136371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.135743    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9da431-openshift-default-dockercfg-bq94q\""
Nov 03 01:31:13.136371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.136260    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.136371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.136311    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.136371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.136377    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.136714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.136432    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.136714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.136468    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.136714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.136493    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.136714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.136566    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.136714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.136616    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.136714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.136642    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k95hw\" (UniqueName: \"kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-kube-api-access-k95hw\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.141326 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.141291    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"]
Nov 03 01:31:13.238113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.238061    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.238314 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.238162    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.238314 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.238199    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k95hw\" (UniqueName: \"kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-kube-api-access-k95hw\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.238314 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.238278    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.238314 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.238310    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.238570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.238372    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.238570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.238433    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.238570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.238466    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.245554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.242838    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.245554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.243783    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.245554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.243813    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.245554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.244047    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.245554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.244063    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.245554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.238491    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.268372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.261964    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.285579 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.285505    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.297463 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.297421    5508 patch_prober.go:29] interesting pod/e2e-ait-21c25193-openshift-default-967b44fd9-smz46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:31:13.297744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.297713    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:31:13.307807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.307775    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k95hw\" (UniqueName: \"kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-kube-api-access-k95hw\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.313378 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.311899    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.340301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.340273    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:13.803374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.801701    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:31:13.803374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.801862    5508 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 01:31:13.803374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.801905    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99" podUID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 01:31:13.831698 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:31:13.830796    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:31:13.904165 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.904116    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"]
Nov 03 01:31:13.938061 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:31:13.938022    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0b8bfb_8d40_46b7_8277_03d07f5e401a.slice/crio-87a4531c69789b5f04f7e0bd06e524a06dcf3690e3a43c56c29e77c093b60c7a WatchSource:0}: Error finding container 87a4531c69789b5f04f7e0bd06e524a06dcf3690e3a43c56c29e77c093b60c7a: Status 404 returned error can't find the container with id 87a4531c69789b5f04f7e0bd06e524a06dcf3690e3a43c56c29e77c093b60c7a
Nov 03 01:31:13.940007 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.939972    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:31:13.940123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.940057    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:31:13.940123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:13.940100    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:31:14.803371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:14.802743    5508 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 01:31:14.803371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:14.802839    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99" podUID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 01:31:14.838981 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:14.838948    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" event={"ID":"ba0b8bfb-8d40-46b7-8277-03d07f5e401a","Type":"ContainerStarted","Data":"ef28432c89aa40bbcc22d0198b2917633de262180d70f3432ccba38da79191c5"}
Nov 03 01:31:14.839157 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:14.839009    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" event={"ID":"ba0b8bfb-8d40-46b7-8277-03d07f5e401a","Type":"ContainerStarted","Data":"87a4531c69789b5f04f7e0bd06e524a06dcf3690e3a43c56c29e77c093b60c7a"}
Nov 03 01:31:15.341240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:15.341200    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:15.346086 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:15.346059    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:15.475209 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:15.475154    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" podStartSLOduration=3.475133593 podStartE2EDuration="3.475133593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:14.969689543 +0000 UTC m=+754.395296008" watchObservedRunningTime="2026-11-03 01:31:15.475133593 +0000 UTC m=+754.900740060"
Nov 03 01:31:15.800717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:15.800677    5508 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 01:31:15.800933 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:15.800741    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99" podUID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 01:31:15.845135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:15.845113    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:15.846217 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:15.846200    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:16.800750 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:16.800711    5508 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 01:31:16.800962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:16.800781    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99" podUID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 01:31:16.906587 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:16.906553    5508 generic.go:356] "Generic (PLEG): container finished" podID="a745a282-70f9-46f3-8a35-a64d6807040e" containerID="33778f13edfa11483a8955014dfb9eb95f2d44e0005dcbbad55eac2154b74146" exitCode=0
Nov 03 01:31:16.907045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:16.906633    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" event={"ID":"a745a282-70f9-46f3-8a35-a64d6807040e","Type":"ContainerDied","Data":"33778f13edfa11483a8955014dfb9eb95f2d44e0005dcbbad55eac2154b74146"}
Nov 03 01:31:16.907045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:16.906725    5508 scope.go:122] "RemoveContainer" containerID="a05644fb4eaaf625384e83fd9bbec5dc50e84a2c986da995411909922440d54e"
Nov 03 01:31:17.029841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.029820    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:31:17.139633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.139607    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a745a282-70f9-46f3-8a35-a64d6807040e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a745a282-70f9-46f3-8a35-a64d6807040e-istio-podinfo\") pod \"a745a282-70f9-46f3-8a35-a64d6807040e\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") "
Nov 03 01:31:17.139792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.139643    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-certs\") pod \"a745a282-70f9-46f3-8a35-a64d6807040e\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") "
Nov 03 01:31:17.139792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.139676    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a745a282-70f9-46f3-8a35-a64d6807040e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a745a282-70f9-46f3-8a35-a64d6807040e-istiod-ca-cert\") pod \"a745a282-70f9-46f3-8a35-a64d6807040e\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") "
Nov 03 01:31:17.139792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.139700    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-istio-token\" (UniqueName: \"kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-istio-token\") pod \"a745a282-70f9-46f3-8a35-a64d6807040e\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") "
Nov 03 01:31:17.139792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.139739    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-socket\") pod \"a745a282-70f9-46f3-8a35-a64d6807040e\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") "
Nov 03 01:31:17.139792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.139777    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-envoy\") pod \"a745a282-70f9-46f3-8a35-a64d6807040e\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") "
Nov 03 01:31:17.140064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.139806    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-data\") pod \"a745a282-70f9-46f3-8a35-a64d6807040e\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") "
Nov 03 01:31:17.140064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.139873    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-certs" pod "a745a282-70f9-46f3-8a35-a64d6807040e" (UID: "a745a282-70f9-46f3-8a35-a64d6807040e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:31:17.140064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.139880    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-credential-socket\") pod \"a745a282-70f9-46f3-8a35-a64d6807040e\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") "
Nov 03 01:31:17.140064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.139974    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-kube-api-access-qtvzt\" (UniqueName: \"kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-kube-api-access-qtvzt\") pod \"a745a282-70f9-46f3-8a35-a64d6807040e\" (UID: \"a745a282-70f9-46f3-8a35-a64d6807040e\") "
Nov 03 01:31:17.140282 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.140113    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-credential-socket" pod "a745a282-70f9-46f3-8a35-a64d6807040e" (UID: "a745a282-70f9-46f3-8a35-a64d6807040e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:31:17.140334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.140290    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-socket" pod "a745a282-70f9-46f3-8a35-a64d6807040e" (UID: "a745a282-70f9-46f3-8a35-a64d6807040e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:31:17.140464 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.140423    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:17.140525 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.140464    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-data" pod "a745a282-70f9-46f3-8a35-a64d6807040e" (UID: "a745a282-70f9-46f3-8a35-a64d6807040e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:31:17.140525 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.140472    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:17.140525 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.140487    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:17.140636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.140587    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a745a282-70f9-46f3-8a35-a64d6807040e-istiod-ca-cert" pod "a745a282-70f9-46f3-8a35-a64d6807040e" (UID: "a745a282-70f9-46f3-8a35-a64d6807040e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:31:17.142076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.142058    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a745a282-70f9-46f3-8a35-a64d6807040e-istio-podinfo" pod "a745a282-70f9-46f3-8a35-a64d6807040e" (UID: "a745a282-70f9-46f3-8a35-a64d6807040e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:31:17.142328 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.142307    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-istio-token" pod "a745a282-70f9-46f3-8a35-a64d6807040e" (UID: "a745a282-70f9-46f3-8a35-a64d6807040e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:31:17.142485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.142458    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-kube-api-access-qtvzt" pod "a745a282-70f9-46f3-8a35-a64d6807040e" (UID: "a745a282-70f9-46f3-8a35-a64d6807040e"). InnerVolumeSpecName "kube-api-access-qtvzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:31:17.142814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.142797    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-envoy" pod "a745a282-70f9-46f3-8a35-a64d6807040e" (UID: "a745a282-70f9-46f3-8a35-a64d6807040e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:31:17.241385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.241281    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtvzt\" (UniqueName: \"kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-kube-api-access-qtvzt\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:17.241385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.241310    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a745a282-70f9-46f3-8a35-a64d6807040e-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:17.241385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.241328    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a745a282-70f9-46f3-8a35-a64d6807040e-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:17.241385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.241336    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a745a282-70f9-46f3-8a35-a64d6807040e-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:17.241385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.241365    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:17.241385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.241374    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a745a282-70f9-46f3-8a35-a64d6807040e-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:31:17.800477 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.800434    5508 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 03 01:31:17.800660 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.800486    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99" podUID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 03 01:31:17.916580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.916547    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"
Nov 03 01:31:17.917022 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.916585    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46" event={"ID":"a745a282-70f9-46f3-8a35-a64d6807040e","Type":"ContainerDied","Data":"2f2620f3dc1635c6ba4751fadab17403083a4122ca00991cd69f42bce70d6256"}
Nov 03 01:31:17.917022 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:17.916651    5508 scope.go:122] "RemoveContainer" containerID="33778f13edfa11483a8955014dfb9eb95f2d44e0005dcbbad55eac2154b74146"
Nov 03 01:31:18.122177 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:18.122101    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"]
Nov 03 01:31:18.172380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:18.168978    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-21c25193-openshift-default-967b44fd9-smz46"]
Nov 03 01:31:18.692689 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:18.692656    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" path="/var/lib/kubelet/pods/a745a282-70f9-46f3-8a35-a64d6807040e/volumes"
Nov 03 01:31:18.803971 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:18.803944    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:31:18.804328 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:18.804310    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:31:18.805107 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:18.805079    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:31:20.938549 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:20.938524    5508 generic.go:356] "Generic (PLEG): container finished" podID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerID="8a5d0b4639a3103d1d71a767196b77d62f4c9de25693cafbf543dd0a1a8414cd" exitCode=0
Nov 03 01:31:20.939005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:20.938603    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl" event={"ID":"8e99b241-cf23-4d04-9087-87e5b89aaf9b","Type":"ContainerDied","Data":"8a5d0b4639a3103d1d71a767196b77d62f4c9de25693cafbf543dd0a1a8414cd"}
Nov 03 01:31:20.939073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:20.939047    5508 scope.go:122] "RemoveContainer" containerID="8a5d0b4639a3103d1d71a767196b77d62f4c9de25693cafbf543dd0a1a8414cd"
Nov 03 01:31:20.940236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:20.940209    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:31:20.940318 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:20.940271    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:31:20.940318 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:20.940302    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:31:21.947148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:21.947113    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl" event={"ID":"8e99b241-cf23-4d04-9087-87e5b89aaf9b","Type":"ContainerStarted","Data":"123c591325d9bbc960f98671ab7afc24dd51af539b3e835c7109d0efaabdea4f"}
Nov 03 01:31:22.297714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:22.297631    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:31:22.298828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:22.298799    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 01:31:22.298941 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:22.298846    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl" podUID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 01:31:23.297228 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:23.297191    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 01:31:23.297671 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:23.297243    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl" podUID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 01:31:23.488749 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:23.488707    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:31:24.298031 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:24.297995    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 01:31:24.298467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:24.298059    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl" podUID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 01:31:25.297542 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:25.297509    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 01:31:25.297721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:25.297566    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl" podUID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 01:31:26.298048 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:26.298010    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 01:31:26.298572 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:26.298073    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl" podUID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 01:31:27.301310 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:27.301283    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:31:27.302145 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:27.302128    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 01:31:28.323133 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.323101    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"]
Nov 03 01:31:28.323521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.323506    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy"
Nov 03 01:31:28.323521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.323517    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy"
Nov 03 01:31:28.323606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.323527    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy"
Nov 03 01:31:28.323606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.323533    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy"
Nov 03 01:31:28.323669 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.323622    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy"
Nov 03 01:31:28.323669 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.323632    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a745a282-70f9-46f3-8a35-a64d6807040e" containerName="istio-proxy"
Nov 03 01:31:28.327011 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.326994    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"
Nov 03 01:31:28.380420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.380375    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-e1793d\""
Nov 03 01:31:28.380606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.380544    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-e1793d-dockercfg-xp6rq\""
Nov 03 01:31:28.383374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.383324    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"]
Nov 03 01:31:28.503187 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.503144    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a3ebaf-96d0-4664-9775-2207983e2a96-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6\" (UID: \"27a3ebaf-96d0-4664-9775-2207983e2a96\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"
Nov 03 01:31:28.503412 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.503316    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjsz4\" (UniqueName: \"kubernetes.io/projected/27a3ebaf-96d0-4664-9775-2207983e2a96-kube-api-access-rjsz4\") pod \"payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6\" (UID: \"27a3ebaf-96d0-4664-9775-2207983e2a96\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"
Nov 03 01:31:28.604564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.604470    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjsz4\" (UniqueName: \"kubernetes.io/projected/27a3ebaf-96d0-4664-9775-2207983e2a96-kube-api-access-rjsz4\") pod \"payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6\" (UID: \"27a3ebaf-96d0-4664-9775-2207983e2a96\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"
Nov 03 01:31:28.604564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.604555    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a3ebaf-96d0-4664-9775-2207983e2a96-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6\" (UID: \"27a3ebaf-96d0-4664-9775-2207983e2a96\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"
Nov 03 01:31:28.605234 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.605214    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a3ebaf-96d0-4664-9775-2207983e2a96-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6\" (UID: \"27a3ebaf-96d0-4664-9775-2207983e2a96\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"
Nov 03 01:31:28.662306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.662268    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjsz4\" (UniqueName: \"kubernetes.io/projected/27a3ebaf-96d0-4664-9775-2207983e2a96-kube-api-access-rjsz4\") pod \"payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6\" (UID: \"27a3ebaf-96d0-4664-9775-2207983e2a96\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"
Nov 03 01:31:28.939369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:28.939320    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"
Nov 03 01:31:29.309095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:29.309019    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"]
Nov 03 01:31:29.316406 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:31:29.316375    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a3ebaf_96d0_4664_9775_2207983e2a96.slice/crio-4f7c70adf67a2bf5906dc06f6f99188741d164398f5ae807c5f85a6cae2d0941 WatchSource:0}: Error finding container 4f7c70adf67a2bf5906dc06f6f99188741d164398f5ae807c5f85a6cae2d0941: Status 404 returned error can't find the container with id 4f7c70adf67a2bf5906dc06f6f99188741d164398f5ae807c5f85a6cae2d0941
Nov 03 01:31:30.022993 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:30.022960    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6" event={"ID":"27a3ebaf-96d0-4664-9775-2207983e2a96","Type":"ContainerStarted","Data":"f4e81d9417c12e7eece1e3f98fdce565d248175be1b526dba3e3a09718a6aea2"}
Nov 03 01:31:30.023406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:30.023009    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6" event={"ID":"27a3ebaf-96d0-4664-9775-2207983e2a96","Type":"ContainerStarted","Data":"4f7c70adf67a2bf5906dc06f6f99188741d164398f5ae807c5f85a6cae2d0941"}
Nov 03 01:31:30.023406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:30.023093    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"
Nov 03 01:31:30.179686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:30.179632    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6" podStartSLOduration=2.179616772 podStartE2EDuration="2.179616772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:30.131992087 +0000 UTC m=+769.557598570" watchObservedRunningTime="2026-11-03 01:31:30.179616772 +0000 UTC m=+769.605223237"
Nov 03 01:31:34.616005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:34.615968    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:31:41.031541 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:41.031505    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"
Nov 03 01:31:43.811400 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:31:43.811335    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:31:48.296786 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:48.296747    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:31:58.209714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:58.209676    5508 generic.go:356] "Generic (PLEG): container finished" podID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerID="ef28432c89aa40bbcc22d0198b2917633de262180d70f3432ccba38da79191c5" exitCode=0
Nov 03 01:31:58.209714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:58.209716    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" event={"ID":"ba0b8bfb-8d40-46b7-8277-03d07f5e401a","Type":"ContainerDied","Data":"ef28432c89aa40bbcc22d0198b2917633de262180d70f3432ccba38da79191c5"}
Nov 03 01:31:58.210308 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:58.210113    5508 scope.go:122] "RemoveContainer" containerID="ef28432c89aa40bbcc22d0198b2917633de262180d70f3432ccba38da79191c5"
Nov 03 01:31:58.211585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:58.211552    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:31:58.211678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:58.211642    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:31:58.211738 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:58.211695    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:31:59.019796 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:59.019760    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:31:59.242655 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:59.241717    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" event={"ID":"ba0b8bfb-8d40-46b7-8277-03d07f5e401a","Type":"ContainerStarted","Data":"a8d01124001ebf7cd1352b7da1a5ecd2e05b15ef3bb2fbc33efd3cf2ed1aa553"}
Nov 03 01:31:59.340610 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:59.340517    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:31:59.342269 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:59.342241    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 01:31:59.342519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:31:59.342484    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 01:32:00.340781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:00.340746    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 01:32:00.341451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:00.341414    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 01:32:00.845475 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:00.845441    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:32:01.340945 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:01.340912    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 01:32:01.341338 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:01.340971    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 01:32:02.340817 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:02.340780    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 01:32:02.340993 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:02.340832    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 01:32:03.341056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:03.341026    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 01:32:03.341595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:03.341078    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 01:32:04.344574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:04.344538    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:32:04.345494 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:04.345474    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:32:13.810646 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:32:13.810611    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:32:16.181218 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:16.181137    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:32:41.673326 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:41.673285    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:32:43.810911 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:32:43.810875    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:32:44.660772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.660735    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"]
Nov 03 01:32:44.664712 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.664689    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.713916 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.713891    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9da431\"/\"default-dockercfg-8v9fs\""
Nov 03 01:32:44.746769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.746738    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx5gk\" (UniqueName: \"kubernetes.io/projected/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-kube-api-access-wx5gk\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.746913 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.746782    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tmp-dir\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.746913 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.746884    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-model-cache\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.746991 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.746947    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-home\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.746991 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.746984    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-dshm\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.747052 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.747011    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tls-certs\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.760111 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.760090    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9da431\"/\"kube-root-ca.crt\""
Nov 03 01:32:44.760210 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.760118    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9da431\"/\"openshift-service-ca.crt\""
Nov 03 01:32:44.760210 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.760152    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9da431\"/\"simulator-w2-9da431-kserve-self-signed-certs\""
Nov 03 01:32:44.765625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.765605    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"]
Nov 03 01:32:44.847791 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.847741    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tls-certs\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.848279 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.847855    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx5gk\" (UniqueName: \"kubernetes.io/projected/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-kube-api-access-wx5gk\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.848279 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.847908    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tmp-dir\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.848279 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.847975    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-model-cache\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.848279 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.848013    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-home\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.848279 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.848049    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-dshm\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.848568 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.848448    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tmp-dir\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.848568 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.848469    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-model-cache\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.848568 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.848535    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-home\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.850073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.850049    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-dshm\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.850461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.850440    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tls-certs\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.904932 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.904903    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx5gk\" (UniqueName: \"kubernetes.io/projected/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-kube-api-access-wx5gk\") pod \"simulator-w2-9da431-kserve-8df84bcb9-dhmcs\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") " pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:44.976689 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:44.976621    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:45.350772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:45.350692    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"]
Nov 03 01:32:45.357795 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:32:45.357769    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32669dc7_6c3a_4e8d_805b_651dbc1ab37f.slice/crio-459b271145ebf4e07784e9481206db33fd37f322ecf15c35f8bd9df18bab2d89 WatchSource:0}: Error finding container 459b271145ebf4e07784e9481206db33fd37f322ecf15c35f8bd9df18bab2d89: Status 404 returned error can't find the container with id 459b271145ebf4e07784e9481206db33fd37f322ecf15c35f8bd9df18bab2d89
Nov 03 01:32:45.525681 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:45.525652    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" event={"ID":"32669dc7-6c3a-4e8d-805b-651dbc1ab37f","Type":"ContainerStarted","Data":"459b271145ebf4e07784e9481206db33fd37f322ecf15c35f8bd9df18bab2d89"}
Nov 03 01:32:47.538819 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:47.538785    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" event={"ID":"32669dc7-6c3a-4e8d-805b-651dbc1ab37f","Type":"ContainerStarted","Data":"e338c94661370d4f774e35063c5b6d0967fc1a5fadca152d02e00fdb12715b80"}
Nov 03 01:32:47.650038 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:47.649987    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podStartSLOduration=2.111483961 podStartE2EDuration="3.649973622s" totalImagesPullingTime="1.538489661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:47.647497725 +0000 UTC m=+847.073104189" watchObservedRunningTime="2026-11-03 01:32:47.649973622 +0000 UTC m=+847.075580085"
Nov 03 01:32:47.977248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:47.977220    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:54.976836 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:54.976802    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:54.993017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:54.992984    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:32:55.627015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:32:55.626986    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:33:02.589411 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:02.589382    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:33:07.083761 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.083730    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"]
Nov 03 01:33:07.089951 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.089931    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"
Nov 03 01:33:07.141610 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.141583    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"]
Nov 03 01:33:07.184933 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.184907    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:33:07.185101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.184907    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:33:07.185101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.184921    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:33:07.185101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.184987    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:33:07.197553 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.197526    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlfg9\" (UniqueName: \"kubernetes.io/projected/82394f3d-e4c7-43d3-be66-92409381ed99-kube-api-access-hlfg9\") pod \"maas-api-revoke-sub-09c16e77-9jk84\" (UID: \"82394f3d-e4c7-43d3-be66-92409381ed99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"
Nov 03 01:33:07.197677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.197569    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c16e77-9jk84\" (UID: \"82394f3d-e4c7-43d3-be66-92409381ed99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"
Nov 03 01:33:07.197742 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.197693    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c16e77-9jk84\" (UID: \"82394f3d-e4c7-43d3-be66-92409381ed99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"
Nov 03 01:33:07.298971 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.298939    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlfg9\" (UniqueName: \"kubernetes.io/projected/82394f3d-e4c7-43d3-be66-92409381ed99-kube-api-access-hlfg9\") pod \"maas-api-revoke-sub-09c16e77-9jk84\" (UID: \"82394f3d-e4c7-43d3-be66-92409381ed99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"
Nov 03 01:33:07.299126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.298978    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c16e77-9jk84\" (UID: \"82394f3d-e4c7-43d3-be66-92409381ed99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"
Nov 03 01:33:07.299126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.299007    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c16e77-9jk84\" (UID: \"82394f3d-e4c7-43d3-be66-92409381ed99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"
Nov 03 01:33:07.299779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.299755    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c16e77-9jk84\" (UID: \"82394f3d-e4c7-43d3-be66-92409381ed99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"
Nov 03 01:33:07.301309 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.301291    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c16e77-9jk84\" (UID: \"82394f3d-e4c7-43d3-be66-92409381ed99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"
Nov 03 01:33:07.355813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.355763    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlfg9\" (UniqueName: \"kubernetes.io/projected/82394f3d-e4c7-43d3-be66-92409381ed99-kube-api-access-hlfg9\") pod \"maas-api-revoke-sub-09c16e77-9jk84\" (UID: \"82394f3d-e4c7-43d3-be66-92409381ed99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"
Nov 03 01:33:07.403098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.403072    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"
Nov 03 01:33:07.770011 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:07.769979    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"]
Nov 03 01:33:07.777146 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:33:07.777119    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82394f3d_e4c7_43d3_be66_92409381ed99.slice/crio-636317b5a1bd14cca3a093afa82bef5a2ca73a954b8fd181e8fd58861441c029 WatchSource:0}: Error finding container 636317b5a1bd14cca3a093afa82bef5a2ca73a954b8fd181e8fd58861441c029: Status 404 returned error can't find the container with id 636317b5a1bd14cca3a093afa82bef5a2ca73a954b8fd181e8fd58861441c029
Nov 03 01:33:08.687894 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:08.687859    5508 generic.go:356] "Generic (PLEG): container finished" podID="82394f3d-e4c7-43d3-be66-92409381ed99" containerID="607878e4c2921c2b0a4302461785d5bfe52561f7d28c8d54af5e8f171bf7d3d7" exitCode=0
Nov 03 01:33:08.690805 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:08.690778    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84" event={"ID":"82394f3d-e4c7-43d3-be66-92409381ed99","Type":"ContainerDied","Data":"607878e4c2921c2b0a4302461785d5bfe52561f7d28c8d54af5e8f171bf7d3d7"}
Nov 03 01:33:08.690925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:08.690812    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84" event={"ID":"82394f3d-e4c7-43d3-be66-92409381ed99","Type":"ContainerStarted","Data":"636317b5a1bd14cca3a093afa82bef5a2ca73a954b8fd181e8fd58861441c029"}
Nov 03 01:33:09.835546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:09.835518    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"
Nov 03 01:33:09.922563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:09.922534    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82394f3d-e4c7-43d3-be66-92409381ed99-kube-api-access-hlfg9\" (UniqueName: \"kubernetes.io/projected/82394f3d-e4c7-43d3-be66-92409381ed99-kube-api-access-hlfg9\") pod \"82394f3d-e4c7-43d3-be66-92409381ed99\" (UID: \"82394f3d-e4c7-43d3-be66-92409381ed99\") "
Nov 03 01:33:09.922726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:09.922586    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-xks-ca\") pod \"82394f3d-e4c7-43d3-be66-92409381ed99\" (UID: \"82394f3d-e4c7-43d3-be66-92409381ed99\") "
Nov 03 01:33:09.922726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:09.922690    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-openshift-ca\") pod \"82394f3d-e4c7-43d3-be66-92409381ed99\" (UID: \"82394f3d-e4c7-43d3-be66-92409381ed99\") "
Nov 03 01:33:09.923079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:09.923046    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-openshift-ca" pod "82394f3d-e4c7-43d3-be66-92409381ed99" (UID: "82394f3d-e4c7-43d3-be66-92409381ed99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:33:09.924580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:09.924557    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-xks-ca" pod "82394f3d-e4c7-43d3-be66-92409381ed99" (UID: "82394f3d-e4c7-43d3-be66-92409381ed99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:33:09.924694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:09.924678    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82394f3d-e4c7-43d3-be66-92409381ed99-kube-api-access-hlfg9" pod "82394f3d-e4c7-43d3-be66-92409381ed99" (UID: "82394f3d-e4c7-43d3-be66-92409381ed99"). InnerVolumeSpecName "kube-api-access-hlfg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:10.024424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:10.024340    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlfg9\" (UniqueName: \"kubernetes.io/projected/82394f3d-e4c7-43d3-be66-92409381ed99-kube-api-access-hlfg9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:10.024424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:10.024383    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:10.024424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:10.024392    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82394f3d-e4c7-43d3-be66-92409381ed99-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:10.732175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:10.732149    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"
Nov 03 01:33:10.732175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:10.732164    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84" event={"ID":"82394f3d-e4c7-43d3-be66-92409381ed99","Type":"ContainerDied","Data":"636317b5a1bd14cca3a093afa82bef5a2ca73a954b8fd181e8fd58861441c029"}
Nov 03 01:33:10.732498 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:10.732208    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="636317b5a1bd14cca3a093afa82bef5a2ca73a954b8fd181e8fd58861441c029"
Nov 03 01:33:13.811438 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:33:13.811398    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:33:18.453837 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:18.453797    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:33:24.972149 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:24.972105    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"]
Nov 03 01:33:24.973495 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:24.973467    5508 cpu_manager.go:414] "containerMap: removing container" podUID="82394f3d-e4c7-43d3-be66-92409381ed99" containerName="revoke-keys"
Nov 03 01:33:24.973783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:24.973762    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82394f3d-e4c7-43d3-be66-92409381ed99" containerName="revoke-keys"
Nov 03 01:33:24.974008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:24.973995    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="82394f3d-e4c7-43d3-be66-92409381ed99" containerName="revoke-keys"
Nov 03 01:33:24.979337 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:24.979315    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"
Nov 03 01:33:25.028570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.028541    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"]
Nov 03 01:33:25.071849 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.071818    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:33:25.071976 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.071853    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:33:25.071976 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.071887    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:33:25.073269 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.073247    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047b4a38-czq4r\" (UID: \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"
Nov 03 01:33:25.073337 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.073288    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047b4a38-czq4r\" (UID: \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"
Nov 03 01:33:25.073488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.073470    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7q8\" (UniqueName: \"kubernetes.io/projected/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-kube-api-access-9c7q8\") pod \"maas-api-revoke-sub-047b4a38-czq4r\" (UID: \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"
Nov 03 01:33:25.074691 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.074675    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:33:25.174734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.174709    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7q8\" (UniqueName: \"kubernetes.io/projected/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-kube-api-access-9c7q8\") pod \"maas-api-revoke-sub-047b4a38-czq4r\" (UID: \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"
Nov 03 01:33:25.174867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.174766    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047b4a38-czq4r\" (UID: \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"
Nov 03 01:33:25.174867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.174795    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047b4a38-czq4r\" (UID: \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"
Nov 03 01:33:25.175443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.175423    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047b4a38-czq4r\" (UID: \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"
Nov 03 01:33:25.177243 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.177219    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047b4a38-czq4r\" (UID: \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"
Nov 03 01:33:25.232628 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.232573    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7q8\" (UniqueName: \"kubernetes.io/projected/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-kube-api-access-9c7q8\") pod \"maas-api-revoke-sub-047b4a38-czq4r\" (UID: \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"
Nov 03 01:33:25.295396 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.295373    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"
Nov 03 01:33:25.664963 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.664935    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"]
Nov 03 01:33:25.671860 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:33:25.671827    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6021a8d_d122_42f5_a61e_4efa4e93ffbf.slice/crio-e1099762b59251aafdbad9b115fc89de69d59b251648cb3cb4df90014b9b57a6 WatchSource:0}: Error finding container e1099762b59251aafdbad9b115fc89de69d59b251648cb3cb4df90014b9b57a6: Status 404 returned error can't find the container with id e1099762b59251aafdbad9b115fc89de69d59b251648cb3cb4df90014b9b57a6
Nov 03 01:33:25.820005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:25.819984    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r" event={"ID":"d6021a8d-d122-42f5-a61e-4efa4e93ffbf","Type":"ContainerStarted","Data":"e1099762b59251aafdbad9b115fc89de69d59b251648cb3cb4df90014b9b57a6"}
Nov 03 01:33:26.826239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:26.826206    5508 generic.go:356] "Generic (PLEG): container finished" podID="d6021a8d-d122-42f5-a61e-4efa4e93ffbf" containerID="e80c2ee021c84ac12b2580f38ebf3ebf4c0b4eef77f329a1c0da8ad685edba3b" exitCode=0
Nov 03 01:33:26.826649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:26.826284    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r" event={"ID":"d6021a8d-d122-42f5-a61e-4efa4e93ffbf","Type":"ContainerDied","Data":"e80c2ee021c84ac12b2580f38ebf3ebf4c0b4eef77f329a1c0da8ad685edba3b"}
Nov 03 01:33:27.966299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:27.966267    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"
Nov 03 01:33:28.106539 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:28.106459    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-xks-ca\") pod \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\" (UID: \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\") "
Nov 03 01:33:28.106539 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:28.106534    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-openshift-ca\") pod \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\" (UID: \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\") "
Nov 03 01:33:28.106756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:28.106570    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-kube-api-access-9c7q8\" (UniqueName: \"kubernetes.io/projected/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-kube-api-access-9c7q8\") pod \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\" (UID: \"d6021a8d-d122-42f5-a61e-4efa4e93ffbf\") "
Nov 03 01:33:28.106958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:28.106932    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-openshift-ca" pod "d6021a8d-d122-42f5-a61e-4efa4e93ffbf" (UID: "d6021a8d-d122-42f5-a61e-4efa4e93ffbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:33:28.108612 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:28.108588    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-kube-api-access-9c7q8" pod "d6021a8d-d122-42f5-a61e-4efa4e93ffbf" (UID: "d6021a8d-d122-42f5-a61e-4efa4e93ffbf"). InnerVolumeSpecName "kube-api-access-9c7q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:28.108612 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:28.108599    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-xks-ca" pod "d6021a8d-d122-42f5-a61e-4efa4e93ffbf" (UID: "d6021a8d-d122-42f5-a61e-4efa4e93ffbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:33:28.207688 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:28.207658    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:28.207688 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:28.207683    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:28.207688 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:28.207694    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c7q8\" (UniqueName: \"kubernetes.io/projected/d6021a8d-d122-42f5-a61e-4efa4e93ffbf-kube-api-access-9c7q8\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:28.838093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:28.838062    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r" event={"ID":"d6021a8d-d122-42f5-a61e-4efa4e93ffbf","Type":"ContainerDied","Data":"e1099762b59251aafdbad9b115fc89de69d59b251648cb3cb4df90014b9b57a6"}
Nov 03 01:33:28.838266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:28.838109    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1099762b59251aafdbad9b115fc89de69d59b251648cb3cb4df90014b9b57a6"
Nov 03 01:33:28.838266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:28.838112    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"
Nov 03 01:33:37.343888 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.343855    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"]
Nov 03 01:33:37.344374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.344269    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d6021a8d-d122-42f5-a61e-4efa4e93ffbf" containerName="revoke-keys"
Nov 03 01:33:37.344374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.344280    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6021a8d-d122-42f5-a61e-4efa4e93ffbf" containerName="revoke-keys"
Nov 03 01:33:37.344374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.344341    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6021a8d-d122-42f5-a61e-4efa4e93ffbf" containerName="revoke-keys"
Nov 03 01:33:37.347561 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.347544    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.445957 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.445916    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"]
Nov 03 01:33:37.446132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.446068    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9da431\"/\"premium-w2-9da431-kserve-self-signed-certs\""
Nov 03 01:33:37.507022 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.506908    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-home\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.507022 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.506979    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rw2j\" (UniqueName: \"kubernetes.io/projected/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-kube-api-access-8rw2j\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.507395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.507088    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-model-cache\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.507395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.507162    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tls-certs\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.507395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.507272    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tmp-dir\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.507395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.507388    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-dshm\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.608211 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.608123    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rw2j\" (UniqueName: \"kubernetes.io/projected/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-kube-api-access-8rw2j\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.608211 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.608186    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-model-cache\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.608460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.608247    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tls-certs\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.608460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.608281    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tmp-dir\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.608460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.608314    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-dshm\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.608460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.608426    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-home\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.608695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.608671    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-model-cache\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.608758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.608690    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tmp-dir\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.608826 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.608806    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-home\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.611118 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.611079    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-dshm\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.611411 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.611368    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tls-certs\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.665318 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.665286    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rw2j\" (UniqueName: \"kubernetes.io/projected/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-kube-api-access-8rw2j\") pod \"premium-w2-9da431-kserve-6d58665668-s2ztz\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") " pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:37.959330 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:37.959292    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:38.329305 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:38.329236    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"]
Nov 03 01:33:38.336278 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:33:38.336252    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e6d6d46_c350_4e74_9bad_964e2fd33dbd.slice/crio-e9c682dc818159b4b2e9942d9e9c4e4573d9983587a3d3d1f37be0468bafb03b WatchSource:0}: Error finding container e9c682dc818159b4b2e9942d9e9c4e4573d9983587a3d3d1f37be0468bafb03b: Status 404 returned error can't find the container with id e9c682dc818159b4b2e9942d9e9c4e4573d9983587a3d3d1f37be0468bafb03b
Nov 03 01:33:38.894045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:38.894008    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" event={"ID":"9e6d6d46-c350-4e74-9bad-964e2fd33dbd","Type":"ContainerStarted","Data":"acd4fd150c0d041dd62ce2741b856cb315878e487105ed126f56d91b769e1363"}
Nov 03 01:33:38.894453 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:38.894052    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" event={"ID":"9e6d6d46-c350-4e74-9bad-964e2fd33dbd","Type":"ContainerStarted","Data":"e9c682dc818159b4b2e9942d9e9c4e4573d9983587a3d3d1f37be0468bafb03b"}
Nov 03 01:33:38.959795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:38.959765    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:39.005574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:39.005524    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podStartSLOduration=2.005510626 podStartE2EDuration="2.005510626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:33:39.002795317 +0000 UTC m=+898.428401799" watchObservedRunningTime="2026-11-03 01:33:39.005510626 +0000 UTC m=+898.431117099"
Nov 03 01:33:40.811499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:40.811472    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:33:40.815446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:40.815425    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:33:42.930556 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:42.930518    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:33:43.833722 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:33:43.833679    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:33:47.960087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:47.960003    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:47.972260 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:47.972237    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:48.985543 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:48.985508    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:33:49.091359 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:49.091315    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"]
Nov 03 01:33:49.091555 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:49.091534    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" podUID="15f26262-d73f-4709-9aba-388786aec407" containerName="istio-proxy" containerID="cri-o://9514be1043e5ee23d334095804c6dd7272ffe486fc5b1da902312354e8f92db3" gracePeriod=30
Nov 03 01:33:49.138148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:49.138108    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"]
Nov 03 01:33:49.138364 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:49.138322    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5" podUID="8e843812-79f6-4c3c-b215-474e4c966d49" containerName="istio-proxy" containerID="cri-o://d83545769c89a5d2b225cd5848cc5d2744e4b2f0687dd149822e11c75d41d937" gracePeriod=30
Nov 03 01:33:54.384373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.384336    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:33:54.415403 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.415384    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:33:54.492950 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.492923    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f26262-d73f-4709-9aba-388786aec407-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f26262-d73f-4709-9aba-388786aec407-istiod-ca-cert\") pod \"15f26262-d73f-4709-9aba-388786aec407\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") "
Nov 03 01:33:54.493079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.492958    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15f26262-d73f-4709-9aba-388786aec407-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f26262-d73f-4709-9aba-388786aec407-istio-podinfo\") pod \"15f26262-d73f-4709-9aba-388786aec407\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") "
Nov 03 01:33:54.493079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.492985    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-credential-socket\") pod \"15f26262-d73f-4709-9aba-388786aec407\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") "
Nov 03 01:33:54.493079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493011    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e843812-79f6-4c3c-b215-474e4c966d49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e843812-79f6-4c3c-b215-474e4c966d49-istiod-ca-cert\") pod \"8e843812-79f6-4c3c-b215-474e4c966d49\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") "
Nov 03 01:33:54.493079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493042    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-certs\") pod \"8e843812-79f6-4c3c-b215-474e4c966d49\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") "
Nov 03 01:33:54.493079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493072    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-istio-token\" (UniqueName: \"kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-istio-token\") pod \"8e843812-79f6-4c3c-b215-474e4c966d49\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") "
Nov 03 01:33:54.493334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493120    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-credential-socket\") pod \"8e843812-79f6-4c3c-b215-474e4c966d49\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") "
Nov 03 01:33:54.493334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493157    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-kube-api-access-t422n\" (UniqueName: \"kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-kube-api-access-t422n\") pod \"15f26262-d73f-4709-9aba-388786aec407\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") "
Nov 03 01:33:54.493334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493207    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-istio-token\" (UniqueName: \"kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-istio-token\") pod \"15f26262-d73f-4709-9aba-388786aec407\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") "
Nov 03 01:33:54.493334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493249    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-credential-socket" pod "15f26262-d73f-4709-9aba-388786aec407" (UID: "15f26262-d73f-4709-9aba-388786aec407"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:54.493334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493282    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-socket\") pod \"15f26262-d73f-4709-9aba-388786aec407\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") "
Nov 03 01:33:54.493643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493334    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-certs\") pod \"15f26262-d73f-4709-9aba-388786aec407\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") "
Nov 03 01:33:54.493643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493408    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-data\") pod \"15f26262-d73f-4709-9aba-388786aec407\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") "
Nov 03 01:33:54.493643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493409    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f26262-d73f-4709-9aba-388786aec407-istiod-ca-cert" pod "15f26262-d73f-4709-9aba-388786aec407" (UID: "15f26262-d73f-4709-9aba-388786aec407"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:33:54.493643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493410    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-certs" pod "8e843812-79f6-4c3c-b215-474e4c966d49" (UID: "8e843812-79f6-4c3c-b215-474e4c966d49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:54.493643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493446    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-envoy\") pod \"8e843812-79f6-4c3c-b215-474e4c966d49\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") "
Nov 03 01:33:54.493643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493484    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-envoy\") pod \"15f26262-d73f-4709-9aba-388786aec407\" (UID: \"15f26262-d73f-4709-9aba-388786aec407\") "
Nov 03 01:33:54.493643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493510    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-data\") pod \"8e843812-79f6-4c3c-b215-474e4c966d49\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") "
Nov 03 01:33:54.493643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493603    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-socket\") pod \"8e843812-79f6-4c3c-b215-474e4c966d49\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") "
Nov 03 01:33:54.493643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493635    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-kube-api-access-jltwf\" (UniqueName: \"kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-kube-api-access-jltwf\") pod \"8e843812-79f6-4c3c-b215-474e4c966d49\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") "
Nov 03 01:33:54.494095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493661    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e843812-79f6-4c3c-b215-474e4c966d49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e843812-79f6-4c3c-b215-474e4c966d49-istio-podinfo\") pod \"8e843812-79f6-4c3c-b215-474e4c966d49\" (UID: \"8e843812-79f6-4c3c-b215-474e4c966d49\") "
Nov 03 01:33:54.494095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493507    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-socket" pod "15f26262-d73f-4709-9aba-388786aec407" (UID: "15f26262-d73f-4709-9aba-388786aec407"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:54.494095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.494064    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f26262-d73f-4709-9aba-388786aec407-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.494095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.494084    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.494292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.494099    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.494292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.494112    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.494292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493569    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e843812-79f6-4c3c-b215-474e4c966d49-istiod-ca-cert" pod "8e843812-79f6-4c3c-b215-474e4c966d49" (UID: "8e843812-79f6-4c3c-b215-474e4c966d49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:33:54.494292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493601    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-certs" pod "15f26262-d73f-4709-9aba-388786aec407" (UID: "15f26262-d73f-4709-9aba-388786aec407"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:54.494292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493639    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-data" pod "15f26262-d73f-4709-9aba-388786aec407" (UID: "15f26262-d73f-4709-9aba-388786aec407"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:54.494292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493715    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-credential-socket" pod "8e843812-79f6-4c3c-b215-474e4c966d49" (UID: "8e843812-79f6-4c3c-b215-474e4c966d49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:54.494292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493789    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-data" pod "8e843812-79f6-4c3c-b215-474e4c966d49" (UID: "8e843812-79f6-4c3c-b215-474e4c966d49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:54.494292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.493970    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-socket" pod "8e843812-79f6-4c3c-b215-474e4c966d49" (UID: "8e843812-79f6-4c3c-b215-474e4c966d49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:54.495624 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.495586    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15f26262-d73f-4709-9aba-388786aec407-istio-podinfo" pod "15f26262-d73f-4709-9aba-388786aec407" (UID: "15f26262-d73f-4709-9aba-388786aec407"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:33:54.495998 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.495978    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-envoy" pod "8e843812-79f6-4c3c-b215-474e4c966d49" (UID: "8e843812-79f6-4c3c-b215-474e4c966d49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:54.495998 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.495990    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-istio-token" pod "15f26262-d73f-4709-9aba-388786aec407" (UID: "15f26262-d73f-4709-9aba-388786aec407"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:54.496127 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.496048    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-istio-token" pod "8e843812-79f6-4c3c-b215-474e4c966d49" (UID: "8e843812-79f6-4c3c-b215-474e4c966d49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:54.496178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.496169    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e843812-79f6-4c3c-b215-474e4c966d49-istio-podinfo" pod "8e843812-79f6-4c3c-b215-474e4c966d49" (UID: "8e843812-79f6-4c3c-b215-474e4c966d49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:33:54.496329 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.496304    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-envoy" pod "15f26262-d73f-4709-9aba-388786aec407" (UID: "15f26262-d73f-4709-9aba-388786aec407"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:54.496440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.496426    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-kube-api-access-t422n" pod "15f26262-d73f-4709-9aba-388786aec407" (UID: "15f26262-d73f-4709-9aba-388786aec407"). InnerVolumeSpecName "kube-api-access-t422n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:54.496667 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.496649    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-kube-api-access-jltwf" pod "8e843812-79f6-4c3c-b215-474e4c966d49" (UID: "8e843812-79f6-4c3c-b215-474e4c966d49"). InnerVolumeSpecName "kube-api-access-jltwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:54.594703 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594672    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.594703 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594697    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.594703 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594707    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.594912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594717    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.594912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594725    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f26262-d73f-4709-9aba-388786aec407-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.594912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594733    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.594912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594741    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.594912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594749    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jltwf\" (UniqueName: \"kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-kube-api-access-jltwf\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.594912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594758    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e843812-79f6-4c3c-b215-474e4c966d49-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.594912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594766    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f26262-d73f-4709-9aba-388786aec407-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.594912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594774    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e843812-79f6-4c3c-b215-474e4c966d49-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.594912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594781    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e843812-79f6-4c3c-b215-474e4c966d49-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.594912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594789    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e843812-79f6-4c3c-b215-474e4c966d49-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.594912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.594796    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t422n\" (UniqueName: \"kubernetes.io/projected/15f26262-d73f-4709-9aba-388786aec407-kube-api-access-t422n\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:54.991243 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.991166    5508 generic.go:356] "Generic (PLEG): container finished" podID="15f26262-d73f-4709-9aba-388786aec407" containerID="9514be1043e5ee23d334095804c6dd7272ffe486fc5b1da902312354e8f92db3" exitCode=0
Nov 03 01:33:54.991243 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.991214    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"
Nov 03 01:33:54.991499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.991249    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" event={"ID":"15f26262-d73f-4709-9aba-388786aec407","Type":"ContainerDied","Data":"9514be1043e5ee23d334095804c6dd7272ffe486fc5b1da902312354e8f92db3"}
Nov 03 01:33:54.991499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.991300    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g" event={"ID":"15f26262-d73f-4709-9aba-388786aec407","Type":"ContainerDied","Data":"79e1ed145808c8ad7f149442cac67869afb66a93297ed652a3f928ec9609a318"}
Nov 03 01:33:54.991499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.991329    5508 scope.go:122] "RemoveContainer" containerID="9514be1043e5ee23d334095804c6dd7272ffe486fc5b1da902312354e8f92db3"
Nov 03 01:33:54.992737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.992711    5508 generic.go:356] "Generic (PLEG): container finished" podID="8e843812-79f6-4c3c-b215-474e4c966d49" containerID="d83545769c89a5d2b225cd5848cc5d2744e4b2f0687dd149822e11c75d41d937" exitCode=0
Nov 03 01:33:54.992872 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.992755    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"
Nov 03 01:33:54.992872 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.992797    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5" event={"ID":"8e843812-79f6-4c3c-b215-474e4c966d49","Type":"ContainerDied","Data":"d83545769c89a5d2b225cd5848cc5d2744e4b2f0687dd149822e11c75d41d937"}
Nov 03 01:33:54.992872 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:54.992859    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5" event={"ID":"8e843812-79f6-4c3c-b215-474e4c966d49","Type":"ContainerDied","Data":"0d32258d8b34eb4f3f5e5a3dbec9d8344fb59d930c1cde1a7e5d3a97e5129c28"}
Nov 03 01:33:55.000768 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:55.000752    5508 scope.go:122] "RemoveContainer" containerID="9514be1043e5ee23d334095804c6dd7272ffe486fc5b1da902312354e8f92db3"
Nov 03 01:33:55.001009 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:33:55.000989    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9514be1043e5ee23d334095804c6dd7272ffe486fc5b1da902312354e8f92db3\": container with ID starting with 9514be1043e5ee23d334095804c6dd7272ffe486fc5b1da902312354e8f92db3 not found: ID does not exist" containerID="9514be1043e5ee23d334095804c6dd7272ffe486fc5b1da902312354e8f92db3"
Nov 03 01:33:55.001086 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:55.001019    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9514be1043e5ee23d334095804c6dd7272ffe486fc5b1da902312354e8f92db3"} err="failed to get container status \"9514be1043e5ee23d334095804c6dd7272ffe486fc5b1da902312354e8f92db3\": rpc error: code = NotFound desc = could not find container \"9514be1043e5ee23d334095804c6dd7272ffe486fc5b1da902312354e8f92db3\": container with ID starting with 9514be1043e5ee23d334095804c6dd7272ffe486fc5b1da902312354e8f92db3 not found: ID does not exist"
Nov 03 01:33:55.001086 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:55.001041    5508 scope.go:122] "RemoveContainer" containerID="d83545769c89a5d2b225cd5848cc5d2744e4b2f0687dd149822e11c75d41d937"
Nov 03 01:33:55.008922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:55.008905    5508 scope.go:122] "RemoveContainer" containerID="d83545769c89a5d2b225cd5848cc5d2744e4b2f0687dd149822e11c75d41d937"
Nov 03 01:33:55.009179 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:33:55.009161    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d83545769c89a5d2b225cd5848cc5d2744e4b2f0687dd149822e11c75d41d937\": container with ID starting with d83545769c89a5d2b225cd5848cc5d2744e4b2f0687dd149822e11c75d41d937 not found: ID does not exist" containerID="d83545769c89a5d2b225cd5848cc5d2744e4b2f0687dd149822e11c75d41d937"
Nov 03 01:33:55.009240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:55.009184    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d83545769c89a5d2b225cd5848cc5d2744e4b2f0687dd149822e11c75d41d937"} err="failed to get container status \"d83545769c89a5d2b225cd5848cc5d2744e4b2f0687dd149822e11c75d41d937\": rpc error: code = NotFound desc = could not find container \"d83545769c89a5d2b225cd5848cc5d2744e4b2f0687dd149822e11c75d41d937\": container with ID starting with d83545769c89a5d2b225cd5848cc5d2744e4b2f0687dd149822e11c75d41d937 not found: ID does not exist"
Nov 03 01:33:55.145511 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:55.145488    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"]
Nov 03 01:33:55.191871 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:55.191849    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4724g"]
Nov 03 01:33:55.252703 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:55.252644    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"]
Nov 03 01:33:55.257731 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:55.257707    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-6t5b5"]
Nov 03 01:33:56.176935 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.176902    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"]
Nov 03 01:33:56.177301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.177278    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8e843812-79f6-4c3c-b215-474e4c966d49" containerName="istio-proxy"
Nov 03 01:33:56.177301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.177288    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e843812-79f6-4c3c-b215-474e4c966d49" containerName="istio-proxy"
Nov 03 01:33:56.177393 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.177318    5508 cpu_manager.go:414] "containerMap: removing container" podUID="15f26262-d73f-4709-9aba-388786aec407" containerName="istio-proxy"
Nov 03 01:33:56.177393 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.177324    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f26262-d73f-4709-9aba-388786aec407" containerName="istio-proxy"
Nov 03 01:33:56.177455 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.177399    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f26262-d73f-4709-9aba-388786aec407" containerName="istio-proxy"
Nov 03 01:33:56.177455 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.177410    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e843812-79f6-4c3c-b215-474e4c966d49" containerName="istio-proxy"
Nov 03 01:33:56.181803 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.181785    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"
Nov 03 01:33:56.231932 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.231908    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:33:56.278087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.278061    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:33:56.278087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.278083    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:33:56.278268 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.278133    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:33:56.280298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.280276    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"]
Nov 03 01:33:56.309798 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.309774    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd41e900-kp2gj\" (UID: \"c12b0837-6877-4748-bae8-00ecb45fb7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"
Nov 03 01:33:56.309938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.309812    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6vjm\" (UniqueName: \"kubernetes.io/projected/c12b0837-6877-4748-bae8-00ecb45fb7f6-kube-api-access-k6vjm\") pod \"maas-api-revoke-sub-dd41e900-kp2gj\" (UID: \"c12b0837-6877-4748-bae8-00ecb45fb7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"
Nov 03 01:33:56.310013 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.309990    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd41e900-kp2gj\" (UID: \"c12b0837-6877-4748-bae8-00ecb45fb7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"
Nov 03 01:33:56.410568 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.410538    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd41e900-kp2gj\" (UID: \"c12b0837-6877-4748-bae8-00ecb45fb7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"
Nov 03 01:33:56.410674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.410578    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd41e900-kp2gj\" (UID: \"c12b0837-6877-4748-bae8-00ecb45fb7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"
Nov 03 01:33:56.410674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.410606    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vjm\" (UniqueName: \"kubernetes.io/projected/c12b0837-6877-4748-bae8-00ecb45fb7f6-kube-api-access-k6vjm\") pod \"maas-api-revoke-sub-dd41e900-kp2gj\" (UID: \"c12b0837-6877-4748-bae8-00ecb45fb7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"
Nov 03 01:33:56.411287 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.411251    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd41e900-kp2gj\" (UID: \"c12b0837-6877-4748-bae8-00ecb45fb7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"
Nov 03 01:33:56.412694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.412676    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd41e900-kp2gj\" (UID: \"c12b0837-6877-4748-bae8-00ecb45fb7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"
Nov 03 01:33:56.467947 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.467895    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vjm\" (UniqueName: \"kubernetes.io/projected/c12b0837-6877-4748-bae8-00ecb45fb7f6-kube-api-access-k6vjm\") pod \"maas-api-revoke-sub-dd41e900-kp2gj\" (UID: \"c12b0837-6877-4748-bae8-00ecb45fb7f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"
Nov 03 01:33:56.494921 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.494901    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"
Nov 03 01:33:56.692679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.692637    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f26262-d73f-4709-9aba-388786aec407" path="/var/lib/kubelet/pods/15f26262-d73f-4709-9aba-388786aec407/volumes"
Nov 03 01:33:56.693236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.693212    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e843812-79f6-4c3c-b215-474e4c966d49" path="/var/lib/kubelet/pods/8e843812-79f6-4c3c-b215-474e4c966d49/volumes"
Nov 03 01:33:56.863002 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:56.862936    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"]
Nov 03 01:33:56.870238 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:33:56.870213    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12b0837_6877_4748_bae8_00ecb45fb7f6.slice/crio-fab34b0bbd4329323d0135d96b53800f939342d34668b5b799164d200585a32f WatchSource:0}: Error finding container fab34b0bbd4329323d0135d96b53800f939342d34668b5b799164d200585a32f: Status 404 returned error can't find the container with id fab34b0bbd4329323d0135d96b53800f939342d34668b5b799164d200585a32f
Nov 03 01:33:57.012139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:57.012112    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj" event={"ID":"c12b0837-6877-4748-bae8-00ecb45fb7f6","Type":"ContainerStarted","Data":"fab34b0bbd4329323d0135d96b53800f939342d34668b5b799164d200585a32f"}
Nov 03 01:33:58.018877 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:58.018790    5508 generic.go:356] "Generic (PLEG): container finished" podID="c12b0837-6877-4748-bae8-00ecb45fb7f6" containerID="e41745219f24904bb3db7de8d615d8911ca02eb32af8715781c7a9cb0d4d3a15" exitCode=0
Nov 03 01:33:58.018877 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:58.018852    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj" event={"ID":"c12b0837-6877-4748-bae8-00ecb45fb7f6","Type":"ContainerDied","Data":"e41745219f24904bb3db7de8d615d8911ca02eb32af8715781c7a9cb0d4d3a15"}
Nov 03 01:33:59.170167 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:59.170142    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"
Nov 03 01:33:59.236315 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:59.236281    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-xks-ca\") pod \"c12b0837-6877-4748-bae8-00ecb45fb7f6\" (UID: \"c12b0837-6877-4748-bae8-00ecb45fb7f6\") "
Nov 03 01:33:59.236510 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:59.236372    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12b0837-6877-4748-bae8-00ecb45fb7f6-kube-api-access-k6vjm\" (UniqueName: \"kubernetes.io/projected/c12b0837-6877-4748-bae8-00ecb45fb7f6-kube-api-access-k6vjm\") pod \"c12b0837-6877-4748-bae8-00ecb45fb7f6\" (UID: \"c12b0837-6877-4748-bae8-00ecb45fb7f6\") "
Nov 03 01:33:59.236510 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:59.236439    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-openshift-ca\") pod \"c12b0837-6877-4748-bae8-00ecb45fb7f6\" (UID: \"c12b0837-6877-4748-bae8-00ecb45fb7f6\") "
Nov 03 01:33:59.236815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:59.236791    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-openshift-ca" pod "c12b0837-6877-4748-bae8-00ecb45fb7f6" (UID: "c12b0837-6877-4748-bae8-00ecb45fb7f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:33:59.238420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:59.238391    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12b0837-6877-4748-bae8-00ecb45fb7f6-kube-api-access-k6vjm" pod "c12b0837-6877-4748-bae8-00ecb45fb7f6" (UID: "c12b0837-6877-4748-bae8-00ecb45fb7f6"). InnerVolumeSpecName "kube-api-access-k6vjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:59.238538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:59.238459    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-xks-ca" pod "c12b0837-6877-4748-bae8-00ecb45fb7f6" (UID: "c12b0837-6877-4748-bae8-00ecb45fb7f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:33:59.338089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:59.338017    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:59.338089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:59.338045    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12b0837-6877-4748-bae8-00ecb45fb7f6-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:33:59.338089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:33:59.338055    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6vjm\" (UniqueName: \"kubernetes.io/projected/c12b0837-6877-4748-bae8-00ecb45fb7f6-kube-api-access-k6vjm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:00.030763 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:00.030731    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"
Nov 03 01:34:00.030934 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:00.030767    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj" event={"ID":"c12b0837-6877-4748-bae8-00ecb45fb7f6","Type":"ContainerDied","Data":"fab34b0bbd4329323d0135d96b53800f939342d34668b5b799164d200585a32f"}
Nov 03 01:34:00.030934 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:00.030828    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fab34b0bbd4329323d0135d96b53800f939342d34668b5b799164d200585a32f"
Nov 03 01:34:03.562219 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.562184    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"]
Nov 03 01:34:03.562803 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.562788    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c12b0837-6877-4748-bae8-00ecb45fb7f6" containerName="revoke-keys"
Nov 03 01:34:03.562853 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.562806    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12b0837-6877-4748-bae8-00ecb45fb7f6" containerName="revoke-keys"
Nov 03 01:34:03.562920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.562909    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12b0837-6877-4748-bae8-00ecb45fb7f6" containerName="revoke-keys"
Nov 03 01:34:03.566637 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.566615    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"
Nov 03 01:34:03.622338 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.619647    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"]
Nov 03 01:34:03.662017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.661980    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:34:03.662250 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.662064    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:34:03.662250 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.662138    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:34:03.662250 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.662219    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:34:03.678701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.678669    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ff137f-vhffk\" (UID: \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"
Nov 03 01:34:03.678822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.678726    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkpmh\" (UniqueName: \"kubernetes.io/projected/b06f6e57-fb78-435a-aa0c-4d624111a8d8-kube-api-access-jkpmh\") pod \"maas-api-revoke-sub-20ff137f-vhffk\" (UID: \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"
Nov 03 01:34:03.678919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.678893    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ff137f-vhffk\" (UID: \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"
Nov 03 01:34:03.780430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.780373    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ff137f-vhffk\" (UID: \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"
Nov 03 01:34:03.780679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.780487    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ff137f-vhffk\" (UID: \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"
Nov 03 01:34:03.780679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.780529    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkpmh\" (UniqueName: \"kubernetes.io/projected/b06f6e57-fb78-435a-aa0c-4d624111a8d8-kube-api-access-jkpmh\") pod \"maas-api-revoke-sub-20ff137f-vhffk\" (UID: \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"
Nov 03 01:34:03.781374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.781324    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ff137f-vhffk\" (UID: \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"
Nov 03 01:34:03.783071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.783043    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ff137f-vhffk\" (UID: \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"
Nov 03 01:34:03.838316 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.838247    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkpmh\" (UniqueName: \"kubernetes.io/projected/b06f6e57-fb78-435a-aa0c-4d624111a8d8-kube-api-access-jkpmh\") pod \"maas-api-revoke-sub-20ff137f-vhffk\" (UID: \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"
Nov 03 01:34:03.880899 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:03.880870    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"
Nov 03 01:34:04.257241 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.252135    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"]
Nov 03 01:34:04.261626 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:34:04.261597    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06f6e57_fb78_435a_aa0c_4d624111a8d8.slice/crio-acda90c0903b03231fc4a97eb61fb3a2fb8accd1a3f6e751a1f7e40dc944059d WatchSource:0}: Error finding container acda90c0903b03231fc4a97eb61fb3a2fb8accd1a3f6e751a1f7e40dc944059d: Status 404 returned error can't find the container with id acda90c0903b03231fc4a97eb61fb3a2fb8accd1a3f6e751a1f7e40dc944059d
Nov 03 01:34:04.754429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.754399    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"]
Nov 03 01:34:04.758311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.758294    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.808912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.808838    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e1793d\"/\"kube-root-ca.crt\""
Nov 03 01:34:04.855117 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.855086    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e1793d\"/\"premium-w1-e1793d-kserve-self-signed-certs\""
Nov 03 01:34:04.855117 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.855099    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e1793d\"/\"openshift-service-ca.crt\""
Nov 03 01:34:04.855117 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.855111    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e1793d\"/\"default-dockercfg-s7nrz\""
Nov 03 01:34:04.857475 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.857456    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"]
Nov 03 01:34:04.891582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.891174    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdc8x\" (UniqueName: \"kubernetes.io/projected/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-kube-api-access-kdc8x\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.891582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.891259    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tmp-dir\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.891582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.891290    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-home\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.891582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.891317    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tls-certs\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.891582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.891371    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-dshm\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.891582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.891476    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-model-cache\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.992658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.992626    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdc8x\" (UniqueName: \"kubernetes.io/projected/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-kube-api-access-kdc8x\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.992804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.992676    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tmp-dir\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.992804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.992699    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-home\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.992804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.992727    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tls-certs\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.992804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.992754    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-dshm\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.992994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.992840    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-model-cache\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.993102 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.993083    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tmp-dir\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.993164 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.993108    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-home\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.993164 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.993142    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-model-cache\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.994951 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.994934    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-dshm\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:04.995215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:04.995201    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tls-certs\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:05.049844 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:05.049817    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdc8x\" (UniqueName: \"kubernetes.io/projected/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-kube-api-access-kdc8x\") pod \"premium-w1-e1793d-kserve-7dd5954c8-kkgw9\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") " pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:05.061154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:05.061080    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk" event={"ID":"b06f6e57-fb78-435a-aa0c-4d624111a8d8","Type":"ContainerStarted","Data":"39462141eb5bcec093c48a6a1468f45415f039004b421fc4f35ad4b24ddfeea3"}
Nov 03 01:34:05.061154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:05.061141    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk" event={"ID":"b06f6e57-fb78-435a-aa0c-4d624111a8d8","Type":"ContainerStarted","Data":"acda90c0903b03231fc4a97eb61fb3a2fb8accd1a3f6e751a1f7e40dc944059d"}
Nov 03 01:34:05.070409 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:05.070389    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:05.175741 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:05.175690    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk" podStartSLOduration=2.175674237 podStartE2EDuration="2.175674237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:05.171466113 +0000 UTC m=+924.597072606" watchObservedRunningTime="2026-11-03 01:34:05.175674237 +0000 UTC m=+924.601280701"
Nov 03 01:34:05.448901 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:05.448868    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"]
Nov 03 01:34:05.455966 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:34:05.455938    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fb5a7e5_d503_4be7_a6be_fe9f7f936028.slice/crio-3e7afc3b18559d587371632865a0a82347a35ea62aae07e5852f0bd878616bb6 WatchSource:0}: Error finding container 3e7afc3b18559d587371632865a0a82347a35ea62aae07e5852f0bd878616bb6: Status 404 returned error can't find the container with id 3e7afc3b18559d587371632865a0a82347a35ea62aae07e5852f0bd878616bb6
Nov 03 01:34:06.069693 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:06.069578    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" event={"ID":"3fb5a7e5-d503-4be7-a6be-fe9f7f936028","Type":"ContainerStarted","Data":"5ec5a6471468a361081683eb32dc521adfe418e8fae4f39c1d6f513737986fb9"}
Nov 03 01:34:06.070320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:06.070294    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" event={"ID":"3fb5a7e5-d503-4be7-a6be-fe9f7f936028","Type":"ContainerStarted","Data":"3e7afc3b18559d587371632865a0a82347a35ea62aae07e5852f0bd878616bb6"}
Nov 03 01:34:06.070614 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:06.070593    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:06.071326 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:06.071302    5508 generic.go:356] "Generic (PLEG): container finished" podID="b06f6e57-fb78-435a-aa0c-4d624111a8d8" containerID="39462141eb5bcec093c48a6a1468f45415f039004b421fc4f35ad4b24ddfeea3" exitCode=0
Nov 03 01:34:06.071474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:06.071393    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk" event={"ID":"b06f6e57-fb78-435a-aa0c-4d624111a8d8","Type":"ContainerDied","Data":"39462141eb5bcec093c48a6a1468f45415f039004b421fc4f35ad4b24ddfeea3"}
Nov 03 01:34:06.182387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:06.182327    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podStartSLOduration=2.182314092 podStartE2EDuration="2.182314092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:06.181024567 +0000 UTC m=+925.606631049" watchObservedRunningTime="2026-11-03 01:34:06.182314092 +0000 UTC m=+925.607920556"
Nov 03 01:34:07.207786 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:07.207762    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"
Nov 03 01:34:07.314992 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:07.314956    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-openshift-ca\") pod \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\" (UID: \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\") "
Nov 03 01:34:07.315147 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:07.315026    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-xks-ca\") pod \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\" (UID: \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\") "
Nov 03 01:34:07.315232 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:07.315154    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b06f6e57-fb78-435a-aa0c-4d624111a8d8-kube-api-access-jkpmh\" (UniqueName: \"kubernetes.io/projected/b06f6e57-fb78-435a-aa0c-4d624111a8d8-kube-api-access-jkpmh\") pod \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\" (UID: \"b06f6e57-fb78-435a-aa0c-4d624111a8d8\") "
Nov 03 01:34:07.315432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:07.315406    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-openshift-ca" pod "b06f6e57-fb78-435a-aa0c-4d624111a8d8" (UID: "b06f6e57-fb78-435a-aa0c-4d624111a8d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:07.315535 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:07.315512    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:07.317538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:07.317491    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-xks-ca" pod "b06f6e57-fb78-435a-aa0c-4d624111a8d8" (UID: "b06f6e57-fb78-435a-aa0c-4d624111a8d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:34:07.318134 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:07.318108    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b06f6e57-fb78-435a-aa0c-4d624111a8d8-kube-api-access-jkpmh" pod "b06f6e57-fb78-435a-aa0c-4d624111a8d8" (UID: "b06f6e57-fb78-435a-aa0c-4d624111a8d8"). InnerVolumeSpecName "kube-api-access-jkpmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:07.416583 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:07.416555    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkpmh\" (UniqueName: \"kubernetes.io/projected/b06f6e57-fb78-435a-aa0c-4d624111a8d8-kube-api-access-jkpmh\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:07.416583 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:07.416586    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b06f6e57-fb78-435a-aa0c-4d624111a8d8-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:08.084763 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:08.084737    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"
Nov 03 01:34:08.085040 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:08.084773    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk" event={"ID":"b06f6e57-fb78-435a-aa0c-4d624111a8d8","Type":"ContainerDied","Data":"acda90c0903b03231fc4a97eb61fb3a2fb8accd1a3f6e751a1f7e40dc944059d"}
Nov 03 01:34:08.085040 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:08.084816    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acda90c0903b03231fc4a97eb61fb3a2fb8accd1a3f6e751a1f7e40dc944059d"
Nov 03 01:34:10.243620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:10.243584    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:34:11.541679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.541643    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"]
Nov 03 01:34:11.542103 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.542090    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b06f6e57-fb78-435a-aa0c-4d624111a8d8" containerName="revoke-keys"
Nov 03 01:34:11.542148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.542104    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b06f6e57-fb78-435a-aa0c-4d624111a8d8" containerName="revoke-keys"
Nov 03 01:34:11.542184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.542178    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b06f6e57-fb78-435a-aa0c-4d624111a8d8" containerName="revoke-keys"
Nov 03 01:34:11.546528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.546511    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"
Nov 03 01:34:11.604171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.604132    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:34:11.604171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.604147    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:34:11.604171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.604147    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:34:11.650307 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.650268    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:34:11.652585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.652564    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"]
Nov 03 01:34:11.658604 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.658569    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44441b6-kpnst\" (UID: \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"
Nov 03 01:34:11.658717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.658633    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vc8\" (UniqueName: \"kubernetes.io/projected/c93c4422-ae6d-4532-bdb2-957a9a0280c6-kube-api-access-k9vc8\") pod \"maas-api-revoke-sub-c44441b6-kpnst\" (UID: \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"
Nov 03 01:34:11.658770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.658749    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44441b6-kpnst\" (UID: \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"
Nov 03 01:34:11.760263 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.760223    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vc8\" (UniqueName: \"kubernetes.io/projected/c93c4422-ae6d-4532-bdb2-957a9a0280c6-kube-api-access-k9vc8\") pod \"maas-api-revoke-sub-c44441b6-kpnst\" (UID: \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"
Nov 03 01:34:11.760483 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.760290    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44441b6-kpnst\" (UID: \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"
Nov 03 01:34:11.760483 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.760444    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44441b6-kpnst\" (UID: \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"
Nov 03 01:34:11.761100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.761074    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44441b6-kpnst\" (UID: \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"
Nov 03 01:34:11.763301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.763272    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44441b6-kpnst\" (UID: \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"
Nov 03 01:34:11.819444 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.819380    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vc8\" (UniqueName: \"kubernetes.io/projected/c93c4422-ae6d-4532-bdb2-957a9a0280c6-kube-api-access-k9vc8\") pod \"maas-api-revoke-sub-c44441b6-kpnst\" (UID: \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"
Nov 03 01:34:11.859768 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:11.859744    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"
Nov 03 01:34:12.228005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:12.227971    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"]
Nov 03 01:34:12.235235 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:34:12.235208    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93c4422_ae6d_4532_bdb2_957a9a0280c6.slice/crio-5ae89e136053ac7ebefcc0d0bd9c7ef5fe25ec5e18242955110cdbcf3110a294 WatchSource:0}: Error finding container 5ae89e136053ac7ebefcc0d0bd9c7ef5fe25ec5e18242955110cdbcf3110a294: Status 404 returned error can't find the container with id 5ae89e136053ac7ebefcc0d0bd9c7ef5fe25ec5e18242955110cdbcf3110a294
Nov 03 01:34:13.117732 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:13.117696    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst" event={"ID":"c93c4422-ae6d-4532-bdb2-957a9a0280c6","Type":"ContainerStarted","Data":"cd76470a1275cb93a2377fa3b440747c5859bc1271c2db09fe623f2c3b734a96"}
Nov 03 01:34:13.118150 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:13.117755    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst" event={"ID":"c93c4422-ae6d-4532-bdb2-957a9a0280c6","Type":"ContainerStarted","Data":"5ae89e136053ac7ebefcc0d0bd9c7ef5fe25ec5e18242955110cdbcf3110a294"}
Nov 03 01:34:13.227178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:13.227124    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst" podStartSLOduration=2.227109457 podStartE2EDuration="2.227109457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:13.225800149 +0000 UTC m=+932.651406633" watchObservedRunningTime="2026-11-03 01:34:13.227109457 +0000 UTC m=+932.652715946"
Nov 03 01:34:13.811524 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:34:13.811494    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:34:14.125701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:14.125612    5508 generic.go:356] "Generic (PLEG): container finished" podID="c93c4422-ae6d-4532-bdb2-957a9a0280c6" containerID="cd76470a1275cb93a2377fa3b440747c5859bc1271c2db09fe623f2c3b734a96" exitCode=0
Nov 03 01:34:14.125701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:14.125658    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst" event={"ID":"c93c4422-ae6d-4532-bdb2-957a9a0280c6","Type":"ContainerDied","Data":"cd76470a1275cb93a2377fa3b440747c5859bc1271c2db09fe623f2c3b734a96"}
Nov 03 01:34:15.071414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:15.071383    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:15.083550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:15.083519    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:15.144450 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:15.144423    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:34:15.269575 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:15.269550    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"
Nov 03 01:34:15.400952 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:15.400918    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-openshift-ca\") pod \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\" (UID: \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\") "
Nov 03 01:34:15.401146 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:15.401012    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c93c4422-ae6d-4532-bdb2-957a9a0280c6-kube-api-access-k9vc8\" (UniqueName: \"kubernetes.io/projected/c93c4422-ae6d-4532-bdb2-957a9a0280c6-kube-api-access-k9vc8\") pod \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\" (UID: \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\") "
Nov 03 01:34:15.401146 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:15.401121    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-xks-ca\") pod \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\" (UID: \"c93c4422-ae6d-4532-bdb2-957a9a0280c6\") "
Nov 03 01:34:15.401318 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:15.401287    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-openshift-ca" pod "c93c4422-ae6d-4532-bdb2-957a9a0280c6" (UID: "c93c4422-ae6d-4532-bdb2-957a9a0280c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:15.401536 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:15.401515    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:15.406077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:15.406054    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93c4422-ae6d-4532-bdb2-957a9a0280c6-kube-api-access-k9vc8" pod "c93c4422-ae6d-4532-bdb2-957a9a0280c6" (UID: "c93c4422-ae6d-4532-bdb2-957a9a0280c6"). InnerVolumeSpecName "kube-api-access-k9vc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:15.406077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:15.406058    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-xks-ca" pod "c93c4422-ae6d-4532-bdb2-957a9a0280c6" (UID: "c93c4422-ae6d-4532-bdb2-957a9a0280c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:34:15.502238 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:15.502202    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9vc8\" (UniqueName: \"kubernetes.io/projected/c93c4422-ae6d-4532-bdb2-957a9a0280c6-kube-api-access-k9vc8\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:15.502238 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:15.502232    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93c4422-ae6d-4532-bdb2-957a9a0280c6-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:16.139481 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:16.139450    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"
Nov 03 01:34:16.139481 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:16.139472    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst" event={"ID":"c93c4422-ae6d-4532-bdb2-957a9a0280c6","Type":"ContainerDied","Data":"5ae89e136053ac7ebefcc0d0bd9c7ef5fe25ec5e18242955110cdbcf3110a294"}
Nov 03 01:34:16.139682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:16.139515    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ae89e136053ac7ebefcc0d0bd9c7ef5fe25ec5e18242955110cdbcf3110a294"
Nov 03 01:34:23.666799 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.666766    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"]
Nov 03 01:34:23.667197 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.667177    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c93c4422-ae6d-4532-bdb2-957a9a0280c6" containerName="revoke-keys"
Nov 03 01:34:23.667197 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.667188    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c93c4422-ae6d-4532-bdb2-957a9a0280c6" containerName="revoke-keys"
Nov 03 01:34:23.667277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.667258    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c93c4422-ae6d-4532-bdb2-957a9a0280c6" containerName="revoke-keys"
Nov 03 01:34:23.670337 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.670318    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"
Nov 03 01:34:23.724194 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.724168    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"]
Nov 03 01:34:23.767173 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.767137    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:34:23.767173 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.767148    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:34:23.767357 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.767229    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:34:23.767357 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.767245    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:34:23.791761 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.791741    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvz7l\" (UniqueName: \"kubernetes.io/projected/de781385-68e0-42c5-869b-b00a6a15dd3c-kube-api-access-tvz7l\") pod \"maas-api-revoke-sub-cbf38d6a-z5dv6\" (UID: \"de781385-68e0-42c5-869b-b00a6a15dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"
Nov 03 01:34:23.791873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.791806    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbf38d6a-z5dv6\" (UID: \"de781385-68e0-42c5-869b-b00a6a15dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"
Nov 03 01:34:23.791933 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.791919    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbf38d6a-z5dv6\" (UID: \"de781385-68e0-42c5-869b-b00a6a15dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"
Nov 03 01:34:23.892546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.892516    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbf38d6a-z5dv6\" (UID: \"de781385-68e0-42c5-869b-b00a6a15dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"
Nov 03 01:34:23.892709 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.892583    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbf38d6a-z5dv6\" (UID: \"de781385-68e0-42c5-869b-b00a6a15dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"
Nov 03 01:34:23.892709 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.892653    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvz7l\" (UniqueName: \"kubernetes.io/projected/de781385-68e0-42c5-869b-b00a6a15dd3c-kube-api-access-tvz7l\") pod \"maas-api-revoke-sub-cbf38d6a-z5dv6\" (UID: \"de781385-68e0-42c5-869b-b00a6a15dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"
Nov 03 01:34:23.893742 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.893718    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbf38d6a-z5dv6\" (UID: \"de781385-68e0-42c5-869b-b00a6a15dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"
Nov 03 01:34:23.895787 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.895750    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbf38d6a-z5dv6\" (UID: \"de781385-68e0-42c5-869b-b00a6a15dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"
Nov 03 01:34:23.950106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.950041    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvz7l\" (UniqueName: \"kubernetes.io/projected/de781385-68e0-42c5-869b-b00a6a15dd3c-kube-api-access-tvz7l\") pod \"maas-api-revoke-sub-cbf38d6a-z5dv6\" (UID: \"de781385-68e0-42c5-869b-b00a6a15dd3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"
Nov 03 01:34:23.983392 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:23.983365    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"
Nov 03 01:34:24.351523 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:24.351436    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"]
Nov 03 01:34:24.360468 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:34:24.360442    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde781385_68e0_42c5_869b_b00a6a15dd3c.slice/crio-f39c36ad2b90fdaebf58f1073048d018260113fd970c10aad870250e10ded2cd WatchSource:0}: Error finding container f39c36ad2b90fdaebf58f1073048d018260113fd970c10aad870250e10ded2cd: Status 404 returned error can't find the container with id f39c36ad2b90fdaebf58f1073048d018260113fd970c10aad870250e10ded2cd
Nov 03 01:34:25.192506 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:25.192479    5508 generic.go:356] "Generic (PLEG): container finished" podID="de781385-68e0-42c5-869b-b00a6a15dd3c" containerID="ecad5c82051500b6dbacf5c6180e4617983eb6093bc4a1463063c49659ee35f8" exitCode=0
Nov 03 01:34:25.192880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:25.192553    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6" event={"ID":"de781385-68e0-42c5-869b-b00a6a15dd3c","Type":"ContainerDied","Data":"ecad5c82051500b6dbacf5c6180e4617983eb6093bc4a1463063c49659ee35f8"}
Nov 03 01:34:25.192880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:25.192610    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6" event={"ID":"de781385-68e0-42c5-869b-b00a6a15dd3c","Type":"ContainerStarted","Data":"f39c36ad2b90fdaebf58f1073048d018260113fd970c10aad870250e10ded2cd"}
Nov 03 01:34:26.392608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:26.392582    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"
Nov 03 01:34:26.420294 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:26.420260    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-xks-ca\") pod \"de781385-68e0-42c5-869b-b00a6a15dd3c\" (UID: \"de781385-68e0-42c5-869b-b00a6a15dd3c\") "
Nov 03 01:34:26.420458 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:26.420439    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de781385-68e0-42c5-869b-b00a6a15dd3c-kube-api-access-tvz7l\" (UniqueName: \"kubernetes.io/projected/de781385-68e0-42c5-869b-b00a6a15dd3c-kube-api-access-tvz7l\") pod \"de781385-68e0-42c5-869b-b00a6a15dd3c\" (UID: \"de781385-68e0-42c5-869b-b00a6a15dd3c\") "
Nov 03 01:34:26.420557 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:26.420541    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-openshift-ca\") pod \"de781385-68e0-42c5-869b-b00a6a15dd3c\" (UID: \"de781385-68e0-42c5-869b-b00a6a15dd3c\") "
Nov 03 01:34:26.420907 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:26.420880    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-openshift-ca" pod "de781385-68e0-42c5-869b-b00a6a15dd3c" (UID: "de781385-68e0-42c5-869b-b00a6a15dd3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:26.422515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:26.422484    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-xks-ca" pod "de781385-68e0-42c5-869b-b00a6a15dd3c" (UID: "de781385-68e0-42c5-869b-b00a6a15dd3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:34:26.422652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:26.422515    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de781385-68e0-42c5-869b-b00a6a15dd3c-kube-api-access-tvz7l" pod "de781385-68e0-42c5-869b-b00a6a15dd3c" (UID: "de781385-68e0-42c5-869b-b00a6a15dd3c"). InnerVolumeSpecName "kube-api-access-tvz7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:26.522426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:26.522316    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:26.522426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:26.522371    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de781385-68e0-42c5-869b-b00a6a15dd3c-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:26.522426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:26.522391    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvz7l\" (UniqueName: \"kubernetes.io/projected/de781385-68e0-42c5-869b-b00a6a15dd3c-kube-api-access-tvz7l\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:27.209800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:27.209773    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"
Nov 03 01:34:27.209800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:27.209779    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6" event={"ID":"de781385-68e0-42c5-869b-b00a6a15dd3c","Type":"ContainerDied","Data":"f39c36ad2b90fdaebf58f1073048d018260113fd970c10aad870250e10ded2cd"}
Nov 03 01:34:27.210010 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:27.209832    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f39c36ad2b90fdaebf58f1073048d018260113fd970c10aad870250e10ded2cd"
Nov 03 01:34:30.800679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.800648    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"]
Nov 03 01:34:30.801065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.801053    5508 cpu_manager.go:414] "containerMap: removing container" podUID="de781385-68e0-42c5-869b-b00a6a15dd3c" containerName="revoke-keys"
Nov 03 01:34:30.801112 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.801066    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de781385-68e0-42c5-869b-b00a6a15dd3c" containerName="revoke-keys"
Nov 03 01:34:30.801146 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.801140    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="de781385-68e0-42c5-869b-b00a6a15dd3c" containerName="revoke-keys"
Nov 03 01:34:30.804588 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.804571    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.853528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.853490    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e1793d-openshift-default-dockercfg-4jbqs\""
Nov 03 01:34:30.868582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.868548    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.868713 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.868607    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.868713 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.868677    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.868819 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.868722    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.868881 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.868857    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.868940 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.868886    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.868940 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.868920    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2jnt\" (UniqueName: \"kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-kube-api-access-m2jnt\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.869031 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.868949    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.869031 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.868979    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.902734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.902703    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"]
Nov 03 01:34:30.970442 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.970409    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.970442 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.970445    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.970704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.970474    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.970704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.970521    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.970704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.970536    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.970704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.970557    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2jnt\" (UniqueName: \"kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-kube-api-access-m2jnt\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.970704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.970673    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.970970 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.970844    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.970970 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.970866    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.970970 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.970932    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.971135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.970994    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.971207 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.971183    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.971265 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.971235    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.971265 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.971249    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.972843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.972817    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:30.972987 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:30.972968    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:31.028145 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:31.028077    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:31.028402 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:31.028379    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2jnt\" (UniqueName: \"kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-kube-api-access-m2jnt\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:31.118062 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:31.118015    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:31.550489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:31.550453    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"]
Nov 03 01:34:31.557141 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:34:31.557114    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0e9d58_d98f_4c20_bb5e_c094ff18a48c.slice/crio-429cb121da4f6c69bff3e88e59b5f986dc797fb5130cea01ca89bd40584c201e WatchSource:0}: Error finding container 429cb121da4f6c69bff3e88e59b5f986dc797fb5130cea01ca89bd40584c201e: Status 404 returned error can't find the container with id 429cb121da4f6c69bff3e88e59b5f986dc797fb5130cea01ca89bd40584c201e
Nov 03 01:34:31.559643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:31.559605    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:34:31.559717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:31.559694    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:34:31.559756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:31.559738    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:34:32.239504 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:32.239473    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx" event={"ID":"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c","Type":"ContainerStarted","Data":"06212159354a4f6e7f334edb6d69fef2240eb27e429495059907fcc3aeba1ff6"}
Nov 03 01:34:32.239904 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:32.239528    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx" event={"ID":"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c","Type":"ContainerStarted","Data":"429cb121da4f6c69bff3e88e59b5f986dc797fb5130cea01ca89bd40584c201e"}
Nov 03 01:34:32.396853 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:32.396799    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx" podStartSLOduration=2.396784257 podStartE2EDuration="2.396784257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:32.365427414 +0000 UTC m=+951.791033903" watchObservedRunningTime="2026-11-03 01:34:32.396784257 +0000 UTC m=+951.822390771"
Nov 03 01:34:32.930317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:32.930279    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"]
Nov 03 01:34:32.934238 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:32.934216    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:32.985576 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:32.985546    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-016c34e32e9f\""
Nov 03 01:34:33.004003 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.003964    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-plugins-config-volume\") pod \"payload-processing-external-model-c56f7b8f9-q9st6\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") " pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:33.004195 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.004011    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials\") pod \"payload-processing-external-model-c56f7b8f9-q9st6\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") " pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:33.004195 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.004176    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj4kb\" (UniqueName: \"kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-kube-api-access-mj4kb\") pod \"payload-processing-external-model-c56f7b8f9-q9st6\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") " pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:33.004304 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.004246    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-routing-overlay-volume\") pod \"payload-processing-external-model-c56f7b8f9-q9st6\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") " pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:33.031664 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.031638    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d5931ee79a9a\""
Nov 03 01:34:33.032150 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.032129    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 01:34:33.032266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.032159    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-p8phq\""
Nov 03 01:34:33.032266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.032190    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 01:34:33.032266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.032202    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 01:34:33.032266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.032221    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 01:34:33.035088 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.035065    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"]
Nov 03 01:34:33.104986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.104953    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj4kb\" (UniqueName: \"kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-kube-api-access-mj4kb\") pod \"payload-processing-external-model-c56f7b8f9-q9st6\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") " pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:33.105186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.105005    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-routing-overlay-volume\") pod \"payload-processing-external-model-c56f7b8f9-q9st6\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") " pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:33.105186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.105065    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-plugins-config-volume\") pod \"payload-processing-external-model-c56f7b8f9-q9st6\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") " pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:33.105186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.105083    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials\") pod \"payload-processing-external-model-c56f7b8f9-q9st6\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") " pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:33.105852 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.105808    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-routing-overlay-volume\") pod \"payload-processing-external-model-c56f7b8f9-q9st6\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") " pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:33.105852 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.105846    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-plugins-config-volume\") pod \"payload-processing-external-model-c56f7b8f9-q9st6\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") " pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:33.107577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.107562    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials\") pod \"payload-processing-external-model-c56f7b8f9-q9st6\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") " pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:33.118498 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.118478    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:33.120062 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.120034    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 01:34:33.120142 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.120092    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx" podUID="9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 01:34:33.180455 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.176214    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj4kb\" (UniqueName: \"kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-kube-api-access-mj4kb\") pod \"payload-processing-external-model-c56f7b8f9-q9st6\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") " pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:33.247261 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.247232    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:33.720474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:33.720441    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"]
Nov 03 01:34:33.725601 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:34:33.725571    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0784acc9_0dd1_4ba3_86c3_c558bc8ccc40.slice/crio-720a29401f8a76c3e8ebe927c8d94d8a143a174e48405c276f44184248548e6f WatchSource:0}: Error finding container 720a29401f8a76c3e8ebe927c8d94d8a143a174e48405c276f44184248548e6f: Status 404 returned error can't find the container with id 720a29401f8a76c3e8ebe927c8d94d8a143a174e48405c276f44184248548e6f
Nov 03 01:34:34.118913 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.118837    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 01:34:34.118913 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.118888    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx" podUID="9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 01:34:34.278373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.276343    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6" event={"ID":"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40","Type":"ContainerStarted","Data":"aa19a2d3f5fa44be47f9b05697498d0dd8f6ff3495b7c77667fee4615c1d4345"}
Nov 03 01:34:34.278373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.276691    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:34.278373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.276725    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6" event={"ID":"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40","Type":"ContainerStarted","Data":"720a29401f8a76c3e8ebe927c8d94d8a143a174e48405c276f44184248548e6f"}
Nov 03 01:34:34.387571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.387470    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6" podStartSLOduration=2.387456175 podStartE2EDuration="2.387456175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:34.385711432 +0000 UTC m=+953.811317916" watchObservedRunningTime="2026-11-03 01:34:34.387456175 +0000 UTC m=+953.813062639"
Nov 03 01:34:34.475182 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.475145    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"]
Nov 03 01:34:34.479893 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.479869    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"
Nov 03 01:34:34.530414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.530377    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:34:34.530414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.530392    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:34:34.530698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.530471    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:34:34.530876 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.530853    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:34:34.531260 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.531237    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b02a3ac-ktv8h\" (UID: \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"
Nov 03 01:34:34.531371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.531320    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvzs\" (UniqueName: \"kubernetes.io/projected/4569eb3c-cd20-4d31-9567-54b06c1e80ca-kube-api-access-wcvzs\") pod \"maas-api-revoke-sub-3b02a3ac-ktv8h\" (UID: \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"
Nov 03 01:34:34.531453 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.531432    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b02a3ac-ktv8h\" (UID: \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"
Nov 03 01:34:34.533613 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.533591    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"]
Nov 03 01:34:34.632537 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.632503    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b02a3ac-ktv8h\" (UID: \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"
Nov 03 01:34:34.632708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.632572    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b02a3ac-ktv8h\" (UID: \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"
Nov 03 01:34:34.632708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.632623    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvzs\" (UniqueName: \"kubernetes.io/projected/4569eb3c-cd20-4d31-9567-54b06c1e80ca-kube-api-access-wcvzs\") pod \"maas-api-revoke-sub-3b02a3ac-ktv8h\" (UID: \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"
Nov 03 01:34:34.633170 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.633147    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b02a3ac-ktv8h\" (UID: \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"
Nov 03 01:34:34.634912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.634896    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b02a3ac-ktv8h\" (UID: \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"
Nov 03 01:34:34.692271 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.692241    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvzs\" (UniqueName: \"kubernetes.io/projected/4569eb3c-cd20-4d31-9567-54b06c1e80ca-kube-api-access-wcvzs\") pod \"maas-api-revoke-sub-3b02a3ac-ktv8h\" (UID: \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"
Nov 03 01:34:34.793080 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:34.793035    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"
Nov 03 01:34:35.118519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:35.118440    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 01:34:35.118519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:35.118489    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx" podUID="9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 01:34:35.174017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:35.173977    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"]
Nov 03 01:34:35.178337 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:34:35.178303    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4569eb3c_cd20_4d31_9567_54b06c1e80ca.slice/crio-6f1a5d382df5ed66abc61f5bd02091ec95e398f769b98e88b43a6f87c607ee12 WatchSource:0}: Error finding container 6f1a5d382df5ed66abc61f5bd02091ec95e398f769b98e88b43a6f87c607ee12: Status 404 returned error can't find the container with id 6f1a5d382df5ed66abc61f5bd02091ec95e398f769b98e88b43a6f87c607ee12
Nov 03 01:34:35.283064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:35.283032    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h" event={"ID":"4569eb3c-cd20-4d31-9567-54b06c1e80ca","Type":"ContainerStarted","Data":"6f1a5d382df5ed66abc61f5bd02091ec95e398f769b98e88b43a6f87c607ee12"}
Nov 03 01:34:36.118652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:36.118618    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 01:34:36.118804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:36.118667    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx" podUID="9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 01:34:36.290139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:36.290106    5508 generic.go:356] "Generic (PLEG): container finished" podID="4569eb3c-cd20-4d31-9567-54b06c1e80ca" containerID="5a6f7d6cb9e6793676b63c91f759026ddc20624ea610eaa0feffc9d7ed000f37" exitCode=0
Nov 03 01:34:36.290585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:36.290193    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h" event={"ID":"4569eb3c-cd20-4d31-9567-54b06c1e80ca","Type":"ContainerDied","Data":"5a6f7d6cb9e6793676b63c91f759026ddc20624ea610eaa0feffc9d7ed000f37"}
Nov 03 01:34:37.119327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:37.119296    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 01:34:37.119544 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:37.119386    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx" podUID="9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 01:34:37.486860 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:37.486829    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"
Nov 03 01:34:37.583949 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:37.583918    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4569eb3c-cd20-4d31-9567-54b06c1e80ca-kube-api-access-wcvzs\" (UniqueName: \"kubernetes.io/projected/4569eb3c-cd20-4d31-9567-54b06c1e80ca-kube-api-access-wcvzs\") pod \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\" (UID: \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\") "
Nov 03 01:34:37.584127 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:37.584050    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-xks-ca\") pod \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\" (UID: \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\") "
Nov 03 01:34:37.584127 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:37.584113    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-openshift-ca\") pod \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\" (UID: \"4569eb3c-cd20-4d31-9567-54b06c1e80ca\") "
Nov 03 01:34:37.584447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:37.584422    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-openshift-ca" pod "4569eb3c-cd20-4d31-9567-54b06c1e80ca" (UID: "4569eb3c-cd20-4d31-9567-54b06c1e80ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:37.585979 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:37.585949    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-xks-ca" pod "4569eb3c-cd20-4d31-9567-54b06c1e80ca" (UID: "4569eb3c-cd20-4d31-9567-54b06c1e80ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:34:37.586074 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:37.586019    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4569eb3c-cd20-4d31-9567-54b06c1e80ca-kube-api-access-wcvzs" pod "4569eb3c-cd20-4d31-9567-54b06c1e80ca" (UID: "4569eb3c-cd20-4d31-9567-54b06c1e80ca"). InnerVolumeSpecName "kube-api-access-wcvzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:37.685492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:37.685454    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvzs\" (UniqueName: \"kubernetes.io/projected/4569eb3c-cd20-4d31-9567-54b06c1e80ca-kube-api-access-wcvzs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:37.685492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:37.685482    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:37.685492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:37.685492    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4569eb3c-cd20-4d31-9567-54b06c1e80ca-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:34:38.118832 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:38.118799    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Nov 03 01:34:38.118979 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:38.118851    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx" podUID="9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Nov 03 01:34:38.313480 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:38.313450    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"
Nov 03 01:34:38.313674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:38.313441    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h" event={"ID":"4569eb3c-cd20-4d31-9567-54b06c1e80ca","Type":"ContainerDied","Data":"6f1a5d382df5ed66abc61f5bd02091ec95e398f769b98e88b43a6f87c607ee12"}
Nov 03 01:34:38.313674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:38.313589    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f1a5d382df5ed66abc61f5bd02091ec95e398f769b98e88b43a6f87c607ee12"
Nov 03 01:34:39.122931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:39.122903    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:39.123341 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:39.123322    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:39.123959 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:39.123940    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 01:34:43.815004 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:34:43.812172    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:34:45.286974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:45.286945    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:34:46.221151 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:46.221114    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:34:51.215990 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.215961    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"]
Nov 03 01:34:51.216399 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.216374    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4569eb3c-cd20-4d31-9567-54b06c1e80ca" containerName="revoke-keys"
Nov 03 01:34:51.216399 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.216385    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4569eb3c-cd20-4d31-9567-54b06c1e80ca" containerName="revoke-keys"
Nov 03 01:34:51.216492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.216487    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4569eb3c-cd20-4d31-9567-54b06c1e80ca" containerName="revoke-keys"
Nov 03 01:34:51.221647 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.221632    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.270730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.270710    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0bf34a-openshift-default-dockercfg-s9n2w\""
Nov 03 01:34:51.319711 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.319689    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"]
Nov 03 01:34:51.327537 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.327515    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.327640 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.327564    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.327724 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.327701    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.327795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.327777    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.327833 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.327815    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.327888 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.327873    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.327922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.327912    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cj4j\" (UniqueName: \"kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-kube-api-access-5cj4j\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.327961 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.327946    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.328070 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.328058    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.429110 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.429075    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.429110 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.429111    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.429376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.429136    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cj4j\" (UniqueName: \"kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-kube-api-access-5cj4j\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.429376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.429160    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.429376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.429242    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.429376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.429287    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.429376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.429338    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.429689 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.429418    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.429689 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.429454    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.429689 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.429591    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.429689 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.429652    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.429900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.429782    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.429972 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.429949    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.430258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.430235    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.431756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.431736    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.431845 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.431820    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.486801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.486727    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.486999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.486983    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cj4j\" (UniqueName: \"kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-kube-api-access-5cj4j\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:51.535668 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:51.535638    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:52.004083 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.004045    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"]
Nov 03 01:34:52.047753 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:34:52.047725    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9497a7_0c68_4222_b517_fc47fa1bfaf3.slice/crio-74a41bfc01fd48ad360404d4edd9b9c06f830124b93bca5aaa83858bc9b327bf WatchSource:0}: Error finding container 74a41bfc01fd48ad360404d4edd9b9c06f830124b93bca5aaa83858bc9b327bf: Status 404 returned error can't find the container with id 74a41bfc01fd48ad360404d4edd9b9c06f830124b93bca5aaa83858bc9b327bf
Nov 03 01:34:52.050032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.049999    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:34:52.050148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.050079    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:34:52.050148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.050124    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:34:52.408710 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.408677    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg" event={"ID":"af9497a7-0c68-4222-b517-fc47fa1bfaf3","Type":"ContainerStarted","Data":"e50578398abbd4b0193def7b21b69b9864b00f55efba4a2907661308a4032ccf"}
Nov 03 01:34:52.409054 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.408731    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg" event={"ID":"af9497a7-0c68-4222-b517-fc47fa1bfaf3","Type":"ContainerStarted","Data":"74a41bfc01fd48ad360404d4edd9b9c06f830124b93bca5aaa83858bc9b327bf"}
Nov 03 01:34:52.523364 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.522957    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg" podStartSLOduration=1.522918615 podStartE2EDuration="1.522918615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:52.522480322 +0000 UTC m=+971.948086808" watchObservedRunningTime="2026-11-03 01:34:52.522918615 +0000 UTC m=+971.948525079"
Nov 03 01:34:52.536617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.536583    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:52.721092 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.721011    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"]
Nov 03 01:34:52.724692 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.724670    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:52.779554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.779522    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"]
Nov 03 01:34:52.845305 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.845266    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-plugins-config-volume\") pod \"payload-processing-external-model-c784f7df-nvj4w\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") " pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:52.845540 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.845496    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-routing-overlay-volume\") pod \"payload-processing-external-model-c784f7df-nvj4w\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") " pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:52.845620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.845545    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7z8x\" (UniqueName: \"kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-kube-api-access-x7z8x\") pod \"payload-processing-external-model-c784f7df-nvj4w\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") " pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:52.845620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.845596    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-provider-credentials\") pod \"payload-processing-external-model-c784f7df-nvj4w\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") " pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:52.947093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.947057    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-routing-overlay-volume\") pod \"payload-processing-external-model-c784f7df-nvj4w\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") " pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:52.947274 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.947147    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7z8x\" (UniqueName: \"kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-kube-api-access-x7z8x\") pod \"payload-processing-external-model-c784f7df-nvj4w\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") " pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:52.947274 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.947169    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-provider-credentials\") pod \"payload-processing-external-model-c784f7df-nvj4w\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") " pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:52.947274 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.947229    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-plugins-config-volume\") pod \"payload-processing-external-model-c784f7df-nvj4w\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") " pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:52.947765 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.947743    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-routing-overlay-volume\") pod \"payload-processing-external-model-c784f7df-nvj4w\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") " pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:52.947947 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.947925    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-plugins-config-volume\") pod \"payload-processing-external-model-c784f7df-nvj4w\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") " pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:52.975511 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:52.975435    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-provider-credentials\") pod \"payload-processing-external-model-c784f7df-nvj4w\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") " pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:53.004454 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:53.004430    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7z8x\" (UniqueName: \"kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-kube-api-access-x7z8x\") pod \"payload-processing-external-model-c784f7df-nvj4w\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") " pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:53.037314 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:53.037289    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:53.438169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:53.438138    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"]
Nov 03 01:34:53.444743 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:34:53.444714    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9c79d5_42f4_415e_b966_09996472bd0d.slice/crio-4107a6da4a2974c09b43512f50a338d1b47c2540825c69c19a1fc8c287e0c978 WatchSource:0}: Error finding container 4107a6da4a2974c09b43512f50a338d1b47c2540825c69c19a1fc8c287e0c978: Status 404 returned error can't find the container with id 4107a6da4a2974c09b43512f50a338d1b47c2540825c69c19a1fc8c287e0c978
Nov 03 01:34:53.537937 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:53.537909    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 01:34:53.538042 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:53.537956    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg" podUID="af9497a7-0c68-4222-b517-fc47fa1bfaf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 01:34:54.422675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:54.422644    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w" event={"ID":"9f9c79d5-42f4-415e-b966-09996472bd0d","Type":"ContainerStarted","Data":"a82fc260231de1b9da232ee8bd0473cbf1cb8166d5552a385769bb0dcf6ca922"}
Nov 03 01:34:54.422862 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:54.422694    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w" event={"ID":"9f9c79d5-42f4-415e-b966-09996472bd0d","Type":"ContainerStarted","Data":"4107a6da4a2974c09b43512f50a338d1b47c2540825c69c19a1fc8c287e0c978"}
Nov 03 01:34:54.422981 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:54.422961    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:34:54.536252 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:54.536212    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 01:34:54.536715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:54.536280    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg" podUID="af9497a7-0c68-4222-b517-fc47fa1bfaf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 01:34:54.539053 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:54.539005    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w" podStartSLOduration=2.538991149 podStartE2EDuration="2.538991149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:54.535227205 +0000 UTC m=+973.960833681" watchObservedRunningTime="2026-11-03 01:34:54.538991149 +0000 UTC m=+973.964597632"
Nov 03 01:34:55.536338 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:55.536304    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 01:34:55.536741 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:55.536371    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg" podUID="af9497a7-0c68-4222-b517-fc47fa1bfaf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 01:34:56.536794 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:56.536761    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 01:34:56.537191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:56.536823    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg" podUID="af9497a7-0c68-4222-b517-fc47fa1bfaf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 01:34:57.536575 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:57.536537    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 01:34:57.536878 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:57.536598    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg" podUID="af9497a7-0c68-4222-b517-fc47fa1bfaf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 01:34:59.520935 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:59.520905    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:59.521483 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:59.521447    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:34:59.522269 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:34:59.522252    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:35:00.875544 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:00.875501    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"]
Nov 03 01:35:00.879646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:00.879628    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"
Nov 03 01:35:00.976443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:00.976407    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:35:00.976674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:00.976453    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:35:00.976674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:00.976591    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:35:00.976674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:00.976603    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:35:00.979218 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:00.979195    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp5wj\" (UniqueName: \"kubernetes.io/projected/a7283494-829e-41e5-a67d-99bee0c7271e-kube-api-access-rp5wj\") pod \"maas-api-revoke-sub-5ccfebe6-2j7bs\" (UID: \"a7283494-829e-41e5-a67d-99bee0c7271e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"
Nov 03 01:35:00.979396 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:00.979369    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ccfebe6-2j7bs\" (UID: \"a7283494-829e-41e5-a67d-99bee0c7271e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"
Nov 03 01:35:00.979396 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:00.979385    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"]
Nov 03 01:35:00.979489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:00.979460    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ccfebe6-2j7bs\" (UID: \"a7283494-829e-41e5-a67d-99bee0c7271e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"
Nov 03 01:35:01.080573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:01.080540    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp5wj\" (UniqueName: \"kubernetes.io/projected/a7283494-829e-41e5-a67d-99bee0c7271e-kube-api-access-rp5wj\") pod \"maas-api-revoke-sub-5ccfebe6-2j7bs\" (UID: \"a7283494-829e-41e5-a67d-99bee0c7271e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"
Nov 03 01:35:01.080749 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:01.080658    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ccfebe6-2j7bs\" (UID: \"a7283494-829e-41e5-a67d-99bee0c7271e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"
Nov 03 01:35:01.080842 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:01.080821    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ccfebe6-2j7bs\" (UID: \"a7283494-829e-41e5-a67d-99bee0c7271e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"
Nov 03 01:35:01.081292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:01.081273    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ccfebe6-2j7bs\" (UID: \"a7283494-829e-41e5-a67d-99bee0c7271e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"
Nov 03 01:35:01.083160 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:01.083143    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ccfebe6-2j7bs\" (UID: \"a7283494-829e-41e5-a67d-99bee0c7271e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"
Nov 03 01:35:01.138408 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:01.138309    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp5wj\" (UniqueName: \"kubernetes.io/projected/a7283494-829e-41e5-a67d-99bee0c7271e-kube-api-access-rp5wj\") pod \"maas-api-revoke-sub-5ccfebe6-2j7bs\" (UID: \"a7283494-829e-41e5-a67d-99bee0c7271e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"
Nov 03 01:35:01.192658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:01.192628    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"
Nov 03 01:35:01.567413 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:01.567375    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"]
Nov 03 01:35:01.574683 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:35:01.574652    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7283494_829e_41e5_a67d_99bee0c7271e.slice/crio-159d7b968344d3844fce6c37d782cfd9062b44537cdd8b52758e12ff6b2f28a7 WatchSource:0}: Error finding container 159d7b968344d3844fce6c37d782cfd9062b44537cdd8b52758e12ff6b2f28a7: Status 404 returned error can't find the container with id 159d7b968344d3844fce6c37d782cfd9062b44537cdd8b52758e12ff6b2f28a7
Nov 03 01:35:02.485754 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:02.485721    5508 generic.go:356] "Generic (PLEG): container finished" podID="a7283494-829e-41e5-a67d-99bee0c7271e" containerID="3e725af250f317c38645e9d9deb5ff9495d390366210c5d7b8ed0faf5d040a76" exitCode=0
Nov 03 01:35:02.486158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:02.485810    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs" event={"ID":"a7283494-829e-41e5-a67d-99bee0c7271e","Type":"ContainerDied","Data":"3e725af250f317c38645e9d9deb5ff9495d390366210c5d7b8ed0faf5d040a76"}
Nov 03 01:35:02.486158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:02.485869    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs" event={"ID":"a7283494-829e-41e5-a67d-99bee0c7271e","Type":"ContainerStarted","Data":"159d7b968344d3844fce6c37d782cfd9062b44537cdd8b52758e12ff6b2f28a7"}
Nov 03 01:35:03.640821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:03.640794    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"
Nov 03 01:35:03.724135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:03.724091    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-xks-ca\") pod \"a7283494-829e-41e5-a67d-99bee0c7271e\" (UID: \"a7283494-829e-41e5-a67d-99bee0c7271e\") "
Nov 03 01:35:03.724334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:03.724248    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-openshift-ca\") pod \"a7283494-829e-41e5-a67d-99bee0c7271e\" (UID: \"a7283494-829e-41e5-a67d-99bee0c7271e\") "
Nov 03 01:35:03.724334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:03.724327    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7283494-829e-41e5-a67d-99bee0c7271e-kube-api-access-rp5wj\" (UniqueName: \"kubernetes.io/projected/a7283494-829e-41e5-a67d-99bee0c7271e-kube-api-access-rp5wj\") pod \"a7283494-829e-41e5-a67d-99bee0c7271e\" (UID: \"a7283494-829e-41e5-a67d-99bee0c7271e\") "
Nov 03 01:35:03.724684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:03.724654    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-openshift-ca" pod "a7283494-829e-41e5-a67d-99bee0c7271e" (UID: "a7283494-829e-41e5-a67d-99bee0c7271e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:35:03.726414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:03.726383    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-xks-ca" pod "a7283494-829e-41e5-a67d-99bee0c7271e" (UID: "a7283494-829e-41e5-a67d-99bee0c7271e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:35:03.726596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:03.726558    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7283494-829e-41e5-a67d-99bee0c7271e-kube-api-access-rp5wj" pod "a7283494-829e-41e5-a67d-99bee0c7271e" (UID: "a7283494-829e-41e5-a67d-99bee0c7271e"). InnerVolumeSpecName "kube-api-access-rp5wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:35:03.825538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:03.825457    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:03.825538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:03.825484    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp5wj\" (UniqueName: \"kubernetes.io/projected/a7283494-829e-41e5-a67d-99bee0c7271e-kube-api-access-rp5wj\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:03.825538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:03.825495    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7283494-829e-41e5-a67d-99bee0c7271e-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:04.122610 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.122521    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"]
Nov 03 01:35:04.122958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.122946    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a7283494-829e-41e5-a67d-99bee0c7271e" containerName="revoke-keys"
Nov 03 01:35:04.123004 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.122960    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7283494-829e-41e5-a67d-99bee0c7271e" containerName="revoke-keys"
Nov 03 01:35:04.123066 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.123057    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7283494-829e-41e5-a67d-99bee0c7271e" containerName="revoke-keys"
Nov 03 01:35:04.126500 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.126481    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.143777 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.143752    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"]
Nov 03 01:35:04.147575 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.147557    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.169024 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.168996    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"]
Nov 03 01:35:04.172730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.172713    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.223996 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.223962    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"]
Nov 03 01:35:04.229765 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.229724    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.229958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.229855    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.229958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.229901    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.230078 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.229995    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.230078 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230065    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.230180 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230093    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f420022-cf2a-44db-a72b-2934f355375e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.230180 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230122    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.230288 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230186    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.230288 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230215    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbwg\" (UniqueName: \"kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-kube-api-access-qvbwg\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.230288 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230249    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.230461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230289    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.230461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230412    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nthh\" (UniqueName: \"kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-kube-api-access-6nthh\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.230461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230443    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7zmf\" (UniqueName: \"kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-kube-api-access-l7zmf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.230630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230479    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.230630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230517    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.230630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230541    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801be76b-b7a4-468b-b72e-b6790c0bef0b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.230630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230571    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.230630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230609    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.230873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230635    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.230873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230666    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.230873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230688    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.230873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230745    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.230873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230779    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.230873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230804    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.230873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230857    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.231211 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230879    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f420022-cf2a-44db-a72b-2934f355375e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.231211 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.230900    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.269572 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.269522    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"]
Nov 03 01:35:04.320656 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.320621    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"]
Nov 03 01:35:04.331730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.331699    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nthh\" (UniqueName: \"kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-kube-api-access-6nthh\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.331899 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.331752    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7zmf\" (UniqueName: \"kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-kube-api-access-l7zmf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.331899 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.331799    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.331899 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.331839    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.331899 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.331863    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801be76b-b7a4-468b-b72e-b6790c0bef0b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.331899 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.331889    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.332165 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.331915    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.332303 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332283    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.332402 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332380    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.332450 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332421    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.332450 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332433    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.332515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332494    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.332547 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332523    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.332594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332577    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801be76b-b7a4-468b-b72e-b6790c0bef0b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.332640 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332582    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.332640 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332624    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.332739 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332649    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.332739 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332690    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.332739 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332712    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f420022-cf2a-44db-a72b-2934f355375e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.332739 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332735    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.332973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332776    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.332973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332816    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.332973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332832    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.332973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332838    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.332973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332887    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.332973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332927    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.332973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332961    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.333286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.332991    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f420022-cf2a-44db-a72b-2934f355375e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.333286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.333023    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.333286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.333057    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.333286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.333063    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.333286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.333085    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbwg\" (UniqueName: \"kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-kube-api-access-qvbwg\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.333286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.333137    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.333286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.333164    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.333286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.333213    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.333770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.333582    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.334068 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.334039    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.334178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.334113    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.334380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.334325    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.334618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.334595    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.335122 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.335077    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.335122 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.335111    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f420022-cf2a-44db-a72b-2934f355375e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.335283 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.335130    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.335633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.335610    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.335978 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.335958    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.336920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.336892    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.337022 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.336972    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.337577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.337557    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f420022-cf2a-44db-a72b-2934f355375e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.390147 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.389653    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.390147 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.390041    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.390441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.390217    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nthh\" (UniqueName: \"kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-kube-api-access-6nthh\") pod \"maas-default-gateway-openshift-default-59f957fdd7-wj656\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.390871 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.390846    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7zmf\" (UniqueName: \"kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-kube-api-access-l7zmf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.390935 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.390921    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-kmx46\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.403476 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.403444    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbwg\" (UniqueName: \"kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-kube-api-access-qvbwg\") pod \"maas-default-gateway-openshift-default-59f957fdd7-knxsx\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.441073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.441041    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:04.459594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.459557    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:04.485200 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.485171    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:04.506550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.504541    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs" event={"ID":"a7283494-829e-41e5-a67d-99bee0c7271e","Type":"ContainerDied","Data":"159d7b968344d3844fce6c37d782cfd9062b44537cdd8b52758e12ff6b2f28a7"}
Nov 03 01:35:04.506550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.504604    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="159d7b968344d3844fce6c37d782cfd9062b44537cdd8b52758e12ff6b2f28a7"
Nov 03 01:35:04.506550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.504683    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"
Nov 03 01:35:04.880782 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.880731    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"]
Nov 03 01:35:04.885787 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.885756    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"]
Nov 03 01:35:04.885926 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:35:04.885897    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801be76b_b7a4_468b_b72e_b6790c0bef0b.slice/crio-d8ab56a98657d0bc6a282b0474c412d17dd63be475d38ba59577e6f5539b0219 WatchSource:0}: Error finding container d8ab56a98657d0bc6a282b0474c412d17dd63be475d38ba59577e6f5539b0219: Status 404 returned error can't find the container with id d8ab56a98657d0bc6a282b0474c412d17dd63be475d38ba59577e6f5539b0219
Nov 03 01:35:04.889915 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:35:04.889887    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f420022_cf2a_44db_a72b_2934f355375e.slice/crio-c7a52e2edfb2b5976bd64273b08510afb5b0130f30ec7d708553c7ae91e01ae8 WatchSource:0}: Error finding container c7a52e2edfb2b5976bd64273b08510afb5b0130f30ec7d708553c7ae91e01ae8: Status 404 returned error can't find the container with id c7a52e2edfb2b5976bd64273b08510afb5b0130f30ec7d708553c7ae91e01ae8
Nov 03 01:35:04.890460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.890413    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:35:04.890565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.890531    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:35:04.890644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.890604    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:35:04.893389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.893310    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:35:04.893544 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.893517    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:35:04.893649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.893631    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:35:04.934484 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.934453    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"]
Nov 03 01:35:04.946636 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:35:04.946608    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c8b2c7_6e14_42dd_a9e5_457550da27ff.slice/crio-6c8da091d14e110e4f2bdd573cf47d3b1c9a683709160bbdd2c21f9a52b93205 WatchSource:0}: Error finding container 6c8da091d14e110e4f2bdd573cf47d3b1c9a683709160bbdd2c21f9a52b93205: Status 404 returned error can't find the container with id 6c8da091d14e110e4f2bdd573cf47d3b1c9a683709160bbdd2c21f9a52b93205
Nov 03 01:35:04.949705 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.949663    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:35:04.949804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.949760    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:35:04.949870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:04.949806    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:35:05.431895 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:05.431854    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w" podUID="9f9c79d5-42f4-415e-b966-09996472bd0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:35:05.512600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:05.512555    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" event={"ID":"5f420022-cf2a-44db-a72b-2934f355375e","Type":"ContainerStarted","Data":"ce85b59582911c6192c85b5cd341087b3392f46f34afec47b956b073a4276042"}
Nov 03 01:35:05.512859 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:05.512619    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" event={"ID":"5f420022-cf2a-44db-a72b-2934f355375e","Type":"ContainerStarted","Data":"c7a52e2edfb2b5976bd64273b08510afb5b0130f30ec7d708553c7ae91e01ae8"}
Nov 03 01:35:05.514412 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:05.514374    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" event={"ID":"801be76b-b7a4-468b-b72e-b6790c0bef0b","Type":"ContainerStarted","Data":"d11e7b0fdcc95a5ea3eacf901838f00f4e2658808823fb60db1e7c5aa9707f03"}
Nov 03 01:35:05.514547 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:05.514436    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" event={"ID":"801be76b-b7a4-468b-b72e-b6790c0bef0b","Type":"ContainerStarted","Data":"d8ab56a98657d0bc6a282b0474c412d17dd63be475d38ba59577e6f5539b0219"}
Nov 03 01:35:05.516017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:05.515989    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" event={"ID":"33c8b2c7-6e14-42dd-a9e5-457550da27ff","Type":"ContainerStarted","Data":"ff1f253cf2f0176e052dc612a1a185b550457ce5cc1f822e386f13f04c459a67"}
Nov 03 01:35:05.516100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:05.516033    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" event={"ID":"33c8b2c7-6e14-42dd-a9e5-457550da27ff","Type":"ContainerStarted","Data":"6c8da091d14e110e4f2bdd573cf47d3b1c9a683709160bbdd2c21f9a52b93205"}
Nov 03 01:35:05.627019 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:05.626966    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" podStartSLOduration=1.626952067 podStartE2EDuration="1.626952067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:05.625471044 +0000 UTC m=+985.051077529" watchObservedRunningTime="2026-11-03 01:35:05.626952067 +0000 UTC m=+985.052558530"
Nov 03 01:35:05.890430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:05.876996    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" podStartSLOduration=1.876972044 podStartE2EDuration="1.876972044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:05.868867615 +0000 UTC m=+985.294474114" watchObservedRunningTime="2026-11-03 01:35:05.876972044 +0000 UTC m=+985.302578509"
Nov 03 01:35:05.890430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:05.877170    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" podStartSLOduration=1.877159417 podStartE2EDuration="1.877159417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:05.749696748 +0000 UTC m=+985.175303215" watchObservedRunningTime="2026-11-03 01:35:05.877159417 +0000 UTC m=+985.302765880"
Nov 03 01:35:06.450369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:06.445713    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:06.450369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:06.449560    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-kmx46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 01:35:06.450369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:06.449618    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 01:35:06.471469 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:06.469553    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:06.476368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:06.473492    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wj656 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 01:35:06.476368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:06.473553    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" podUID="5f420022-cf2a-44db-a72b-2934f355375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 01:35:06.503403 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:06.499779    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:06.507370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:06.504465    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-knxsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 01:35:06.507370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:06.504520    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 01:35:07.441711 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:07.441679    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-kmx46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 01:35:07.442185 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:07.441734    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 01:35:07.469373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:07.461789    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wj656 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 01:35:07.469373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:07.461854    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" podUID="5f420022-cf2a-44db-a72b-2934f355375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 01:35:07.489391 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:07.485799    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-knxsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 01:35:07.489391 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:07.485861    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 01:35:08.443797 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:08.443749    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-kmx46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 01:35:08.444282 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:08.443812    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 01:35:08.465389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:08.463708    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wj656 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 01:35:08.465389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:08.463767    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" podUID="5f420022-cf2a-44db-a72b-2934f355375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 01:35:08.488371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:08.487762    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-knxsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 01:35:08.488371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:08.487825    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 01:35:09.446371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:09.443743    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-kmx46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 01:35:09.446371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:09.443805    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 01:35:09.469389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:09.466759    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wj656 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 01:35:09.469389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:09.466823    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" podUID="5f420022-cf2a-44db-a72b-2934f355375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 01:35:09.496375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:09.495454    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-knxsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 01:35:09.496375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:09.495516    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 01:35:10.442087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:10.442040    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-kmx46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 01:35:10.442277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:10.442115    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 01:35:10.461390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:10.460776    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wj656 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 01:35:10.461390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:10.460838    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" podUID="5f420022-cf2a-44db-a72b-2934f355375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 01:35:10.493388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:10.492775    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-knxsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 01:35:10.493388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:10.492836    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 01:35:11.441578 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:11.441543    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-kmx46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 01:35:11.441754 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:11.441602    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 01:35:11.460173 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:11.460132    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wj656 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 01:35:11.460507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:11.460479    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" podUID="5f420022-cf2a-44db-a72b-2934f355375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 01:35:11.486084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:11.486047    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-knxsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 01:35:11.486547 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:11.486105    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 01:35:12.441981 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:12.441947    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-kmx46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 01:35:12.442162 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:12.442019    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 01:35:12.460302 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:12.460276    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wj656 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 01:35:12.460441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:12.460327    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" podUID="5f420022-cf2a-44db-a72b-2934f355375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 01:35:12.486239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:12.486208    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-knxsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 01:35:12.486610 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:12.486262    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 01:35:13.875522 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:13.875426    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:35:14.448611 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.448520    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-kmx46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:35:14.448611 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.448591    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:35:14.460553 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.460510    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wj656 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:35:14.460738 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.460571    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" podUID="5f420022-cf2a-44db-a72b-2934f355375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:35:14.486291 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.486249    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-knxsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:35:14.486515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.486325    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:35:14.524272 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.524239    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:14.525910 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.525888    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:14.527332 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.527312    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:35:14.559432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.559400    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:14.559871 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.559842    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:14.560702 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.560686    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:14.561358 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.561312    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:14.562804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.562781    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:35:14.562925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:14.562842    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:35:15.429478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:15.429445    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w" podUID="9f9c79d5-42f4-415e-b966-09996472bd0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:35:20.408959 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:20.408918    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:35:25.429342 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:25.429300    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w" podUID="9f9c79d5-42f4-415e-b966-09996472bd0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:35:35.429526 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:35.429491    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w" podUID="9f9c79d5-42f4-415e-b966-09996472bd0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:35:41.200340 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.200307    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"]
Nov 03 01:35:41.201670 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.201644    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w" podUID="9f9c79d5-42f4-415e-b966-09996472bd0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:35:41.236476 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.236450    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"]
Nov 03 01:35:41.241461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.241443    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.305283 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.305254    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"]
Nov 03 01:35:41.319574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.319540    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-routing-overlay-volume\") pod \"payload-processing-external-model-88bf9945c-nqsfh\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.319742 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.319601    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-plugins-config-volume\") pod \"payload-processing-external-model-88bf9945c-nqsfh\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.319823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.319798    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-295zc\" (UniqueName: \"kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-kube-api-access-295zc\") pod \"payload-processing-external-model-88bf9945c-nqsfh\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.319876 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.319849    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-provider-credentials\") pod \"payload-processing-external-model-88bf9945c-nqsfh\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.420787 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.420750    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-routing-overlay-volume\") pod \"payload-processing-external-model-88bf9945c-nqsfh\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.420937 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.420819    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-plugins-config-volume\") pod \"payload-processing-external-model-88bf9945c-nqsfh\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.420937 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.420896    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-295zc\" (UniqueName: \"kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-kube-api-access-295zc\") pod \"payload-processing-external-model-88bf9945c-nqsfh\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.420937 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.420929    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-provider-credentials\") pod \"payload-processing-external-model-88bf9945c-nqsfh\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.421515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.421493    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-routing-overlay-volume\") pod \"payload-processing-external-model-88bf9945c-nqsfh\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.421608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.421590    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-plugins-config-volume\") pod \"payload-processing-external-model-88bf9945c-nqsfh\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.423498 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.423480    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-provider-credentials\") pod \"payload-processing-external-model-88bf9945c-nqsfh\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.477900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.477851    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-295zc\" (UniqueName: \"kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-kube-api-access-295zc\") pod \"payload-processing-external-model-88bf9945c-nqsfh\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.554295 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.554272    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:41.936056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:41.936024    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"]
Nov 03 01:35:41.942720 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:35:41.942697    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0e6967_fe5d_4c57_87fa_0b284effe171.slice/crio-e39c37bad309a0d4dc8a0f5d52e7021fd36680b14096f842d8883945cc98e095 WatchSource:0}: Error finding container e39c37bad309a0d4dc8a0f5d52e7021fd36680b14096f842d8883945cc98e095: Status 404 returned error can't find the container with id e39c37bad309a0d4dc8a0f5d52e7021fd36680b14096f842d8883945cc98e095
Nov 03 01:35:42.855674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:42.855629    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh" event={"ID":"cd0e6967-fe5d-4c57-87fa-0b284effe171","Type":"ContainerStarted","Data":"f8e806d0f13a9b56c0b3f8be2f41cdfa853da6a31213f478f7d3017aa9c97fe1"}
Nov 03 01:35:42.856078 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:42.855684    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh" event={"ID":"cd0e6967-fe5d-4c57-87fa-0b284effe171","Type":"ContainerStarted","Data":"e39c37bad309a0d4dc8a0f5d52e7021fd36680b14096f842d8883945cc98e095"}
Nov 03 01:35:42.856078 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:42.855836    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:35:42.966036 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:42.965971    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh" podStartSLOduration=1.965954482 podStartE2EDuration="1.965954482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:42.964961552 +0000 UTC m=+1022.390568027" watchObservedRunningTime="2026-11-03 01:35:42.965954482 +0000 UTC m=+1022.391560947"
Nov 03 01:35:43.810406 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:43.810373    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:35:46.205373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:46.204402    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w" podUID="9f9c79d5-42f4-415e-b966-09996472bd0d" containerName="payload-processing" containerID="cri-o://a82fc260231de1b9da232ee8bd0473cbf1cb8166d5552a385769bb0dcf6ca922" gracePeriod=25
Nov 03 01:35:47.180585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.180557    5508 generic.go:356] "Generic (PLEG): container finished" podID="9f9c79d5-42f4-415e-b966-09996472bd0d" containerID="a82fc260231de1b9da232ee8bd0473cbf1cb8166d5552a385769bb0dcf6ca922" exitCode=0
Nov 03 01:35:47.180931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.180904    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w" event={"ID":"9f9c79d5-42f4-415e-b966-09996472bd0d","Type":"ContainerDied","Data":"a82fc260231de1b9da232ee8bd0473cbf1cb8166d5552a385769bb0dcf6ca922"}
Nov 03 01:35:47.211745 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.211722    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:35:47.323887 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.323795    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-provider-credentials\") pod \"9f9c79d5-42f4-415e-b966-09996472bd0d\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") "
Nov 03 01:35:47.324159 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.324120    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-routing-overlay-volume\") pod \"9f9c79d5-42f4-415e-b966-09996472bd0d\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") "
Nov 03 01:35:47.324429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.324410    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-plugins-config-volume\") pod \"9f9c79d5-42f4-415e-b966-09996472bd0d\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") "
Nov 03 01:35:47.324706 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.324676    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-kube-api-access-x7z8x\" (UniqueName: \"kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-kube-api-access-x7z8x\") pod \"9f9c79d5-42f4-415e-b966-09996472bd0d\" (UID: \"9f9c79d5-42f4-415e-b966-09996472bd0d\") "
Nov 03 01:35:47.332956 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.332926    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-routing-overlay-volume" pod "9f9c79d5-42f4-415e-b966-09996472bd0d" (UID: "9f9c79d5-42f4-415e-b966-09996472bd0d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:35:47.333198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.333181    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-plugins-config-volume" pod "9f9c79d5-42f4-415e-b966-09996472bd0d" (UID: "9f9c79d5-42f4-415e-b966-09996472bd0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:35:47.344154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.343893    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-kube-api-access-x7z8x" pod "9f9c79d5-42f4-415e-b966-09996472bd0d" (UID: "9f9c79d5-42f4-415e-b966-09996472bd0d"). InnerVolumeSpecName "kube-api-access-x7z8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:35:47.361487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.361454    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-provider-credentials" pod "9f9c79d5-42f4-415e-b966-09996472bd0d" (UID: "9f9c79d5-42f4-415e-b966-09996472bd0d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:35:47.529377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.527780    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7z8x\" (UniqueName: \"kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-kube-api-access-x7z8x\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:47.529377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.527810    5508 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f9c79d5-42f4-415e-b966-09996472bd0d-provider-credentials\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:47.529377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.527825    5508 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-routing-overlay-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:47.529377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:47.527838    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9c79d5-42f4-415e-b966-09996472bd0d-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:35:47.738851 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:47.738817    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d5931ee79a9a: secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:47.739072 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:47.739056    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6: secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:47.739236 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:47.739222    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials podName:0784acc9-0dd1-4ba3-86c3-c558bc8ccc40 nodeName:}" failed. No retries permitted until 2026-11-03 01:35:48.239200803 +0000 UTC m=+1027.664807258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials") pod "payload-processing-external-model-c56f7b8f9-q9st6" (UID: "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40") : secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:48.189123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:48.189100    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"
Nov 03 01:35:48.189296 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:48.189093    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c784f7df-nvj4w" event={"ID":"9f9c79d5-42f4-415e-b966-09996472bd0d","Type":"ContainerDied","Data":"4107a6da4a2974c09b43512f50a338d1b47c2540825c69c19a1fc8c287e0c978"}
Nov 03 01:35:48.189296 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:48.189240    5508 scope.go:122] "RemoveContainer" containerID="a82fc260231de1b9da232ee8bd0473cbf1cb8166d5552a385769bb0dcf6ca922"
Nov 03 01:35:48.242669 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:48.242647    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d5931ee79a9a: secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:48.242669 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:48.242667    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6: secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:48.243041 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:48.242713    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials podName:0784acc9-0dd1-4ba3-86c3-c558bc8ccc40 nodeName:}" failed. No retries permitted until 2026-11-03 01:35:49.242696705 +0000 UTC m=+1028.668303164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials") pod "payload-processing-external-model-c56f7b8f9-q9st6" (UID: "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40") : secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:48.343189 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:48.343141    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"]
Nov 03 01:35:48.351331 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:48.351302    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-c784f7df-nvj4w"]
Nov 03 01:35:48.691239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:48.691205    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f9c79d5-42f4-415e-b966-09996472bd0d" path="/var/lib/kubelet/pods/9f9c79d5-42f4-415e-b966-09996472bd0d/volumes"
Nov 03 01:35:49.252874 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:49.252839    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d5931ee79a9a: secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:49.252874 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:49.252869    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6: secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:49.253413 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:49.252945    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials podName:0784acc9-0dd1-4ba3-86c3-c558bc8ccc40 nodeName:}" failed. No retries permitted until 2026-11-03 01:35:51.252924356 +0000 UTC m=+1030.678530809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials") pod "payload-processing-external-model-c56f7b8f9-q9st6" (UID: "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40") : secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:51.279295 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:51.279258    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d5931ee79a9a: secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:51.279843 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:51.279826    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6: secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:51.280020 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:51.280008    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials podName:0784acc9-0dd1-4ba3-86c3-c558bc8ccc40 nodeName:}" failed. No retries permitted until 2026-11-03 01:35:55.279984596 +0000 UTC m=+1034.705591051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials") pod "payload-processing-external-model-c56f7b8f9-q9st6" (UID: "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40") : secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:53.864936 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:53.864904    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh" podUID="cd0e6967-fe5d-4c57-87fa-0b284effe171" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:35:55.345870 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:55.345839    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d5931ee79a9a: secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:55.345870 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:55.345862    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6: secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:55.346278 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:35:55.345912    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials podName:0784acc9-0dd1-4ba3-86c3-c558bc8ccc40 nodeName:}" failed. No retries permitted until 2026-11-03 01:36:03.345897598 +0000 UTC m=+1042.771504060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials") pod "payload-processing-external-model-c56f7b8f9-q9st6" (UID: "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40") : secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:35:55.374006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.373975    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"]
Nov 03 01:35:55.375660 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.375592    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh" podUID="cd0e6967-fe5d-4c57-87fa-0b284effe171" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:35:55.419778 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.419749    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"]
Nov 03 01:35:55.420199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.420188    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9f9c79d5-42f4-415e-b966-09996472bd0d" containerName="payload-processing"
Nov 03 01:35:55.420246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.420200    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f9c79d5-42f4-415e-b966-09996472bd0d" containerName="payload-processing"
Nov 03 01:35:55.420292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.420284    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f9c79d5-42f4-415e-b966-09996472bd0d" containerName="payload-processing"
Nov 03 01:35:55.423562 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.423544    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:55.464056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.464023    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"]
Nov 03 01:35:55.472320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.472294    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e0c8f512f861\""
Nov 03 01:35:55.472468 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.472392    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4f993d3b5832\""
Nov 03 01:35:55.547949 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.547919    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-routing-overlay-volume\") pod \"payload-processing-external-model-694f985c7-bnq2x\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") " pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:55.548115 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.548005    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-provider-credentials\") pod \"payload-processing-external-model-694f985c7-bnq2x\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") " pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:55.548115 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.548109    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ld65\" (UniqueName: \"kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-kube-api-access-8ld65\") pod \"payload-processing-external-model-694f985c7-bnq2x\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") " pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:55.548274 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.548146    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-plugins-config-volume\") pod \"payload-processing-external-model-694f985c7-bnq2x\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") " pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:55.649177 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.649146    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-plugins-config-volume\") pod \"payload-processing-external-model-694f985c7-bnq2x\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") " pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:55.649338 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.649273    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-routing-overlay-volume\") pod \"payload-processing-external-model-694f985c7-bnq2x\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") " pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:55.649338 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.649329    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-provider-credentials\") pod \"payload-processing-external-model-694f985c7-bnq2x\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") " pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:55.649513 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.649497    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ld65\" (UniqueName: \"kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-kube-api-access-8ld65\") pod \"payload-processing-external-model-694f985c7-bnq2x\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") " pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:55.650085 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.650057    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-routing-overlay-volume\") pod \"payload-processing-external-model-694f985c7-bnq2x\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") " pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:55.650209 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.650087    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-plugins-config-volume\") pod \"payload-processing-external-model-694f985c7-bnq2x\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") " pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:55.652012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.651982    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-provider-credentials\") pod \"payload-processing-external-model-694f985c7-bnq2x\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") " pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:55.706736 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.706707    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ld65\" (UniqueName: \"kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-kube-api-access-8ld65\") pod \"payload-processing-external-model-694f985c7-bnq2x\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") " pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:55.736211 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:55.736182    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:56.115546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:56.115472    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"]
Nov 03 01:35:56.121919 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:35:56.121894    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d0256b_82df_4484_b5a7_e077303869ac.slice/crio-4edda2fe8f41c8dcbd0343f022d42e15a2e1439fab99490e1a38e4e444b6762e WatchSource:0}: Error finding container 4edda2fe8f41c8dcbd0343f022d42e15a2e1439fab99490e1a38e4e444b6762e: Status 404 returned error can't find the container with id 4edda2fe8f41c8dcbd0343f022d42e15a2e1439fab99490e1a38e4e444b6762e
Nov 03 01:35:56.275553 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:56.275517    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x" event={"ID":"88d0256b-82df-4484-b5a7-e077303869ac","Type":"ContainerStarted","Data":"5b63cb24a38890a63866d786de9331e8927e0bf7acc9931aeafa29d0464360b7"}
Nov 03 01:35:56.275766 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:56.275580    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x" event={"ID":"88d0256b-82df-4484-b5a7-e077303869ac","Type":"ContainerStarted","Data":"4edda2fe8f41c8dcbd0343f022d42e15a2e1439fab99490e1a38e4e444b6762e"}
Nov 03 01:35:56.275766 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:56.275707    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:35:56.385931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:56.385825    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x" podStartSLOduration=1.385807792 podStartE2EDuration="1.385807792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:56.384193707 +0000 UTC m=+1035.809800174" watchObservedRunningTime="2026-11-03 01:35:56.385807792 +0000 UTC m=+1035.811414259"
Nov 03 01:35:57.188791 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:35:57.188744    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:36:00.380370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:00.379316    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh" podUID="cd0e6967-fe5d-4c57-87fa-0b284effe171" containerName="payload-processing" containerID="cri-o://f8e806d0f13a9b56c0b3f8be2f41cdfa853da6a31213f478f7d3017aa9c97fe1" gracePeriod=25
Nov 03 01:36:00.582371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:00.581545    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-kmx46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:36:00.582371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:00.581642    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:36:00.618371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:00.614450    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-wj656 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:36:00.618371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:00.614515    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" podUID="5f420022-cf2a-44db-a72b-2934f355375e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:36:00.618371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:00.614625    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-knxsx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:36:00.618371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:00.614652    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:36:01.050429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.049375    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:36:01.072734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.072429    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-s9q4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:36:01.072734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.072489    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:36:01.342383 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.341755    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-kube-api-access-295zc\" (UniqueName: \"kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-kube-api-access-295zc\") pod \"cd0e6967-fe5d-4c57-87fa-0b284effe171\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") "
Nov 03 01:36:01.342383 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.341806    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-provider-credentials\") pod \"cd0e6967-fe5d-4c57-87fa-0b284effe171\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") "
Nov 03 01:36:01.342383 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.341994    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-plugins-config-volume\") pod \"cd0e6967-fe5d-4c57-87fa-0b284effe171\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") "
Nov 03 01:36:01.342383 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.342039    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-routing-overlay-volume\") pod \"cd0e6967-fe5d-4c57-87fa-0b284effe171\" (UID: \"cd0e6967-fe5d-4c57-87fa-0b284effe171\") "
Nov 03 01:36:01.348306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.342830    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-routing-overlay-volume" pod "cd0e6967-fe5d-4c57-87fa-0b284effe171" (UID: "cd0e6967-fe5d-4c57-87fa-0b284effe171"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:01.348306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.343207    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-plugins-config-volume" pod "cd0e6967-fe5d-4c57-87fa-0b284effe171" (UID: "cd0e6967-fe5d-4c57-87fa-0b284effe171"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:01.367687 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.367473    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-provider-credentials" pod "cd0e6967-fe5d-4c57-87fa-0b284effe171" (UID: "cd0e6967-fe5d-4c57-87fa-0b284effe171"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:01.443371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.443314    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:01.467649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.443413    5508 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd0e6967-fe5d-4c57-87fa-0b284effe171-routing-overlay-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:01.467649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.443430    5508 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-provider-credentials\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:01.468320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.468286    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-kube-api-access-295zc" pod "cd0e6967-fe5d-4c57-87fa-0b284effe171" (UID: "cd0e6967-fe5d-4c57-87fa-0b284effe171"). InnerVolumeSpecName "kube-api-access-295zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:01.503371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.493238    5508 generic.go:356] "Generic (PLEG): container finished" podID="cd0e6967-fe5d-4c57-87fa-0b284effe171" containerID="f8e806d0f13a9b56c0b3f8be2f41cdfa853da6a31213f478f7d3017aa9c97fe1" exitCode=0
Nov 03 01:36:01.503371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.493501    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh" event={"ID":"cd0e6967-fe5d-4c57-87fa-0b284effe171","Type":"ContainerDied","Data":"f8e806d0f13a9b56c0b3f8be2f41cdfa853da6a31213f478f7d3017aa9c97fe1"}
Nov 03 01:36:01.503371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.493553    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh" event={"ID":"cd0e6967-fe5d-4c57-87fa-0b284effe171","Type":"ContainerDied","Data":"e39c37bad309a0d4dc8a0f5d52e7021fd36680b14096f842d8883945cc98e095"}
Nov 03 01:36:01.503371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.493582    5508 scope.go:122] "RemoveContainer" containerID="f8e806d0f13a9b56c0b3f8be2f41cdfa853da6a31213f478f7d3017aa9c97fe1"
Nov 03 01:36:01.503371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.493830    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"
Nov 03 01:36:01.526433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.526412    5508 scope.go:122] "RemoveContainer" containerID="f8e806d0f13a9b56c0b3f8be2f41cdfa853da6a31213f478f7d3017aa9c97fe1"
Nov 03 01:36:01.526765 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:36:01.526738    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e806d0f13a9b56c0b3f8be2f41cdfa853da6a31213f478f7d3017aa9c97fe1\": container with ID starting with f8e806d0f13a9b56c0b3f8be2f41cdfa853da6a31213f478f7d3017aa9c97fe1 not found: ID does not exist" containerID="f8e806d0f13a9b56c0b3f8be2f41cdfa853da6a31213f478f7d3017aa9c97fe1"
Nov 03 01:36:01.526850 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.526773    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e806d0f13a9b56c0b3f8be2f41cdfa853da6a31213f478f7d3017aa9c97fe1"} err="failed to get container status \"f8e806d0f13a9b56c0b3f8be2f41cdfa853da6a31213f478f7d3017aa9c97fe1\": rpc error: code = NotFound desc = could not find container \"f8e806d0f13a9b56c0b3f8be2f41cdfa853da6a31213f478f7d3017aa9c97fe1\": container with ID starting with f8e806d0f13a9b56c0b3f8be2f41cdfa853da6a31213f478f7d3017aa9c97fe1 not found: ID does not exist"
Nov 03 01:36:01.544808 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.544785    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-295zc\" (UniqueName: \"kubernetes.io/projected/cd0e6967-fe5d-4c57-87fa-0b284effe171-kube-api-access-295zc\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:01.665894 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.665669    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"]
Nov 03 01:36:01.709672 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:01.709641    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-88bf9945c-nqsfh"]
Nov 03 01:36:02.696869 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:02.696824    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0e6967-fe5d-4c57-87fa-0b284effe171" path="/var/lib/kubelet/pods/cd0e6967-fe5d-4c57-87fa-0b284effe171/volumes"
Nov 03 01:36:03.363452 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:36:03.363425    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d5931ee79a9a: secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:36:03.363452 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:36:03.363447    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6: secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:36:03.363655 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:36:03.363521    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials podName:0784acc9-0dd1-4ba3-86c3-c558bc8ccc40 nodeName:}" failed. No retries permitted until 2026-11-03 01:36:19.363507 +0000 UTC m=+1058.789113455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials") pod "payload-processing-external-model-c56f7b8f9-q9st6" (UID: "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40") : secret "agc-cred-d5931ee79a9a" not found
Nov 03 01:36:07.288067 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:07.288039    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:36:07.448115 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:07.448081    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"]
Nov 03 01:36:12.448721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.448657    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6" podUID="0784acc9-0dd1-4ba3-86c3-c558bc8ccc40" containerName="payload-processing" containerID="cri-o://aa19a2d3f5fa44be47f9b05697498d0dd8f6ff3495b7c77667fee4615c1d4345" gracePeriod=25
Nov 03 01:36:12.581329 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.581295    5508 generic.go:356] "Generic (PLEG): container finished" podID="0784acc9-0dd1-4ba3-86c3-c558bc8ccc40" containerID="aa19a2d3f5fa44be47f9b05697498d0dd8f6ff3495b7c77667fee4615c1d4345" exitCode=0
Nov 03 01:36:12.581524 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.581417    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6" event={"ID":"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40","Type":"ContainerDied","Data":"aa19a2d3f5fa44be47f9b05697498d0dd8f6ff3495b7c77667fee4615c1d4345"}
Nov 03 01:36:12.704954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.704900    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:36:12.801194 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.801160    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-routing-overlay-volume\") pod \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") "
Nov 03 01:36:12.801452 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.801269    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-kube-api-access-mj4kb\" (UniqueName: \"kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-kube-api-access-mj4kb\") pod \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") "
Nov 03 01:36:12.801452 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.801394    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-plugins-config-volume\") pod \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") "
Nov 03 01:36:12.801591 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.801492    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials\") pod \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\" (UID: \"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40\") "
Nov 03 01:36:12.801714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.801682    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-routing-overlay-volume" pod "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40" (UID: "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:12.801819 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.801796    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-plugins-config-volume" pod "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40" (UID: "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:12.801944 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.801930    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:12.801996 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.801950    5508 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-routing-overlay-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:12.803492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.803471    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials" pod "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40" (UID: "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:12.803584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.803505    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-kube-api-access-mj4kb" pod "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40" (UID: "0784acc9-0dd1-4ba3-86c3-c558bc8ccc40"). InnerVolumeSpecName "kube-api-access-mj4kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:12.902997 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.902966    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj4kb\" (UniqueName: \"kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-kube-api-access-mj4kb\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:12.902997 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:12.902994    5508 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40-provider-credentials\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:36:13.589094 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:13.589062    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6" event={"ID":"0784acc9-0dd1-4ba3-86c3-c558bc8ccc40","Type":"ContainerDied","Data":"720a29401f8a76c3e8ebe927c8d94d8a143a174e48405c276f44184248548e6f"}
Nov 03 01:36:13.589605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:13.589098    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"
Nov 03 01:36:13.589605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:13.589122    5508 scope.go:122] "RemoveContainer" containerID="aa19a2d3f5fa44be47f9b05697498d0dd8f6ff3495b7c77667fee4615c1d4345"
Nov 03 01:36:13.744036 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:13.743998    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"]
Nov 03 01:36:13.790591 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:13.790546    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-c56f7b8f9-q9st6"]
Nov 03 01:36:13.813655 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:36:13.813617    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:36:14.752732 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:14.752690    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0784acc9-0dd1-4ba3-86c3-c558bc8ccc40" path="/var/lib/kubelet/pods/0784acc9-0dd1-4ba3-86c3-c558bc8ccc40/volumes"
Nov 03 01:36:25.534327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:25.534285    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:36:40.857728 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:40.857687    5508 scope.go:122] "RemoveContainer" containerID="176dbd36ecfaba153afe0f62edf87aa605118d9ed521ab50dd5cbe470099dcea"
Nov 03 01:36:40.870080 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:40.870054    5508 scope.go:122] "RemoveContainer" containerID="e6508f97438696e96b2ff47d093b28d59a31c7faf1d0d904744f90d6e74b7af9"
Nov 03 01:36:40.886331 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:40.883711    5508 scope.go:122] "RemoveContainer" containerID="4aca470736cb38737e446dce3467c076c34ea6f1f95b1771e1e3ddd1ef67987b"
Nov 03 01:36:43.812549 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:36:43.812508    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:36:45.660417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.660379    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"]
Nov 03 01:36:45.660919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.660866    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0784acc9-0dd1-4ba3-86c3-c558bc8ccc40" containerName="payload-processing"
Nov 03 01:36:45.660919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.660878    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0784acc9-0dd1-4ba3-86c3-c558bc8ccc40" containerName="payload-processing"
Nov 03 01:36:45.660919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.660893    5508 cpu_manager.go:414] "containerMap: removing container" podUID="cd0e6967-fe5d-4c57-87fa-0b284effe171" containerName="payload-processing"
Nov 03 01:36:45.660919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.660897    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0e6967-fe5d-4c57-87fa-0b284effe171" containerName="payload-processing"
Nov 03 01:36:45.661118 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.660971    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0784acc9-0dd1-4ba3-86c3-c558bc8ccc40" containerName="payload-processing"
Nov 03 01:36:45.661118 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.660979    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0e6967-fe5d-4c57-87fa-0b284effe171" containerName="payload-processing"
Nov 03 01:36:45.665931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.665909    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.762095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.762064    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"]
Nov 03 01:36:45.790923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.790892    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.791096 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.790930    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.791096 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.790958    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.791232 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.791215    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.791325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.791298    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.791482 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.791460    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.791576 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.791562    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.791636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.791585    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mvq4\" (UniqueName: \"kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-kube-api-access-7mvq4\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.791723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.791703    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.892339 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.892262    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.892537 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.892342    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.892537 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.892401    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mvq4\" (UniqueName: \"kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-kube-api-access-7mvq4\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.892537 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.892466    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.892537 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.892519    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.892771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.892546    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.892771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.892582    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.892771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.892618    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.892771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.892664    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.893341 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.893134    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.893341 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.893192    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.893341 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.893197    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.893341 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.893240    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.893604 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.893465    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.895005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.894985    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.895098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.895062    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.949772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.949701    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.949772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.949755    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mvq4\" (UniqueName: \"kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-kube-api-access-7mvq4\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:45.979618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:45.979587    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:46.414479 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:46.414444    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"]
Nov 03 01:36:46.422890 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:36:46.422864    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0896f72f_e85a_4e3c_9a25_876e0a78f4b6.slice/crio-293d11ed5f9bbbcce13a33aba0a82e65b4aec4f260dbcb056ae4487b1f9e4539 WatchSource:0}: Error finding container 293d11ed5f9bbbcce13a33aba0a82e65b4aec4f260dbcb056ae4487b1f9e4539: Status 404 returned error can't find the container with id 293d11ed5f9bbbcce13a33aba0a82e65b4aec4f260dbcb056ae4487b1f9e4539
Nov 03 01:36:46.425179 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:46.425150    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:36:46.425255 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:46.425215    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:36:46.425255 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:46.425243    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:36:46.911636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:46.909320    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n" event={"ID":"0896f72f-e85a-4e3c-9a25-876e0a78f4b6","Type":"ContainerStarted","Data":"a9f22498b67b15f50e5544c5254b1cfc167e12ec401715705adc27f5bc973a08"}
Nov 03 01:36:46.911636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:46.909429    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n" event={"ID":"0896f72f-e85a-4e3c-9a25-876e0a78f4b6","Type":"ContainerStarted","Data":"293d11ed5f9bbbcce13a33aba0a82e65b4aec4f260dbcb056ae4487b1f9e4539"}
Nov 03 01:36:46.980734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:46.980696    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:47.027176 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:47.027115    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n" podStartSLOduration=2.027096067 podStartE2EDuration="2.027096067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:47.025705043 +0000 UTC m=+1086.451311512" watchObservedRunningTime="2026-11-03 01:36:47.027096067 +0000 UTC m=+1086.452702532"
Nov 03 01:36:47.982001 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:47.981961    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 01:36:47.982537 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:47.982041    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n" podUID="0896f72f-e85a-4e3c-9a25-876e0a78f4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 01:36:48.981198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:48.980738    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 01:36:48.981198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:48.980802    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n" podUID="0896f72f-e85a-4e3c-9a25-876e0a78f4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 01:36:49.981390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:49.980867    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 01:36:49.981390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:49.980934    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n" podUID="0896f72f-e85a-4e3c-9a25-876e0a78f4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 01:36:50.980887 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:50.980841    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 01:36:50.981116 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:50.980927    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n" podUID="0896f72f-e85a-4e3c-9a25-876e0a78f4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 01:36:51.127357 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:51.127308    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:36:51.980867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:51.980828    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 01:36:51.981056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:51.980883    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n" podUID="0896f72f-e85a-4e3c-9a25-876e0a78f4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 01:36:52.980779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:52.980743    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 01:36:52.981195 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:52.980798    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n" podUID="0896f72f-e85a-4e3c-9a25-876e0a78f4b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 01:36:53.985197 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:53.985164    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:53.985727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:53.985554    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:36:53.986249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:36:53.986230    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:37:10.855904 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:10.855863    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"]
Nov 03 01:37:10.859935 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:10.859917    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:10.914454 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:10.914409    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"]
Nov 03 01:37:10.957068 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:10.957032    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9da431\"/\"distinct-2-w2-9da431-kserve-self-signed-certs\""
Nov 03 01:37:10.961274 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:10.961249    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-home\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:10.961482 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:10.961361    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-dshm\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:10.961542 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:10.961501    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcf59\" (UniqueName: \"kubernetes.io/projected/2d613494-eedc-4a88-86c3-d076b634c9e1-kube-api-access-qcf59\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:10.961583 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:10.961567    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-tmp-dir\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:10.961731 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:10.961690    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-model-cache\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:10.961778 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:10.961747    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d613494-eedc-4a88-86c3-d076b634c9e1-tls-certs\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.065157 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.065111    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-tmp-dir\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.065414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.065184    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-model-cache\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.065414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.065211    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d613494-eedc-4a88-86c3-d076b634c9e1-tls-certs\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.065414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.065315    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-home\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.065599 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.065439    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-dshm\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.065599 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.065472    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcf59\" (UniqueName: \"kubernetes.io/projected/2d613494-eedc-4a88-86c3-d076b634c9e1-kube-api-access-qcf59\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.066256 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.066170    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-home\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.066471 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.066446    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-tmp-dir\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.066729 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.066669    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-model-cache\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.080368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.075250    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-dshm\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.082982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.082952    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d613494-eedc-4a88-86c3-d076b634c9e1-tls-certs\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.123071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.122995    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcf59\" (UniqueName: \"kubernetes.io/projected/2d613494-eedc-4a88-86c3-d076b634c9e1-kube-api-access-qcf59\") pod \"distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") " pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.173608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.173558    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:11.545090 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:11.545051    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"]
Nov 03 01:37:11.552845 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:37:11.552812    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d613494_eedc_4a88_86c3_d076b634c9e1.slice/crio-a467bf38ee3b62f222fe133e6b9c84a0f57907b822f1ba56ae650e024ca01362 WatchSource:0}: Error finding container a467bf38ee3b62f222fe133e6b9c84a0f57907b822f1ba56ae650e024ca01362: Status 404 returned error can't find the container with id a467bf38ee3b62f222fe133e6b9c84a0f57907b822f1ba56ae650e024ca01362
Nov 03 01:37:12.089782 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:12.089749    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" event={"ID":"2d613494-eedc-4a88-86c3-d076b634c9e1","Type":"ContainerStarted","Data":"33a8dbc71ed34f025fc6e8b2a20928178815895a0722d8d269634f69158deb17"}
Nov 03 01:37:12.090297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:12.089800    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" event={"ID":"2d613494-eedc-4a88-86c3-d076b634c9e1","Type":"ContainerStarted","Data":"a467bf38ee3b62f222fe133e6b9c84a0f57907b822f1ba56ae650e024ca01362"}
Nov 03 01:37:12.174664 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:12.174628    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:12.200620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:12.200568    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podStartSLOduration=2.20054881 podStartE2EDuration="2.20054881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:37:12.199750704 +0000 UTC m=+1111.625357198" watchObservedRunningTime="2026-11-03 01:37:12.20054881 +0000 UTC m=+1111.626155279"
Nov 03 01:37:13.811980 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:37:13.811942    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:37:13.914122 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:13.914092    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:37:21.174024 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:21.173987    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:21.186663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:21.186634    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:22.179824 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:22.179793    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:37:36.888121 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:36.888087    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:37:43.811408 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:37:43.811368    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:37:59.011653 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.011611    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"]
Nov 03 01:37:59.015795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.015769    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.086620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.086576    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4243b3-d50b-4995-b51d-29174542a75d-tls-certs\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.086845 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.086679    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-home\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.086845 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.086759    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-model-cache\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.086975 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.086849    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-tmp-dir\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.086975 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.086873    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftxd4\" (UniqueName: \"kubernetes.io/projected/5a4243b3-d50b-4995-b51d-29174542a75d-kube-api-access-ftxd4\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.086975 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.086907    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-dshm\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.113446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.113399    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0bf34a\"/\"default-dockercfg-tk8nr\""
Nov 03 01:37:59.113446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.113428    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0bf34a\"/\"kube-root-ca.crt\""
Nov 03 01:37:59.113446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.113428    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0bf34a\"/\"openshift-service-ca.crt\""
Nov 03 01:37:59.113746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.113525    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0bf34a\"/\"distinct-2-w4-0bf34a-kserve-self-signed-certs\""
Nov 03 01:37:59.116607 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.116585    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"]
Nov 03 01:37:59.189127 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.189082    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4243b3-d50b-4995-b51d-29174542a75d-tls-certs\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.189678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.189625    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-home\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.189963 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.189944    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-model-cache\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.190240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.190118    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-tmp-dir\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.190453 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.190425    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftxd4\" (UniqueName: \"kubernetes.io/projected/5a4243b3-d50b-4995-b51d-29174542a75d-kube-api-access-ftxd4\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.190606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.190591    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-dshm\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.191590 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.191568    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-model-cache\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.192340 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.192321    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-home\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.193025 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.192701    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-tmp-dir\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.223958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.221556    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-dshm\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.225654 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.225624    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4243b3-d50b-4995-b51d-29174542a75d-tls-certs\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.249934 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.249910    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftxd4\" (UniqueName: \"kubernetes.io/projected/5a4243b3-d50b-4995-b51d-29174542a75d-kube-api-access-ftxd4\") pod \"distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") " pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.327252 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.327181    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:37:59.709319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:37:59.709280    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"]
Nov 03 01:37:59.716341 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:37:59.716310    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a4243b3_d50b_4995_b51d_29174542a75d.slice/crio-8d1a088dd6c2a1d757d9a60d6858dfc30229d64db737750c101c751de03d9da1 WatchSource:0}: Error finding container 8d1a088dd6c2a1d757d9a60d6858dfc30229d64db737750c101c751de03d9da1: Status 404 returned error can't find the container with id 8d1a088dd6c2a1d757d9a60d6858dfc30229d64db737750c101c751de03d9da1
Nov 03 01:38:00.230566 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:00.230513    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:38:00.523018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:00.522918    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" event={"ID":"5a4243b3-d50b-4995-b51d-29174542a75d","Type":"ContainerStarted","Data":"e6ab55afcee4122d6c7698fd1dc1819aeb2d2192b90c28e49281dba99f5bc2d0"}
Nov 03 01:38:00.523018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:00.522965    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" event={"ID":"5a4243b3-d50b-4995-b51d-29174542a75d","Type":"ContainerStarted","Data":"8d1a088dd6c2a1d757d9a60d6858dfc30229d64db737750c101c751de03d9da1"}
Nov 03 01:38:00.641375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:00.640493    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podStartSLOduration=2.640472847 podStartE2EDuration="2.640472847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:00.639292731 +0000 UTC m=+1160.064899199" watchObservedRunningTime="2026-11-03 01:38:00.640472847 +0000 UTC m=+1160.066079315"
Nov 03 01:38:01.327605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:01.327563    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:38:09.327563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:09.327525    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:38:09.340190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:09.340162    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:38:09.601521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:09.601444    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:38:10.057310 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:10.057268    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"]
Nov 03 01:38:10.061648 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:10.061616    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c16e77-9jk84"]
Nov 03 01:38:10.692675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:10.692642    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82394f3d-e4c7-43d3-be66-92409381ed99" path="/var/lib/kubelet/pods/82394f3d-e4c7-43d3-be66-92409381ed99/volumes"
Nov 03 01:38:13.812231 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:38:13.812182    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:38:20.428866 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.428776    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"]
Nov 03 01:38:20.434054 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.434030    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"
Nov 03 01:38:20.486207 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.486176    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"]
Nov 03 01:38:20.529059 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.529029    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:38:20.529059 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.529051    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:38:20.529267 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.529085    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:38:20.529267 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.529085    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:38:20.570915 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.570884    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-578a8646-qw69z\" (UID: \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"
Nov 03 01:38:20.571076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.570936    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-578a8646-qw69z\" (UID: \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"
Nov 03 01:38:20.571076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.571050    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw249\" (UniqueName: \"kubernetes.io/projected/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-kube-api-access-bw249\") pod \"maas-api-revoke-sub-578a8646-qw69z\" (UID: \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"
Nov 03 01:38:20.672491 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.672445    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-578a8646-qw69z\" (UID: \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"
Nov 03 01:38:20.672672 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.672517    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-578a8646-qw69z\" (UID: \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"
Nov 03 01:38:20.672672 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.672552    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw249\" (UniqueName: \"kubernetes.io/projected/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-kube-api-access-bw249\") pod \"maas-api-revoke-sub-578a8646-qw69z\" (UID: \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"
Nov 03 01:38:20.673170 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.673152    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-578a8646-qw69z\" (UID: \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"
Nov 03 01:38:20.674946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.674921    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-578a8646-qw69z\" (UID: \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"
Nov 03 01:38:20.730083 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.730014    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw249\" (UniqueName: \"kubernetes.io/projected/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-kube-api-access-bw249\") pod \"maas-api-revoke-sub-578a8646-qw69z\" (UID: \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"
Nov 03 01:38:20.748125 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:20.748102    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"
Nov 03 01:38:21.121722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:21.121646    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"]
Nov 03 01:38:21.128426 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:38:21.128392    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f5b563_9f66_4f89_ba51_96a24ccdce3c.slice/crio-6e6ff6aff93213b4bb2deae13fa41165124641c250aa574363934b456c859e51 WatchSource:0}: Error finding container 6e6ff6aff93213b4bb2deae13fa41165124641c250aa574363934b456c859e51: Status 404 returned error can't find the container with id 6e6ff6aff93213b4bb2deae13fa41165124641c250aa574363934b456c859e51
Nov 03 01:38:21.667085 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:21.667046    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z" event={"ID":"b9f5b563-9f66-4f89-ba51-96a24ccdce3c","Type":"ContainerStarted","Data":"9041b079d5b2333ffb09e0fa173b762c63ed147d4f333345087988620863a577"}
Nov 03 01:38:21.667552 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:21.667141    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z" event={"ID":"b9f5b563-9f66-4f89-ba51-96a24ccdce3c","Type":"ContainerStarted","Data":"6e6ff6aff93213b4bb2deae13fa41165124641c250aa574363934b456c859e51"}
Nov 03 01:38:21.776496 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:21.776444    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z" podStartSLOduration=1.7764297999999998 podStartE2EDuration="1.7764298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:21.774840137 +0000 UTC m=+1181.200446611" watchObservedRunningTime="2026-11-03 01:38:21.7764298 +0000 UTC m=+1181.202036298"
Nov 03 01:38:22.674651 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:22.674618    5508 generic.go:356] "Generic (PLEG): container finished" podID="b9f5b563-9f66-4f89-ba51-96a24ccdce3c" containerID="9041b079d5b2333ffb09e0fa173b762c63ed147d4f333345087988620863a577" exitCode=0
Nov 03 01:38:22.675041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:22.674703    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z" event={"ID":"b9f5b563-9f66-4f89-ba51-96a24ccdce3c","Type":"ContainerDied","Data":"9041b079d5b2333ffb09e0fa173b762c63ed147d4f333345087988620863a577"}
Nov 03 01:38:23.887381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:23.887341    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"
Nov 03 01:38:24.032951 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.032851    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-kube-api-access-bw249\" (UniqueName: \"kubernetes.io/projected/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-kube-api-access-bw249\") pod \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\" (UID: \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\") "
Nov 03 01:38:24.033123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.032951    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-openshift-ca\") pod \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\" (UID: \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\") "
Nov 03 01:38:24.033123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.032973    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-xks-ca\") pod \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\" (UID: \"b9f5b563-9f66-4f89-ba51-96a24ccdce3c\") "
Nov 03 01:38:24.033302 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.033270    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-openshift-ca" pod "b9f5b563-9f66-4f89-ba51-96a24ccdce3c" (UID: "b9f5b563-9f66-4f89-ba51-96a24ccdce3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:38:24.033556 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.033537    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:24.035156 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.035127    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-kube-api-access-bw249" pod "b9f5b563-9f66-4f89-ba51-96a24ccdce3c" (UID: "b9f5b563-9f66-4f89-ba51-96a24ccdce3c"). InnerVolumeSpecName "kube-api-access-bw249". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:24.035507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.035480    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-xks-ca" pod "b9f5b563-9f66-4f89-ba51-96a24ccdce3c" (UID: "b9f5b563-9f66-4f89-ba51-96a24ccdce3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:38:24.134237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.134205    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw249\" (UniqueName: \"kubernetes.io/projected/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-kube-api-access-bw249\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:24.134237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.134233    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9f5b563-9f66-4f89-ba51-96a24ccdce3c-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:24.652429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.652392    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"]
Nov 03 01:38:24.653078 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.653053    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b9f5b563-9f66-4f89-ba51-96a24ccdce3c" containerName="revoke-keys"
Nov 03 01:38:24.653078 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.653077    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f5b563-9f66-4f89-ba51-96a24ccdce3c" containerName="revoke-keys"
Nov 03 01:38:24.653279 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.653239    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f5b563-9f66-4f89-ba51-96a24ccdce3c" containerName="revoke-keys"
Nov 03 01:38:24.657108 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.657089    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"
Nov 03 01:38:24.691787 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.691759    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"
Nov 03 01:38:24.693957 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.693931    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z" event={"ID":"b9f5b563-9f66-4f89-ba51-96a24ccdce3c","Type":"ContainerDied","Data":"6e6ff6aff93213b4bb2deae13fa41165124641c250aa574363934b456c859e51"}
Nov 03 01:38:24.694074 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.693972    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e6ff6aff93213b4bb2deae13fa41165124641c250aa574363934b456c859e51"
Nov 03 01:38:24.709739 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.709713    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"]
Nov 03 01:38:24.841264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.841235    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a5bf2f3-vkgg5\" (UID: \"5601949d-18e6-4ec3-a012-8c8523d2379b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"
Nov 03 01:38:24.841264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.841268    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9mj\" (UniqueName: \"kubernetes.io/projected/5601949d-18e6-4ec3-a012-8c8523d2379b-kube-api-access-lk9mj\") pod \"maas-api-revoke-sub-4a5bf2f3-vkgg5\" (UID: \"5601949d-18e6-4ec3-a012-8c8523d2379b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"
Nov 03 01:38:24.841522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.841457    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a5bf2f3-vkgg5\" (UID: \"5601949d-18e6-4ec3-a012-8c8523d2379b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"
Nov 03 01:38:24.943072 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.942974    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a5bf2f3-vkgg5\" (UID: \"5601949d-18e6-4ec3-a012-8c8523d2379b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"
Nov 03 01:38:24.943072 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.943026    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9mj\" (UniqueName: \"kubernetes.io/projected/5601949d-18e6-4ec3-a012-8c8523d2379b-kube-api-access-lk9mj\") pod \"maas-api-revoke-sub-4a5bf2f3-vkgg5\" (UID: \"5601949d-18e6-4ec3-a012-8c8523d2379b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"
Nov 03 01:38:24.943631 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.943133    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a5bf2f3-vkgg5\" (UID: \"5601949d-18e6-4ec3-a012-8c8523d2379b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"
Nov 03 01:38:24.943674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.943633    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a5bf2f3-vkgg5\" (UID: \"5601949d-18e6-4ec3-a012-8c8523d2379b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"
Nov 03 01:38:24.945560 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:24.945540    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a5bf2f3-vkgg5\" (UID: \"5601949d-18e6-4ec3-a012-8c8523d2379b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"
Nov 03 01:38:25.000394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:25.000366    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9mj\" (UniqueName: \"kubernetes.io/projected/5601949d-18e6-4ec3-a012-8c8523d2379b-kube-api-access-lk9mj\") pod \"maas-api-revoke-sub-4a5bf2f3-vkgg5\" (UID: \"5601949d-18e6-4ec3-a012-8c8523d2379b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"
Nov 03 01:38:25.272455 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:25.272369    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"
Nov 03 01:38:25.645314 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:25.645219    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"]
Nov 03 01:38:25.652717 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:38:25.652686    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5601949d_18e6_4ec3_a012_8c8523d2379b.slice/crio-b246c1e821f9425f2ef343b712641eff31858643a9806dffab81affc668a3c73 WatchSource:0}: Error finding container b246c1e821f9425f2ef343b712641eff31858643a9806dffab81affc668a3c73: Status 404 returned error can't find the container with id b246c1e821f9425f2ef343b712641eff31858643a9806dffab81affc668a3c73
Nov 03 01:38:25.701806 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:25.701772    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5" event={"ID":"5601949d-18e6-4ec3-a012-8c8523d2379b","Type":"ContainerStarted","Data":"b246c1e821f9425f2ef343b712641eff31858643a9806dffab81affc668a3c73"}
Nov 03 01:38:26.709155 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:26.709124    5508 generic.go:356] "Generic (PLEG): container finished" podID="5601949d-18e6-4ec3-a012-8c8523d2379b" containerID="00c22be3cde21736ae6040edb87ca3444387ae5c475ec370ba1981a484eedc11" exitCode=0
Nov 03 01:38:26.709599 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:26.709175    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5" event={"ID":"5601949d-18e6-4ec3-a012-8c8523d2379b","Type":"ContainerDied","Data":"00c22be3cde21736ae6040edb87ca3444387ae5c475ec370ba1981a484eedc11"}
Nov 03 01:38:27.867494 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:27.867465    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"
Nov 03 01:38:27.878079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:27.878042    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-xks-ca\") pod \"5601949d-18e6-4ec3-a012-8c8523d2379b\" (UID: \"5601949d-18e6-4ec3-a012-8c8523d2379b\") "
Nov 03 01:38:27.878277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:27.878128    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-openshift-ca\") pod \"5601949d-18e6-4ec3-a012-8c8523d2379b\" (UID: \"5601949d-18e6-4ec3-a012-8c8523d2379b\") "
Nov 03 01:38:27.878277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:27.878225    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5601949d-18e6-4ec3-a012-8c8523d2379b-kube-api-access-lk9mj\" (UniqueName: \"kubernetes.io/projected/5601949d-18e6-4ec3-a012-8c8523d2379b-kube-api-access-lk9mj\") pod \"5601949d-18e6-4ec3-a012-8c8523d2379b\" (UID: \"5601949d-18e6-4ec3-a012-8c8523d2379b\") "
Nov 03 01:38:27.878525 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:27.878501    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-openshift-ca" pod "5601949d-18e6-4ec3-a012-8c8523d2379b" (UID: "5601949d-18e6-4ec3-a012-8c8523d2379b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:38:27.878706 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:27.878687    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:27.880242 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:27.880216    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-xks-ca" pod "5601949d-18e6-4ec3-a012-8c8523d2379b" (UID: "5601949d-18e6-4ec3-a012-8c8523d2379b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:38:27.880788 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:27.880755    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5601949d-18e6-4ec3-a012-8c8523d2379b-kube-api-access-lk9mj" pod "5601949d-18e6-4ec3-a012-8c8523d2379b" (UID: "5601949d-18e6-4ec3-a012-8c8523d2379b"). InnerVolumeSpecName "kube-api-access-lk9mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:27.980679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:27.980573    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5601949d-18e6-4ec3-a012-8c8523d2379b-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:27.980679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:27.980604    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk9mj\" (UniqueName: \"kubernetes.io/projected/5601949d-18e6-4ec3-a012-8c8523d2379b-kube-api-access-lk9mj\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:28.200222 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:28.200189    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:38:28.723389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:28.723339    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"
Nov 03 01:38:28.723389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:28.723375    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5" event={"ID":"5601949d-18e6-4ec3-a012-8c8523d2379b","Type":"ContainerDied","Data":"b246c1e821f9425f2ef343b712641eff31858643a9806dffab81affc668a3c73"}
Nov 03 01:38:28.723586 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:28.723422    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b246c1e821f9425f2ef343b712641eff31858643a9806dffab81affc668a3c73"
Nov 03 01:38:29.059049 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:29.058917    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"]
Nov 03 01:38:29.068457 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:29.068420    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047b4a38-czq4r"]
Nov 03 01:38:30.733996 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:30.733945    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6021a8d-d122-42f5-a61e-4efa4e93ffbf" path="/var/lib/kubelet/pods/d6021a8d-d122-42f5-a61e-4efa4e93ffbf/volumes"
Nov 03 01:38:35.378551 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.378500    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"]
Nov 03 01:38:35.379017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.379006    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5601949d-18e6-4ec3-a012-8c8523d2379b" containerName="revoke-keys"
Nov 03 01:38:35.379072 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.379018    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5601949d-18e6-4ec3-a012-8c8523d2379b" containerName="revoke-keys"
Nov 03 01:38:35.379124 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.379108    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5601949d-18e6-4ec3-a012-8c8523d2379b" containerName="revoke-keys"
Nov 03 01:38:35.383634 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.383607    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"
Nov 03 01:38:35.419947 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.419915    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c5a642-782c-4f47-abae-a3f376686970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5f0be2e-r7jp7\" (UID: \"46c5a642-782c-4f47-abae-a3f376686970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"
Nov 03 01:38:35.420108 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.420056    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bbt8\" (UniqueName: \"kubernetes.io/projected/46c5a642-782c-4f47-abae-a3f376686970-kube-api-access-7bbt8\") pod \"maas-api-revoke-sub-c5f0be2e-r7jp7\" (UID: \"46c5a642-782c-4f47-abae-a3f376686970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"
Nov 03 01:38:35.420165 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.420114    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c5a642-782c-4f47-abae-a3f376686970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5f0be2e-r7jp7\" (UID: \"46c5a642-782c-4f47-abae-a3f376686970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"
Nov 03 01:38:35.432478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.432455    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:38:35.436227 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.436199    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"]
Nov 03 01:38:35.478569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.478514    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:38:35.478569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.478525    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:38:35.478763 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.478603    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:38:35.521159 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.521126    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bbt8\" (UniqueName: \"kubernetes.io/projected/46c5a642-782c-4f47-abae-a3f376686970-kube-api-access-7bbt8\") pod \"maas-api-revoke-sub-c5f0be2e-r7jp7\" (UID: \"46c5a642-782c-4f47-abae-a3f376686970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"
Nov 03 01:38:35.521291 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.521186    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c5a642-782c-4f47-abae-a3f376686970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5f0be2e-r7jp7\" (UID: \"46c5a642-782c-4f47-abae-a3f376686970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"
Nov 03 01:38:35.521427 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.521399    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c5a642-782c-4f47-abae-a3f376686970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5f0be2e-r7jp7\" (UID: \"46c5a642-782c-4f47-abae-a3f376686970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"
Nov 03 01:38:35.521909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.521887    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c5a642-782c-4f47-abae-a3f376686970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5f0be2e-r7jp7\" (UID: \"46c5a642-782c-4f47-abae-a3f376686970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"
Nov 03 01:38:35.523697 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.523679    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c5a642-782c-4f47-abae-a3f376686970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5f0be2e-r7jp7\" (UID: \"46c5a642-782c-4f47-abae-a3f376686970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"
Nov 03 01:38:35.578928 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.578900    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bbt8\" (UniqueName: \"kubernetes.io/projected/46c5a642-782c-4f47-abae-a3f376686970-kube-api-access-7bbt8\") pod \"maas-api-revoke-sub-c5f0be2e-r7jp7\" (UID: \"46c5a642-782c-4f47-abae-a3f376686970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"
Nov 03 01:38:35.700868 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:35.700833    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"
Nov 03 01:38:36.072603 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:36.072505    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"]
Nov 03 01:38:36.079766 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:38:36.079730    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c5a642_782c_4f47_abae_a3f376686970.slice/crio-0ab36e1370240b6113f071e5c7834fa74275edeff82c2a1eaa7e318cf702d14d WatchSource:0}: Error finding container 0ab36e1370240b6113f071e5c7834fa74275edeff82c2a1eaa7e318cf702d14d: Status 404 returned error can't find the container with id 0ab36e1370240b6113f071e5c7834fa74275edeff82c2a1eaa7e318cf702d14d
Nov 03 01:38:36.803059 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:36.803024    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7" event={"ID":"46c5a642-782c-4f47-abae-a3f376686970","Type":"ContainerStarted","Data":"c80d8f77649378fce0c8218c4e0b797290fcb1200ec5fd5fb26fbc381733dba4"}
Nov 03 01:38:36.803536 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:36.803075    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7" event={"ID":"46c5a642-782c-4f47-abae-a3f376686970","Type":"ContainerStarted","Data":"0ab36e1370240b6113f071e5c7834fa74275edeff82c2a1eaa7e318cf702d14d"}
Nov 03 01:38:36.913809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:36.913756    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7" podStartSLOduration=1.9137418259999999 podStartE2EDuration="1.913741826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:36.910305222 +0000 UTC m=+1196.335911690" watchObservedRunningTime="2026-11-03 01:38:36.913741826 +0000 UTC m=+1196.339348291"
Nov 03 01:38:36.959827 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:36.959799    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"]
Nov 03 01:38:36.963956 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:36.963940    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"
Nov 03 01:38:37.014701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.014673    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"]
Nov 03 01:38:37.040389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.040224    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dee45262-htfff\" (UID: \"784a5f88-f63f-41be-afce-9dddb8dc2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"
Nov 03 01:38:37.040389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.040311    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dee45262-htfff\" (UID: \"784a5f88-f63f-41be-afce-9dddb8dc2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"
Nov 03 01:38:37.040389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.040363    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vx6s\" (UniqueName: \"kubernetes.io/projected/784a5f88-f63f-41be-afce-9dddb8dc2edb-kube-api-access-4vx6s\") pod \"maas-api-revoke-sub-dee45262-htfff\" (UID: \"784a5f88-f63f-41be-afce-9dddb8dc2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"
Nov 03 01:38:37.141795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.141704    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dee45262-htfff\" (UID: \"784a5f88-f63f-41be-afce-9dddb8dc2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"
Nov 03 01:38:37.141795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.141745    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dee45262-htfff\" (UID: \"784a5f88-f63f-41be-afce-9dddb8dc2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"
Nov 03 01:38:37.142010 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.141797    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vx6s\" (UniqueName: \"kubernetes.io/projected/784a5f88-f63f-41be-afce-9dddb8dc2edb-kube-api-access-4vx6s\") pod \"maas-api-revoke-sub-dee45262-htfff\" (UID: \"784a5f88-f63f-41be-afce-9dddb8dc2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"
Nov 03 01:38:37.142497 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.142472    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dee45262-htfff\" (UID: \"784a5f88-f63f-41be-afce-9dddb8dc2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"
Nov 03 01:38:37.144133 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.144109    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dee45262-htfff\" (UID: \"784a5f88-f63f-41be-afce-9dddb8dc2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"
Nov 03 01:38:37.198835 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.198807    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vx6s\" (UniqueName: \"kubernetes.io/projected/784a5f88-f63f-41be-afce-9dddb8dc2edb-kube-api-access-4vx6s\") pod \"maas-api-revoke-sub-dee45262-htfff\" (UID: \"784a5f88-f63f-41be-afce-9dddb8dc2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"
Nov 03 01:38:37.278064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.278033    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"
Nov 03 01:38:37.651292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.651257    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"]
Nov 03 01:38:37.657626 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:38:37.657599    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784a5f88_f63f_41be_afce_9dddb8dc2edb.slice/crio-395f00ca1adb2745636152912a549e351f585856c8aad08b469f1cc4c02b936d WatchSource:0}: Error finding container 395f00ca1adb2745636152912a549e351f585856c8aad08b469f1cc4c02b936d: Status 404 returned error can't find the container with id 395f00ca1adb2745636152912a549e351f585856c8aad08b469f1cc4c02b936d
Nov 03 01:38:37.811944 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.811909    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff" event={"ID":"784a5f88-f63f-41be-afce-9dddb8dc2edb","Type":"ContainerStarted","Data":"395f00ca1adb2745636152912a549e351f585856c8aad08b469f1cc4c02b936d"}
Nov 03 01:38:37.813704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.813680    5508 generic.go:356] "Generic (PLEG): container finished" podID="46c5a642-782c-4f47-abae-a3f376686970" containerID="c80d8f77649378fce0c8218c4e0b797290fcb1200ec5fd5fb26fbc381733dba4" exitCode=0
Nov 03 01:38:37.813837 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:37.813729    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7" event={"ID":"46c5a642-782c-4f47-abae-a3f376686970","Type":"ContainerDied","Data":"c80d8f77649378fce0c8218c4e0b797290fcb1200ec5fd5fb26fbc381733dba4"}
Nov 03 01:38:38.821701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:38.821613    5508 generic.go:356] "Generic (PLEG): container finished" podID="784a5f88-f63f-41be-afce-9dddb8dc2edb" containerID="49280a814147b0a0239539bf17d488d3001d346751ee54ac47deb29ed94b0548" exitCode=0
Nov 03 01:38:38.822075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:38.821696    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff" event={"ID":"784a5f88-f63f-41be-afce-9dddb8dc2edb","Type":"ContainerDied","Data":"49280a814147b0a0239539bf17d488d3001d346751ee54ac47deb29ed94b0548"}
Nov 03 01:38:38.985890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:38.985863    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"
Nov 03 01:38:39.069629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.069595    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c5a642-782c-4f47-abae-a3f376686970-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c5a642-782c-4f47-abae-a3f376686970-maas-api-openshift-ca\") pod \"46c5a642-782c-4f47-abae-a3f376686970\" (UID: \"46c5a642-782c-4f47-abae-a3f376686970\") "
Nov 03 01:38:39.069839 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.069735    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c5a642-782c-4f47-abae-a3f376686970-kube-api-access-7bbt8\" (UniqueName: \"kubernetes.io/projected/46c5a642-782c-4f47-abae-a3f376686970-kube-api-access-7bbt8\") pod \"46c5a642-782c-4f47-abae-a3f376686970\" (UID: \"46c5a642-782c-4f47-abae-a3f376686970\") "
Nov 03 01:38:39.069839 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.069772    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46c5a642-782c-4f47-abae-a3f376686970-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c5a642-782c-4f47-abae-a3f376686970-maas-api-xks-ca\") pod \"46c5a642-782c-4f47-abae-a3f376686970\" (UID: \"46c5a642-782c-4f47-abae-a3f376686970\") "
Nov 03 01:38:39.070002 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.069958    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c5a642-782c-4f47-abae-a3f376686970-maas-api-openshift-ca" pod "46c5a642-782c-4f47-abae-a3f376686970" (UID: "46c5a642-782c-4f47-abae-a3f376686970"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:38:39.070223 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.070197    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c5a642-782c-4f47-abae-a3f376686970-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:39.071908 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.071844    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46c5a642-782c-4f47-abae-a3f376686970-maas-api-xks-ca" pod "46c5a642-782c-4f47-abae-a3f376686970" (UID: "46c5a642-782c-4f47-abae-a3f376686970"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:38:39.072011 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.071919    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c5a642-782c-4f47-abae-a3f376686970-kube-api-access-7bbt8" pod "46c5a642-782c-4f47-abae-a3f376686970" (UID: "46c5a642-782c-4f47-abae-a3f376686970"). InnerVolumeSpecName "kube-api-access-7bbt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:39.171123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.171084    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c5a642-782c-4f47-abae-a3f376686970-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:39.171123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.171124    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bbt8\" (UniqueName: \"kubernetes.io/projected/46c5a642-782c-4f47-abae-a3f376686970-kube-api-access-7bbt8\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:39.810792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.810760    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"]
Nov 03 01:38:39.811262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.811249    5508 cpu_manager.go:414] "containerMap: removing container" podUID="46c5a642-782c-4f47-abae-a3f376686970" containerName="revoke-keys"
Nov 03 01:38:39.811320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.811263    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c5a642-782c-4f47-abae-a3f376686970" containerName="revoke-keys"
Nov 03 01:38:39.811374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.811356    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c5a642-782c-4f47-abae-a3f376686970" containerName="revoke-keys"
Nov 03 01:38:39.814793 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.814773    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"
Nov 03 01:38:39.829948 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.829926    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"
Nov 03 01:38:39.830299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.829960    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7" event={"ID":"46c5a642-782c-4f47-abae-a3f376686970","Type":"ContainerDied","Data":"0ab36e1370240b6113f071e5c7834fa74275edeff82c2a1eaa7e318cf702d14d"}
Nov 03 01:38:39.830299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.830003    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ab36e1370240b6113f071e5c7834fa74275edeff82c2a1eaa7e318cf702d14d"
Nov 03 01:38:39.868509 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.868471    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"]
Nov 03 01:38:39.881101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.881067    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62c1bd13-mswzq\" (UID: \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"
Nov 03 01:38:39.881271 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.881244    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62c1bd13-mswzq\" (UID: \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"
Nov 03 01:38:39.881416 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.881400    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr5cw\" (UniqueName: \"kubernetes.io/projected/439114f0-b0e2-4932-b9c4-1f7edd63df2f-kube-api-access-jr5cw\") pod \"maas-api-revoke-sub-62c1bd13-mswzq\" (UID: \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"
Nov 03 01:38:39.982357 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.982317    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62c1bd13-mswzq\" (UID: \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"
Nov 03 01:38:39.982489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.982456    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62c1bd13-mswzq\" (UID: \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"
Nov 03 01:38:39.982588 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.982572    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr5cw\" (UniqueName: \"kubernetes.io/projected/439114f0-b0e2-4932-b9c4-1f7edd63df2f-kube-api-access-jr5cw\") pod \"maas-api-revoke-sub-62c1bd13-mswzq\" (UID: \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"
Nov 03 01:38:39.983004 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.982981    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62c1bd13-mswzq\" (UID: \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"
Nov 03 01:38:39.984872 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.984853    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62c1bd13-mswzq\" (UID: \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"
Nov 03 01:38:39.987397 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:39.987382    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"
Nov 03 01:38:40.041200 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.041167    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr5cw\" (UniqueName: \"kubernetes.io/projected/439114f0-b0e2-4932-b9c4-1f7edd63df2f-kube-api-access-jr5cw\") pod \"maas-api-revoke-sub-62c1bd13-mswzq\" (UID: \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"
Nov 03 01:38:40.083236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.083171    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784a5f88-f63f-41be-afce-9dddb8dc2edb-kube-api-access-4vx6s\" (UniqueName: \"kubernetes.io/projected/784a5f88-f63f-41be-afce-9dddb8dc2edb-kube-api-access-4vx6s\") pod \"784a5f88-f63f-41be-afce-9dddb8dc2edb\" (UID: \"784a5f88-f63f-41be-afce-9dddb8dc2edb\") "
Nov 03 01:38:40.083380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.083268    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-xks-ca\") pod \"784a5f88-f63f-41be-afce-9dddb8dc2edb\" (UID: \"784a5f88-f63f-41be-afce-9dddb8dc2edb\") "
Nov 03 01:38:40.083428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.083410    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-openshift-ca\") pod \"784a5f88-f63f-41be-afce-9dddb8dc2edb\" (UID: \"784a5f88-f63f-41be-afce-9dddb8dc2edb\") "
Nov 03 01:38:40.083747 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.083717    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-openshift-ca" pod "784a5f88-f63f-41be-afce-9dddb8dc2edb" (UID: "784a5f88-f63f-41be-afce-9dddb8dc2edb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:38:40.085158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.085136    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-xks-ca" pod "784a5f88-f63f-41be-afce-9dddb8dc2edb" (UID: "784a5f88-f63f-41be-afce-9dddb8dc2edb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:38:40.085234 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.085186    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784a5f88-f63f-41be-afce-9dddb8dc2edb-kube-api-access-4vx6s" pod "784a5f88-f63f-41be-afce-9dddb8dc2edb" (UID: "784a5f88-f63f-41be-afce-9dddb8dc2edb"). InnerVolumeSpecName "kube-api-access-4vx6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:40.129270 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.129240    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"
Nov 03 01:38:40.185239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.185209    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:40.185381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.185248    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/784a5f88-f63f-41be-afce-9dddb8dc2edb-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:40.185381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.185264    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vx6s\" (UniqueName: \"kubernetes.io/projected/784a5f88-f63f-41be-afce-9dddb8dc2edb-kube-api-access-4vx6s\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:40.503205 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.503166    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"]
Nov 03 01:38:40.509658 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:38:40.509626    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439114f0_b0e2_4932_b9c4_1f7edd63df2f.slice/crio-15e3c9e77df4aa67c13efe8a20c05c4eff42d692cce2e3b0af8498c72fd53d00 WatchSource:0}: Error finding container 15e3c9e77df4aa67c13efe8a20c05c4eff42d692cce2e3b0af8498c72fd53d00: Status 404 returned error can't find the container with id 15e3c9e77df4aa67c13efe8a20c05c4eff42d692cce2e3b0af8498c72fd53d00
Nov 03 01:38:40.839561 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.839473    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"
Nov 03 01:38:40.839561 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.839507    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff" event={"ID":"784a5f88-f63f-41be-afce-9dddb8dc2edb","Type":"ContainerDied","Data":"395f00ca1adb2745636152912a549e351f585856c8aad08b469f1cc4c02b936d"}
Nov 03 01:38:40.840081 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.839563    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="395f00ca1adb2745636152912a549e351f585856c8aad08b469f1cc4c02b936d"
Nov 03 01:38:40.842317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.842288    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq" event={"ID":"439114f0-b0e2-4932-b9c4-1f7edd63df2f","Type":"ContainerStarted","Data":"0467a80b09d045b84d525338df7bc48ba08f1a21ccec30dcf9c03acee2841a50"}
Nov 03 01:38:40.842655 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.842342    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq" event={"ID":"439114f0-b0e2-4932-b9c4-1f7edd63df2f","Type":"ContainerStarted","Data":"15e3c9e77df4aa67c13efe8a20c05c4eff42d692cce2e3b0af8498c72fd53d00"}
Nov 03 01:38:40.861797 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.861772    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:38:40.862656 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:40.862636    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:38:41.000945 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:41.000894    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq" podStartSLOduration=2.000881185 podStartE2EDuration="2.000881185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:40.997890711 +0000 UTC m=+1200.423497176" watchObservedRunningTime="2026-11-03 01:38:41.000881185 +0000 UTC m=+1200.426487649"
Nov 03 01:38:41.850415 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:41.850371    5508 generic.go:356] "Generic (PLEG): container finished" podID="439114f0-b0e2-4932-b9c4-1f7edd63df2f" containerID="0467a80b09d045b84d525338df7bc48ba08f1a21ccec30dcf9c03acee2841a50" exitCode=0
Nov 03 01:38:41.850850 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:41.850445    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq" event={"ID":"439114f0-b0e2-4932-b9c4-1f7edd63df2f","Type":"ContainerDied","Data":"0467a80b09d045b84d525338df7bc48ba08f1a21ccec30dcf9c03acee2841a50"}
Nov 03 01:38:43.000232 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:43.000206    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"
Nov 03 01:38:43.115589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:43.115504    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-xks-ca\") pod \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\" (UID: \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\") "
Nov 03 01:38:43.115589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:43.115555    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439114f0-b0e2-4932-b9c4-1f7edd63df2f-kube-api-access-jr5cw\" (UniqueName: \"kubernetes.io/projected/439114f0-b0e2-4932-b9c4-1f7edd63df2f-kube-api-access-jr5cw\") pod \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\" (UID: \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\") "
Nov 03 01:38:43.115785 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:43.115771    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-openshift-ca\") pod \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\" (UID: \"439114f0-b0e2-4932-b9c4-1f7edd63df2f\") "
Nov 03 01:38:43.116101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:43.116072    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-openshift-ca" pod "439114f0-b0e2-4932-b9c4-1f7edd63df2f" (UID: "439114f0-b0e2-4932-b9c4-1f7edd63df2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:38:43.116233 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:43.116157    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:43.117602 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:43.117581    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-xks-ca" pod "439114f0-b0e2-4932-b9c4-1f7edd63df2f" (UID: "439114f0-b0e2-4932-b9c4-1f7edd63df2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:38:43.117731 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:43.117714    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439114f0-b0e2-4932-b9c4-1f7edd63df2f-kube-api-access-jr5cw" pod "439114f0-b0e2-4932-b9c4-1f7edd63df2f" (UID: "439114f0-b0e2-4932-b9c4-1f7edd63df2f"). InnerVolumeSpecName "kube-api-access-jr5cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:43.224370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:43.219687    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/439114f0-b0e2-4932-b9c4-1f7edd63df2f-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:43.224370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:43.219730    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr5cw\" (UniqueName: \"kubernetes.io/projected/439114f0-b0e2-4932-b9c4-1f7edd63df2f-kube-api-access-jr5cw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:43.811752 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:38:43.811718    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:38:43.864668 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:43.864639    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"
Nov 03 01:38:43.864844 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:43.864672    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq" event={"ID":"439114f0-b0e2-4932-b9c4-1f7edd63df2f","Type":"ContainerDied","Data":"15e3c9e77df4aa67c13efe8a20c05c4eff42d692cce2e3b0af8498c72fd53d00"}
Nov 03 01:38:43.864844 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:43.864718    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15e3c9e77df4aa67c13efe8a20c05c4eff42d692cce2e3b0af8498c72fd53d00"
Nov 03 01:38:46.591024 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.590988    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"]
Nov 03 01:38:46.591801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.591767    5508 cpu_manager.go:414] "containerMap: removing container" podUID="784a5f88-f63f-41be-afce-9dddb8dc2edb" containerName="revoke-keys"
Nov 03 01:38:46.591801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.591793    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784a5f88-f63f-41be-afce-9dddb8dc2edb" containerName="revoke-keys"
Nov 03 01:38:46.591962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.591807    5508 cpu_manager.go:414] "containerMap: removing container" podUID="439114f0-b0e2-4932-b9c4-1f7edd63df2f" containerName="revoke-keys"
Nov 03 01:38:46.591962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.591814    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439114f0-b0e2-4932-b9c4-1f7edd63df2f" containerName="revoke-keys"
Nov 03 01:38:46.592060 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.591962    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="784a5f88-f63f-41be-afce-9dddb8dc2edb" containerName="revoke-keys"
Nov 03 01:38:46.592060 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.591975    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="439114f0-b0e2-4932-b9c4-1f7edd63df2f" containerName="revoke-keys"
Nov 03 01:38:46.595596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.595577    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.703899 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.703862    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"]
Nov 03 01:38:46.762485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.762442    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.762712 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.762510    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.762712 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.762585    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.762712 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.762618    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f82j8\" (UniqueName: \"kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-kube-api-access-f82j8\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.762712 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.762643    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e25b47-aaa7-4776-8757-c242e4441da2-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.762918 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.762743    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.762918 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.762767    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.762918 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.762789    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.762918 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.762872    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.864309 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.864222    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.864485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.864338    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.864485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.864423    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.864485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.864451    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.864485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.864474    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f82j8\" (UniqueName: \"kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-kube-api-access-f82j8\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.864684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.864503    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e25b47-aaa7-4776-8757-c242e4441da2-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.864684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.864575    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.864684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.864609    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.864684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.864644    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.864884 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.864694    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.864884 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.864840    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.865243 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.865197    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.865243 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.865214    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.865551 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.865527    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e25b47-aaa7-4776-8757-c242e4441da2-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.866902 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.866874    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.867020 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.866999    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.921688 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.921659    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:46.921823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:46.921801    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f82j8\" (UniqueName: \"kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-kube-api-access-f82j8\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:47.211077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:47.211043    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:47.643731 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:47.643649    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"]
Nov 03 01:38:47.650768 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:38:47.650740    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e25b47_aaa7_4776_8757_c242e4441da2.slice/crio-40993728c0504d0cdbc4c184f7b393d7cda010443619cc1c702d80e1ba7ea12d WatchSource:0}: Error finding container 40993728c0504d0cdbc4c184f7b393d7cda010443619cc1c702d80e1ba7ea12d: Status 404 returned error can't find the container with id 40993728c0504d0cdbc4c184f7b393d7cda010443619cc1c702d80e1ba7ea12d
Nov 03 01:38:47.653536 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:47.653500    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:38:47.653626 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:47.653582    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:38:47.653666 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:47.653625    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:38:47.895093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:47.894995    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" event={"ID":"a0e25b47-aaa7-4776-8757-c242e4441da2","Type":"ContainerStarted","Data":"8abaf68244b8e32bf3c6292a0848f65a116b03320e1e13b7eae49869635964d0"}
Nov 03 01:38:47.895093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:47.895059    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" event={"ID":"a0e25b47-aaa7-4776-8757-c242e4441da2","Type":"ContainerStarted","Data":"40993728c0504d0cdbc4c184f7b393d7cda010443619cc1c702d80e1ba7ea12d"}
Nov 03 01:38:48.029374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:48.020810    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" podStartSLOduration=2.020787199 podStartE2EDuration="2.020787199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:48.017182916 +0000 UTC m=+1207.442789381" watchObservedRunningTime="2026-11-03 01:38:48.020787199 +0000 UTC m=+1207.446393685"
Nov 03 01:38:48.211675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:48.211640    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:48.213032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:48.213007    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 01:38:48.213180 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:48.213052    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" podUID="a0e25b47-aaa7-4776-8757-c242e4441da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 01:38:49.211824 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:49.211785    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 01:38:49.212228 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:49.211847    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" podUID="a0e25b47-aaa7-4776-8757-c242e4441da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 01:38:50.212388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:50.211871    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 01:38:50.212388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:50.211942    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" podUID="a0e25b47-aaa7-4776-8757-c242e4441da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 01:38:51.211641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.211595    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 01:38:51.211834 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.211674    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" podUID="a0e25b47-aaa7-4776-8757-c242e4441da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 01:38:51.838658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.838628    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"]
Nov 03 01:38:51.842509 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.842488    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"
Nov 03 01:38:51.889917 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.889885    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e67cf21-2hlq2\" (UID: \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"
Nov 03 01:38:51.890060 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.890024    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lnng\" (UniqueName: \"kubernetes.io/projected/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-kube-api-access-7lnng\") pod \"maas-api-revoke-sub-1e67cf21-2hlq2\" (UID: \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"
Nov 03 01:38:51.890138 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.890123    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e67cf21-2hlq2\" (UID: \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"
Nov 03 01:38:51.892604 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.892580    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:38:51.938811 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.938773    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:38:51.938811 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.938800    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:38:51.939019 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.938779    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:38:51.942184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.942164    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"]
Nov 03 01:38:51.991662 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.991631    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e67cf21-2hlq2\" (UID: \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"
Nov 03 01:38:51.991853 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.991707    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e67cf21-2hlq2\" (UID: \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"
Nov 03 01:38:51.991853 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.991823    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lnng\" (UniqueName: \"kubernetes.io/projected/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-kube-api-access-7lnng\") pod \"maas-api-revoke-sub-1e67cf21-2hlq2\" (UID: \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"
Nov 03 01:38:51.992442 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.992419    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e67cf21-2hlq2\" (UID: \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"
Nov 03 01:38:51.994104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:51.994081    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e67cf21-2hlq2\" (UID: \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"
Nov 03 01:38:52.079370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:52.076036    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lnng\" (UniqueName: \"kubernetes.io/projected/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-kube-api-access-7lnng\") pod \"maas-api-revoke-sub-1e67cf21-2hlq2\" (UID: \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"
Nov 03 01:38:52.180402 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:52.180368    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"
Nov 03 01:38:52.212320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:52.212287    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 01:38:52.212492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:52.212365    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" podUID="a0e25b47-aaa7-4776-8757-c242e4441da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 01:38:52.576703 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:52.576159    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"]
Nov 03 01:38:52.579755 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:38:52.579709    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4b93a1_df1e_4b40_8dab_f2547f2a1224.slice/crio-3b2318afb646817b33dfe257ecea5e9544939f046d74d59355579014bd380585 WatchSource:0}: Error finding container 3b2318afb646817b33dfe257ecea5e9544939f046d74d59355579014bd380585: Status 404 returned error can't find the container with id 3b2318afb646817b33dfe257ecea5e9544939f046d74d59355579014bd380585
Nov 03 01:38:52.992818 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:52.992783    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2" event={"ID":"2a4b93a1-df1e-4b40-8dab-f2547f2a1224","Type":"ContainerStarted","Data":"3b2318afb646817b33dfe257ecea5e9544939f046d74d59355579014bd380585"}
Nov 03 01:38:53.211422 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:53.211388    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 01:38:53.211597 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:53.211443    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" podUID="a0e25b47-aaa7-4776-8757-c242e4441da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 01:38:53.877084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:53.877049    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"]
Nov 03 01:38:53.880931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:53.880916    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"
Nov 03 01:38:53.926177 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:53.926102    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daa5bab2-qzk88\" (UID: \"30ed081b-fb60-4650-8e62-b162c9053888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"
Nov 03 01:38:53.926313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:53.926177    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daa5bab2-qzk88\" (UID: \"30ed081b-fb60-4650-8e62-b162c9053888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"
Nov 03 01:38:53.926313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:53.926234    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z75gk\" (UniqueName: \"kubernetes.io/projected/30ed081b-fb60-4650-8e62-b162c9053888-kube-api-access-z75gk\") pod \"maas-api-revoke-sub-daa5bab2-qzk88\" (UID: \"30ed081b-fb60-4650-8e62-b162c9053888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"
Nov 03 01:38:53.932812 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:53.932784    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"]
Nov 03 01:38:54.002702 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.002661    5508 generic.go:356] "Generic (PLEG): container finished" podID="2a4b93a1-df1e-4b40-8dab-f2547f2a1224" containerID="9d3f6b8fb4bcb8daca00f3a9a70d41b444dee7486dddf980eb7ad251df08860e" exitCode=0
Nov 03 01:38:54.003208 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.002746    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2" event={"ID":"2a4b93a1-df1e-4b40-8dab-f2547f2a1224","Type":"ContainerDied","Data":"9d3f6b8fb4bcb8daca00f3a9a70d41b444dee7486dddf980eb7ad251df08860e"}
Nov 03 01:38:54.027309 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.027268    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daa5bab2-qzk88\" (UID: \"30ed081b-fb60-4650-8e62-b162c9053888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"
Nov 03 01:38:54.027531 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.027329    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z75gk\" (UniqueName: \"kubernetes.io/projected/30ed081b-fb60-4650-8e62-b162c9053888-kube-api-access-z75gk\") pod \"maas-api-revoke-sub-daa5bab2-qzk88\" (UID: \"30ed081b-fb60-4650-8e62-b162c9053888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"
Nov 03 01:38:54.027531 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.027440    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daa5bab2-qzk88\" (UID: \"30ed081b-fb60-4650-8e62-b162c9053888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"
Nov 03 01:38:54.028095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.028072    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daa5bab2-qzk88\" (UID: \"30ed081b-fb60-4650-8e62-b162c9053888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"
Nov 03 01:38:54.029617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.029586    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daa5bab2-qzk88\" (UID: \"30ed081b-fb60-4650-8e62-b162c9053888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"
Nov 03 01:38:54.084392 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.084363    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z75gk\" (UniqueName: \"kubernetes.io/projected/30ed081b-fb60-4650-8e62-b162c9053888-kube-api-access-z75gk\") pod \"maas-api-revoke-sub-daa5bab2-qzk88\" (UID: \"30ed081b-fb60-4650-8e62-b162c9053888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"
Nov 03 01:38:54.193778 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.193696    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"
Nov 03 01:38:54.211795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.211771    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 01:38:54.211903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.211823    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" podUID="a0e25b47-aaa7-4776-8757-c242e4441da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 01:38:54.573721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.573645    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"]
Nov 03 01:38:54.579936 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:38:54.579910    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ed081b_fb60_4650_8e62_b162c9053888.slice/crio-64f5ab9bdd2bc2662cd50fdb3730c15de44b68be7fe825f47c9e0c0b78a2390d WatchSource:0}: Error finding container 64f5ab9bdd2bc2662cd50fdb3730c15de44b68be7fe825f47c9e0c0b78a2390d: Status 404 returned error can't find the container with id 64f5ab9bdd2bc2662cd50fdb3730c15de44b68be7fe825f47c9e0c0b78a2390d
Nov 03 01:38:54.830558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.830465    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"]
Nov 03 01:38:54.835712 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.835694    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:54.938282 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.938222    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8spth\" (UniqueName: \"kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-kube-api-access-8spth\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:54.938502 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.938472    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:54.941878 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.941368    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:54.941878 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.941435    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:54.941878 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.941493    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:54.941878 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.941573    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:54.941878 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.941599    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:54.941878 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.941627    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:54.941878 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.941699    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:54.979564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:54.979531    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"]
Nov 03 01:38:55.014485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.014448    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88" event={"ID":"30ed081b-fb60-4650-8e62-b162c9053888","Type":"ContainerStarted","Data":"95aa91b6484b637d674c1969f6340edf8dd39957cac33bb83bc2e67c44d7d071"}
Nov 03 01:38:55.014964 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.014503    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88" event={"ID":"30ed081b-fb60-4650-8e62-b162c9053888","Type":"ContainerStarted","Data":"64f5ab9bdd2bc2662cd50fdb3730c15de44b68be7fe825f47c9e0c0b78a2390d"}
Nov 03 01:38:55.045367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.042670    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.045367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.042724    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.045367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.042753    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.045367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.042812    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.045367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.042864    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8spth\" (UniqueName: \"kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-kube-api-access-8spth\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.045367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.042955    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.045367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.043026    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.045367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.043076    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.045367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.043125    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.045367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.045020    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.045367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.045256    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.049369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.046583    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.049369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.046836    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.049369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.047503    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.049613 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.049460    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.050905 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.050876    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.103700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.103667    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.104008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.103983    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8spth\" (UniqueName: \"kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-kube-api-access-8spth\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.149771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.146599    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88" podStartSLOduration=2.14657954 podStartE2EDuration="2.14657954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:55.135957788 +0000 UTC m=+1214.561564276" watchObservedRunningTime="2026-11-03 01:38:55.14657954 +0000 UTC m=+1214.572186006"
Nov 03 01:38:55.149771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.149575    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:55.187991 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.187967    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"
Nov 03 01:38:55.252089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.251429    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-xks-ca\") pod \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\" (UID: \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\") "
Nov 03 01:38:55.252089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.251543    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-kube-api-access-7lnng\" (UniqueName: \"kubernetes.io/projected/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-kube-api-access-7lnng\") pod \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\" (UID: \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\") "
Nov 03 01:38:55.252089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.251645    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-openshift-ca\") pod \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\" (UID: \"2a4b93a1-df1e-4b40-8dab-f2547f2a1224\") "
Nov 03 01:38:55.254743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.252798    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-openshift-ca" pod "2a4b93a1-df1e-4b40-8dab-f2547f2a1224" (UID: "2a4b93a1-df1e-4b40-8dab-f2547f2a1224"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:38:55.258905 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.258777    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-xks-ca" pod "2a4b93a1-df1e-4b40-8dab-f2547f2a1224" (UID: "2a4b93a1-df1e-4b40-8dab-f2547f2a1224"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:38:55.258905 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.258880    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-kube-api-access-7lnng" pod "2a4b93a1-df1e-4b40-8dab-f2547f2a1224" (UID: "2a4b93a1-df1e-4b40-8dab-f2547f2a1224"). InnerVolumeSpecName "kube-api-access-7lnng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:55.353892 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.353807    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:55.353892 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.353839    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:55.353892 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.353850    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lnng\" (UniqueName: \"kubernetes.io/projected/2a4b93a1-df1e-4b40-8dab-f2547f2a1224-kube-api-access-7lnng\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:55.543570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.543540    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"]
Nov 03 01:38:55.570421 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:38:55.570385    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bff3dbb_1081_4c8d_a558_d8db9ea4a2c2.slice/crio-ca10a8663d88a61959d525b16720d5ca847059a7ef34ef7e0247e24ad69739af WatchSource:0}: Error finding container ca10a8663d88a61959d525b16720d5ca847059a7ef34ef7e0247e24ad69739af: Status 404 returned error can't find the container with id ca10a8663d88a61959d525b16720d5ca847059a7ef34ef7e0247e24ad69739af
Nov 03 01:38:55.574103 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.574064    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:38:55.574202 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.574160    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:38:55.574253 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:55.574199    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:38:56.035018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.034966    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" event={"ID":"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2","Type":"ContainerStarted","Data":"ed9656e1e22e25b2a91db70830a895a241a81f5c14127c35e13a2a02c3b2da92"}
Nov 03 01:38:56.035606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.035075    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" event={"ID":"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2","Type":"ContainerStarted","Data":"ca10a8663d88a61959d525b16720d5ca847059a7ef34ef7e0247e24ad69739af"}
Nov 03 01:38:56.036675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.036650    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2" event={"ID":"2a4b93a1-df1e-4b40-8dab-f2547f2a1224","Type":"ContainerDied","Data":"3b2318afb646817b33dfe257ecea5e9544939f046d74d59355579014bd380585"}
Nov 03 01:38:56.036771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.036673    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"
Nov 03 01:38:56.036771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.036694    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b2318afb646817b33dfe257ecea5e9544939f046d74d59355579014bd380585"
Nov 03 01:38:56.038278 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.038255    5508 generic.go:356] "Generic (PLEG): container finished" podID="30ed081b-fb60-4650-8e62-b162c9053888" containerID="95aa91b6484b637d674c1969f6340edf8dd39957cac33bb83bc2e67c44d7d071" exitCode=0
Nov 03 01:38:56.038372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.038338    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88" event={"ID":"30ed081b-fb60-4650-8e62-b162c9053888","Type":"ContainerDied","Data":"95aa91b6484b637d674c1969f6340edf8dd39957cac33bb83bc2e67c44d7d071"}
Nov 03 01:38:56.158374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.151740    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:38:56.158374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.153595    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 01:38:56.158374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.153654    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 01:38:56.179372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.175063    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" podStartSLOduration=2.17504253 podStartE2EDuration="2.17504253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:56.152089034 +0000 UTC m=+1215.577695535" watchObservedRunningTime="2026-11-03 01:38:56.17504253 +0000 UTC m=+1215.600648995"
Nov 03 01:38:56.261376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.256467    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:38:56.261376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.256554    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" podUID="a0e25b47-aaa7-4776-8757-c242e4441da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:38:56.694519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.694489    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:56.694986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.694975    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:56.695800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:56.695778    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:38:57.179365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:57.175454    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 01:38:57.179365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:57.175515    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 01:38:57.243148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:57.243125    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"
Nov 03 01:38:57.402360 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:57.402313    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-openshift-ca\") pod \"30ed081b-fb60-4650-8e62-b162c9053888\" (UID: \"30ed081b-fb60-4650-8e62-b162c9053888\") "
Nov 03 01:38:57.402561 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:57.402413    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ed081b-fb60-4650-8e62-b162c9053888-kube-api-access-z75gk\" (UniqueName: \"kubernetes.io/projected/30ed081b-fb60-4650-8e62-b162c9053888-kube-api-access-z75gk\") pod \"30ed081b-fb60-4650-8e62-b162c9053888\" (UID: \"30ed081b-fb60-4650-8e62-b162c9053888\") "
Nov 03 01:38:57.402561 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:57.402484    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-xks-ca\") pod \"30ed081b-fb60-4650-8e62-b162c9053888\" (UID: \"30ed081b-fb60-4650-8e62-b162c9053888\") "
Nov 03 01:38:57.402747 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:57.402724    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-openshift-ca" pod "30ed081b-fb60-4650-8e62-b162c9053888" (UID: "30ed081b-fb60-4650-8e62-b162c9053888"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:38:57.403041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:57.403014    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:57.405098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:57.405069    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-xks-ca" pod "30ed081b-fb60-4650-8e62-b162c9053888" (UID: "30ed081b-fb60-4650-8e62-b162c9053888"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:38:57.405209 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:57.405069    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ed081b-fb60-4650-8e62-b162c9053888-kube-api-access-z75gk" pod "30ed081b-fb60-4650-8e62-b162c9053888" (UID: "30ed081b-fb60-4650-8e62-b162c9053888"). InnerVolumeSpecName "kube-api-access-z75gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:57.504426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:57.504388    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z75gk\" (UniqueName: \"kubernetes.io/projected/30ed081b-fb60-4650-8e62-b162c9053888-kube-api-access-z75gk\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:57.504426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:57.504421    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ed081b-fb60-4650-8e62-b162c9053888-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:38:58.116136 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:58.116100    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"
Nov 03 01:38:58.123495 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:58.123456    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88" event={"ID":"30ed081b-fb60-4650-8e62-b162c9053888","Type":"ContainerDied","Data":"64f5ab9bdd2bc2662cd50fdb3730c15de44b68be7fe825f47c9e0c0b78a2390d"}
Nov 03 01:38:58.123802 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:58.123783    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64f5ab9bdd2bc2662cd50fdb3730c15de44b68be7fe825f47c9e0c0b78a2390d"
Nov 03 01:38:58.154789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:58.154753    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 01:38:58.155085 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:58.155052    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 01:38:59.153708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:59.153670    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 01:38:59.154331 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:38:59.154299    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 01:39:00.046215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:00.046180    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"]
Nov 03 01:39:00.094549 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:00.094497    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd41e900-kp2gj"]
Nov 03 01:39:00.173976 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:00.173936    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 01:39:00.174623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:00.174589    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 01:39:00.691121 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:00.691083    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12b0837-6877-4748-bae8-00ecb45fb7f6" path="/var/lib/kubelet/pods/c12b0837-6877-4748-bae8-00ecb45fb7f6/volumes"
Nov 03 01:39:01.156741 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:01.156701    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 01:39:01.157007 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:01.156976    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 01:39:02.151935 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:02.151899    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 01:39:02.152423 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:02.151953    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 01:39:02.318414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:02.318379    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:39:02.891473 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:02.891437    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"]
Nov 03 01:39:02.891925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:02.891913    5508 cpu_manager.go:414] "containerMap: removing container" podUID="30ed081b-fb60-4650-8e62-b162c9053888" containerName="revoke-keys"
Nov 03 01:39:02.891964 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:02.891927    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ed081b-fb60-4650-8e62-b162c9053888" containerName="revoke-keys"
Nov 03 01:39:02.891964 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:02.891943    5508 cpu_manager.go:414] "containerMap: removing container" podUID="2a4b93a1-df1e-4b40-8dab-f2547f2a1224" containerName="revoke-keys"
Nov 03 01:39:02.891964 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:02.891947    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4b93a1-df1e-4b40-8dab-f2547f2a1224" containerName="revoke-keys"
Nov 03 01:39:02.892076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:02.892038    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ed081b-fb60-4650-8e62-b162c9053888" containerName="revoke-keys"
Nov 03 01:39:02.892076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:02.892047    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4b93a1-df1e-4b40-8dab-f2547f2a1224" containerName="revoke-keys"
Nov 03 01:39:02.896309 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:02.896287    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.015372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.009231    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"]
Nov 03 01:39:03.028098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.028058    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.028503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.028480    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.028678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.028663    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.028821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.028809    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsh92\" (UniqueName: \"kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-kube-api-access-gsh92\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.028935 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.028923    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9367258-844a-46ac-a685-3be44068b879-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.029051 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.029038    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.029205 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.029193    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.029302 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.029282    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.029429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.029415    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9367258-844a-46ac-a685-3be44068b879-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.161841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.160947    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.161841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.161003    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.161841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.161034    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9367258-844a-46ac-a685-3be44068b879-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.161841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.161125    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.161841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.161168    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.161841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.161197    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.161841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.161272    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsh92\" (UniqueName: \"kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-kube-api-access-gsh92\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.161841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.161363    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9367258-844a-46ac-a685-3be44068b879-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.161841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.161420    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.163851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.163825    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.164480 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.164119    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.164480 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.164328    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.167372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.164928    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.167372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.165967    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9367258-844a-46ac-a685-3be44068b879-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.173829 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.173761    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9367258-844a-46ac-a685-3be44068b879-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.174874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.174854    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.219376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.219309    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.222012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.221987    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsh92\" (UniqueName: \"kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-kube-api-access-gsh92\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:03.514419 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:03.512572    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:04.053381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:04.053191    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:04.053381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:04.053283    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:04.057428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:04.053577    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:04.097266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:04.097227    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"]
Nov 03 01:39:04.195432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:04.195393    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf" event={"ID":"b9367258-844a-46ac-a685-3be44068b879","Type":"ContainerStarted","Data":"db7683f8a3d938611e632b1c79d96f174fae72b3d66f63ddd604942dbf56ad55"}
Nov 03 01:39:04.203117 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:04.202499    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:39:04.203117 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:04.202571    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:04.838365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:04.830615    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"]
Nov 03 01:39:04.880748 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:04.880717    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"
Nov 03 01:39:04.920531 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:04.902541    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"]
Nov 03 01:39:04.937629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:04.932675    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:39:04.937629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:04.932923    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:39:04.937629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:04.932977    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:39:05.107427 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.015423    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:39:05.352417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.336910    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:39:05.352417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.337561    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:05.539371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.469199    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:39:05.539371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.482005    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c17edc5-85sdv\" (UID: \"6f28066b-46f4-4ce3-8e94-76442f440e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"
Nov 03 01:39:05.539371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.482087    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c17edc5-85sdv\" (UID: \"6f28066b-46f4-4ce3-8e94-76442f440e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"
Nov 03 01:39:05.539371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.482276    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7mwt\" (UniqueName: \"kubernetes.io/projected/6f28066b-46f4-4ce3-8e94-76442f440e3f-kube-api-access-w7mwt\") pod \"maas-api-revoke-sub-0c17edc5-85sdv\" (UID: \"6f28066b-46f4-4ce3-8e94-76442f440e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"
Nov 03 01:39:05.559177 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.559138    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:39:05.591622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.590494    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:39:05.591622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.590624    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:05.651815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.651784    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:39:05.681311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.681265    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c17edc5-85sdv\" (UID: \"6f28066b-46f4-4ce3-8e94-76442f440e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"
Nov 03 01:39:05.681837 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.681810    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c17edc5-85sdv\" (UID: \"6f28066b-46f4-4ce3-8e94-76442f440e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"
Nov 03 01:39:05.682407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.682247    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c17edc5-85sdv\" (UID: \"6f28066b-46f4-4ce3-8e94-76442f440e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"
Nov 03 01:39:05.689930 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.689900    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c17edc5-85sdv\" (UID: \"6f28066b-46f4-4ce3-8e94-76442f440e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"
Nov 03 01:39:05.694577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.694529    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf" event={"ID":"b9367258-844a-46ac-a685-3be44068b879","Type":"ContainerStarted","Data":"c9113027a1dcdd7d89d9b4b65e2b846948fa08db9fe5d7c92f5a755313dd9b5f"}
Nov 03 01:39:05.696137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.696109    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7mwt\" (UniqueName: \"kubernetes.io/projected/6f28066b-46f4-4ce3-8e94-76442f440e3f-kube-api-access-w7mwt\") pod \"maas-api-revoke-sub-0c17edc5-85sdv\" (UID: \"6f28066b-46f4-4ce3-8e94-76442f440e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"
Nov 03 01:39:05.767761 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.767692    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7mwt\" (UniqueName: \"kubernetes.io/projected/6f28066b-46f4-4ce3-8e94-76442f440e3f-kube-api-access-w7mwt\") pod \"maas-api-revoke-sub-0c17edc5-85sdv\" (UID: \"6f28066b-46f4-4ce3-8e94-76442f440e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"
Nov 03 01:39:05.848755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.848669    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"
Nov 03 01:39:05.930201 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:05.929242    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf" podStartSLOduration=3.929220764 podStartE2EDuration="3.929220764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:05.926953364 +0000 UTC m=+1225.352559829" watchObservedRunningTime="2026-11-03 01:39:05.929220764 +0000 UTC m=+1225.354827229"
Nov 03 01:39:06.381489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:06.381453    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"]
Nov 03 01:39:06.381966 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:39:06.381543    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f28066b_46f4_4ce3_8e94_76442f440e3f.slice/crio-174525d9acd9e45114d78345e22ca59c8f3179916f41abaac14a15cadb1af424 WatchSource:0}: Error finding container 174525d9acd9e45114d78345e22ca59c8f3179916f41abaac14a15cadb1af424: Status 404 returned error can't find the container with id 174525d9acd9e45114d78345e22ca59c8f3179916f41abaac14a15cadb1af424
Nov 03 01:39:06.513906 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:06.513864    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:06.515627 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:06.515602    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 01:39:06.515737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:06.515648    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf" podUID="b9367258-844a-46ac-a685-3be44068b879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 01:39:06.707077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:06.707040    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv" event={"ID":"6f28066b-46f4-4ce3-8e94-76442f440e3f","Type":"ContainerStarted","Data":"174525d9acd9e45114d78345e22ca59c8f3179916f41abaac14a15cadb1af424"}
Nov 03 01:39:07.513935 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:07.513900    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 01:39:07.514334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:07.513955    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf" podUID="b9367258-844a-46ac-a685-3be44068b879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 01:39:07.715235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:07.715155    5508 generic.go:356] "Generic (PLEG): container finished" podID="6f28066b-46f4-4ce3-8e94-76442f440e3f" containerID="9e9d9512a26cfb04d0de269e0b7c0ecb7366e99a12c1497672638c92ec6da9db" exitCode=0
Nov 03 01:39:07.715415 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:07.715242    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv" event={"ID":"6f28066b-46f4-4ce3-8e94-76442f440e3f","Type":"ContainerDied","Data":"9e9d9512a26cfb04d0de269e0b7c0ecb7366e99a12c1497672638c92ec6da9db"}
Nov 03 01:39:08.052436 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:08.049580    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"]
Nov 03 01:39:08.073939 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:08.073909    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ff137f-vhffk"]
Nov 03 01:39:08.514113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:08.514083    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 01:39:08.514501 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:08.514136    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf" podUID="b9367258-844a-46ac-a685-3be44068b879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 01:39:08.693006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:08.692968    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b06f6e57-fb78-435a-aa0c-4d624111a8d8" path="/var/lib/kubelet/pods/b06f6e57-fb78-435a-aa0c-4d624111a8d8/volumes"
Nov 03 01:39:08.918487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:08.918459    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"
Nov 03 01:39:09.089250 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.089153    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f28066b-46f4-4ce3-8e94-76442f440e3f-kube-api-access-w7mwt\" (UniqueName: \"kubernetes.io/projected/6f28066b-46f4-4ce3-8e94-76442f440e3f-kube-api-access-w7mwt\") pod \"6f28066b-46f4-4ce3-8e94-76442f440e3f\" (UID: \"6f28066b-46f4-4ce3-8e94-76442f440e3f\") "
Nov 03 01:39:09.089250 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.089210    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-openshift-ca\") pod \"6f28066b-46f4-4ce3-8e94-76442f440e3f\" (UID: \"6f28066b-46f4-4ce3-8e94-76442f440e3f\") "
Nov 03 01:39:09.089506 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.089325    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-xks-ca\") pod \"6f28066b-46f4-4ce3-8e94-76442f440e3f\" (UID: \"6f28066b-46f4-4ce3-8e94-76442f440e3f\") "
Nov 03 01:39:09.089726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.089699    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-openshift-ca" pod "6f28066b-46f4-4ce3-8e94-76442f440e3f" (UID: "6f28066b-46f4-4ce3-8e94-76442f440e3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:09.089989 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.089970    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:09.092220 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.092194    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f28066b-46f4-4ce3-8e94-76442f440e3f-kube-api-access-w7mwt" pod "6f28066b-46f4-4ce3-8e94-76442f440e3f" (UID: "6f28066b-46f4-4ce3-8e94-76442f440e3f"). InnerVolumeSpecName "kube-api-access-w7mwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:09.092595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.092571    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-xks-ca" pod "6f28066b-46f4-4ce3-8e94-76442f440e3f" (UID: "6f28066b-46f4-4ce3-8e94-76442f440e3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:09.131670 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.131639    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"]
Nov 03 01:39:09.132544 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.132161    5508 cpu_manager.go:414] "containerMap: removing container" podUID="6f28066b-46f4-4ce3-8e94-76442f440e3f" containerName="revoke-keys"
Nov 03 01:39:09.132544 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.132176    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f28066b-46f4-4ce3-8e94-76442f440e3f" containerName="revoke-keys"
Nov 03 01:39:09.132544 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.132267    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f28066b-46f4-4ce3-8e94-76442f440e3f" containerName="revoke-keys"
Nov 03 01:39:09.136852 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.136833    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"
Nov 03 01:39:09.189245 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.189214    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"]
Nov 03 01:39:09.191084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.191055    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7mwt\" (UniqueName: \"kubernetes.io/projected/6f28066b-46f4-4ce3-8e94-76442f440e3f-kube-api-access-w7mwt\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:09.191084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.191084    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f28066b-46f4-4ce3-8e94-76442f440e3f-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:09.292558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.292519    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxlnw\" (UniqueName: \"kubernetes.io/projected/b97e649a-42c7-48d0-80f5-51f24407ee8b-kube-api-access-qxlnw\") pod \"maas-api-revoke-sub-8f6cd7ec-gl27f\" (UID: \"b97e649a-42c7-48d0-80f5-51f24407ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"
Nov 03 01:39:09.292743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.292584    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6cd7ec-gl27f\" (UID: \"b97e649a-42c7-48d0-80f5-51f24407ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"
Nov 03 01:39:09.292743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.292722    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6cd7ec-gl27f\" (UID: \"b97e649a-42c7-48d0-80f5-51f24407ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"
Nov 03 01:39:09.393675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.393587    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxlnw\" (UniqueName: \"kubernetes.io/projected/b97e649a-42c7-48d0-80f5-51f24407ee8b-kube-api-access-qxlnw\") pod \"maas-api-revoke-sub-8f6cd7ec-gl27f\" (UID: \"b97e649a-42c7-48d0-80f5-51f24407ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"
Nov 03 01:39:09.393675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.393667    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6cd7ec-gl27f\" (UID: \"b97e649a-42c7-48d0-80f5-51f24407ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"
Nov 03 01:39:09.393920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.393724    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6cd7ec-gl27f\" (UID: \"b97e649a-42c7-48d0-80f5-51f24407ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"
Nov 03 01:39:09.394400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.394338    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6cd7ec-gl27f\" (UID: \"b97e649a-42c7-48d0-80f5-51f24407ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"
Nov 03 01:39:09.396098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.396076    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6cd7ec-gl27f\" (UID: \"b97e649a-42c7-48d0-80f5-51f24407ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"
Nov 03 01:39:09.465381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.463386    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxlnw\" (UniqueName: \"kubernetes.io/projected/b97e649a-42c7-48d0-80f5-51f24407ee8b-kube-api-access-qxlnw\") pod \"maas-api-revoke-sub-8f6cd7ec-gl27f\" (UID: \"b97e649a-42c7-48d0-80f5-51f24407ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"
Nov 03 01:39:09.477064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.477026    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"
Nov 03 01:39:09.526406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.521512    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 01:39:09.526406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.521585    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf" podUID="b9367258-844a-46ac-a685-3be44068b879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 01:39:09.729836 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.729799    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"
Nov 03 01:39:09.729836 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.729828    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv" event={"ID":"6f28066b-46f4-4ce3-8e94-76442f440e3f","Type":"ContainerDied","Data":"174525d9acd9e45114d78345e22ca59c8f3179916f41abaac14a15cadb1af424"}
Nov 03 01:39:09.730066 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.729877    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="174525d9acd9e45114d78345e22ca59c8f3179916f41abaac14a15cadb1af424"
Nov 03 01:39:09.890506 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:09.885420    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"]
Nov 03 01:39:10.514294 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:10.514262    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 01:39:10.514499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:10.514318    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf" podUID="b9367258-844a-46ac-a685-3be44068b879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 01:39:10.738027 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:10.737991    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f" event={"ID":"b97e649a-42c7-48d0-80f5-51f24407ee8b","Type":"ContainerStarted","Data":"c47e7de2c1051f5e81141f8eceee757757aac202e1da35c24e19700354d6beac"}
Nov 03 01:39:10.738447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:10.738053    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f" event={"ID":"b97e649a-42c7-48d0-80f5-51f24407ee8b","Type":"ContainerStarted","Data":"46479aab49ea2b603904a5c06a604e5d324ccc5a8020853dec4dfced16b2f4ca"}
Nov 03 01:39:10.856435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:10.848410    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f" podStartSLOduration=1.848392355 podStartE2EDuration="1.848392355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:10.845857284 +0000 UTC m=+1230.271463769" watchObservedRunningTime="2026-11-03 01:39:10.848392355 +0000 UTC m=+1230.273998821"
Nov 03 01:39:11.514035 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:11.514001    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 01:39:11.514187 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:11.514055    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf" podUID="b9367258-844a-46ac-a685-3be44068b879" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 01:39:11.747221 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:11.747183    5508 generic.go:356] "Generic (PLEG): container finished" podID="b97e649a-42c7-48d0-80f5-51f24407ee8b" containerID="c47e7de2c1051f5e81141f8eceee757757aac202e1da35c24e19700354d6beac" exitCode=0
Nov 03 01:39:11.747886 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:11.747257    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f" event={"ID":"b97e649a-42c7-48d0-80f5-51f24407ee8b","Type":"ContainerDied","Data":"c47e7de2c1051f5e81141f8eceee757757aac202e1da35c24e19700354d6beac"}
Nov 03 01:39:12.517850 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:12.517818    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:12.518301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:12.518290    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:12.519039 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:12.519019    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:39:13.035968 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:13.035944    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"
Nov 03 01:39:13.142810 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:13.142778    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-xks-ca\") pod \"b97e649a-42c7-48d0-80f5-51f24407ee8b\" (UID: \"b97e649a-42c7-48d0-80f5-51f24407ee8b\") "
Nov 03 01:39:13.142986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:13.142901    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97e649a-42c7-48d0-80f5-51f24407ee8b-kube-api-access-qxlnw\" (UniqueName: \"kubernetes.io/projected/b97e649a-42c7-48d0-80f5-51f24407ee8b-kube-api-access-qxlnw\") pod \"b97e649a-42c7-48d0-80f5-51f24407ee8b\" (UID: \"b97e649a-42c7-48d0-80f5-51f24407ee8b\") "
Nov 03 01:39:13.142986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:13.142981    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-openshift-ca\") pod \"b97e649a-42c7-48d0-80f5-51f24407ee8b\" (UID: \"b97e649a-42c7-48d0-80f5-51f24407ee8b\") "
Nov 03 01:39:13.143319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:13.143291    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-openshift-ca" pod "b97e649a-42c7-48d0-80f5-51f24407ee8b" (UID: "b97e649a-42c7-48d0-80f5-51f24407ee8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:13.143487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:13.143476    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:13.145772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:13.145706    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-xks-ca" pod "b97e649a-42c7-48d0-80f5-51f24407ee8b" (UID: "b97e649a-42c7-48d0-80f5-51f24407ee8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:13.146904 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:13.146869    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97e649a-42c7-48d0-80f5-51f24407ee8b-kube-api-access-qxlnw" pod "b97e649a-42c7-48d0-80f5-51f24407ee8b" (UID: "b97e649a-42c7-48d0-80f5-51f24407ee8b"). InnerVolumeSpecName "kube-api-access-qxlnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:13.244517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:13.244433    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97e649a-42c7-48d0-80f5-51f24407ee8b-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:13.244517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:13.244469    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxlnw\" (UniqueName: \"kubernetes.io/projected/b97e649a-42c7-48d0-80f5-51f24407ee8b-kube-api-access-qxlnw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:13.766124 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:13.766093    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"
Nov 03 01:39:13.766320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:13.766122    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f" event={"ID":"b97e649a-42c7-48d0-80f5-51f24407ee8b","Type":"ContainerDied","Data":"46479aab49ea2b603904a5c06a604e5d324ccc5a8020853dec4dfced16b2f4ca"}
Nov 03 01:39:13.766320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:13.766182    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46479aab49ea2b603904a5c06a604e5d324ccc5a8020853dec4dfced16b2f4ca"
Nov 03 01:39:13.812412 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:39:13.812377    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:39:16.049640 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.049555    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"]
Nov 03 01:39:16.094912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.094881    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44441b6-kpnst"]
Nov 03 01:39:16.764372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.757514    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c93c4422-ae6d-4532-bdb2-957a9a0280c6" path="/var/lib/kubelet/pods/c93c4422-ae6d-4532-bdb2-957a9a0280c6/volumes"
Nov 03 01:39:16.852173 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.852140    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"]
Nov 03 01:39:16.853384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.853257    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b97e649a-42c7-48d0-80f5-51f24407ee8b" containerName="revoke-keys"
Nov 03 01:39:16.853574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.853557    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97e649a-42c7-48d0-80f5-51f24407ee8b" containerName="revoke-keys"
Nov 03 01:39:16.853846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.853823    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97e649a-42c7-48d0-80f5-51f24407ee8b" containerName="revoke-keys"
Nov 03 01:39:16.868404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.858222    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:16.911299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.911264    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29n6n\" (UniqueName: \"kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-kube-api-access-29n6n\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:16.911604 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.911582    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:16.911758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.911742    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:16.911854 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.911840    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:16.911962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.911950    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:16.912049 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.912036    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:16.912159 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.912145    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:16.912285 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.912272    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:16.912464 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.912446    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:16.963593 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:16.963559    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"]
Nov 03 01:39:17.013778 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.013736    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29n6n\" (UniqueName: \"kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-kube-api-access-29n6n\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.014974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.014042    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.014974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.014130    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.014974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.014158    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.014974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.014205    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.014974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.014232    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.014974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.014271    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.014974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.014324    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.014974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.014424    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.016678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.016654    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.018231 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.017498    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.018386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.017737    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.018545 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.017951    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.018672 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.018165    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.021860 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.021816    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.044428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.044179    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.126458 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.126412    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.128720 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.128662    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29n6n\" (UniqueName: \"kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-kube-api-access-29n6n\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:17.188641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:17.188601    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:18.351376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:18.338548    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"]
Nov 03 01:39:18.463235 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:39:18.462887    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2b4424_3a33_4f86_bb7d_28b6cf5f33c1.slice/crio-33145e5155b35cc3b37f167f7afbb21c42ffe7631dcdc298f1c38c6b5e52b015 WatchSource:0}: Error finding container 33145e5155b35cc3b37f167f7afbb21c42ffe7631dcdc298f1c38c6b5e52b015: Status 404 returned error can't find the container with id 33145e5155b35cc3b37f167f7afbb21c42ffe7631dcdc298f1c38c6b5e52b015
Nov 03 01:39:18.468797 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:18.465275    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:18.468797 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:18.465416    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:18.468797 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:18.465471    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:18.635422 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:18.630597    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:19.027636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.027590    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl" event={"ID":"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1","Type":"ContainerStarted","Data":"c0aea2a4031aeadd85ae40f7e66f415f583bfe21b702349e1018985b6d0d7394"}
Nov 03 01:39:19.027836 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.027682    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl" event={"ID":"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1","Type":"ContainerStarted","Data":"33145e5155b35cc3b37f167f7afbb21c42ffe7631dcdc298f1c38c6b5e52b015"}
Nov 03 01:39:19.142699 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.142466    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl" podStartSLOduration=3.142447256 podStartE2EDuration="3.142447256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:19.141893122 +0000 UTC m=+1238.567499607" watchObservedRunningTime="2026-11-03 01:39:19.142447256 +0000 UTC m=+1238.568053724"
Nov 03 01:39:19.190015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.189973    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:19.192104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.191408    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 01:39:19.192104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.191478    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl" podUID="db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 01:39:19.875646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.875585    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"]
Nov 03 01:39:19.881522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.881497    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"
Nov 03 01:39:19.921547 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.921511    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"]
Nov 03 01:39:19.982395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.980590    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:39:19.982395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.980707    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:39:19.982395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.980748    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:39:19.982395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.980835    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:39:19.995021 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.994986    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jl9q\" (UniqueName: \"kubernetes.io/projected/0488b37d-35ed-4e73-9c03-4f7f733fe73e-kube-api-access-5jl9q\") pod \"maas-api-revoke-sub-b388413e-7rvzr\" (UID: \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"
Nov 03 01:39:19.995206 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.995040    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b388413e-7rvzr\" (UID: \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"
Nov 03 01:39:19.995307 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:19.995284    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b388413e-7rvzr\" (UID: \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"
Nov 03 01:39:20.096485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:20.096448    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b388413e-7rvzr\" (UID: \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"
Nov 03 01:39:20.096675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:20.096596    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jl9q\" (UniqueName: \"kubernetes.io/projected/0488b37d-35ed-4e73-9c03-4f7f733fe73e-kube-api-access-5jl9q\") pod \"maas-api-revoke-sub-b388413e-7rvzr\" (UID: \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"
Nov 03 01:39:20.096675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:20.096646    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b388413e-7rvzr\" (UID: \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"
Nov 03 01:39:20.097431 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:20.097404    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b388413e-7rvzr\" (UID: \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"
Nov 03 01:39:20.099074 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:20.099048    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b388413e-7rvzr\" (UID: \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"
Nov 03 01:39:20.172381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:20.164748    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jl9q\" (UniqueName: \"kubernetes.io/projected/0488b37d-35ed-4e73-9c03-4f7f733fe73e-kube-api-access-5jl9q\") pod \"maas-api-revoke-sub-b388413e-7rvzr\" (UID: \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"
Nov 03 01:39:20.190482 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:20.190450    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 01:39:20.190685 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:20.190514    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl" podUID="db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 01:39:20.195327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:20.195302    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"
Nov 03 01:39:20.583893 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:20.583817    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"]
Nov 03 01:39:20.590314 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:39:20.590288    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0488b37d_35ed_4e73_9c03_4f7f733fe73e.slice/crio-e6383230d12bc71f08e6b246fe8cca7970928af42538eb0de11539ca0e096aed WatchSource:0}: Error finding container e6383230d12bc71f08e6b246fe8cca7970928af42538eb0de11539ca0e096aed: Status 404 returned error can't find the container with id e6383230d12bc71f08e6b246fe8cca7970928af42538eb0de11539ca0e096aed
Nov 03 01:39:21.043578 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:21.043546    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr" event={"ID":"0488b37d-35ed-4e73-9c03-4f7f733fe73e","Type":"ContainerStarted","Data":"17ec610628dbe4f4f1d3eb1200304e11e09859cb6b43e860ef3a7bf852b94f08"}
Nov 03 01:39:21.043945 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:21.043597    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr" event={"ID":"0488b37d-35ed-4e73-9c03-4f7f733fe73e","Type":"ContainerStarted","Data":"e6383230d12bc71f08e6b246fe8cca7970928af42538eb0de11539ca0e096aed"}
Nov 03 01:39:21.187014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:21.186962    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr" podStartSLOduration=2.186950089 podStartE2EDuration="2.186950089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:21.18551498 +0000 UTC m=+1240.611121445" watchObservedRunningTime="2026-11-03 01:39:21.186950089 +0000 UTC m=+1240.612556552"
Nov 03 01:39:21.189746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:21.189723    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 01:39:21.189857 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:21.189772    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl" podUID="db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 01:39:22.087532 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:22.087496    5508 generic.go:356] "Generic (PLEG): container finished" podID="0488b37d-35ed-4e73-9c03-4f7f733fe73e" containerID="17ec610628dbe4f4f1d3eb1200304e11e09859cb6b43e860ef3a7bf852b94f08" exitCode=0
Nov 03 01:39:22.087991 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:22.087716    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr" event={"ID":"0488b37d-35ed-4e73-9c03-4f7f733fe73e","Type":"ContainerDied","Data":"17ec610628dbe4f4f1d3eb1200304e11e09859cb6b43e860ef3a7bf852b94f08"}
Nov 03 01:39:22.189985 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:22.189946    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 01:39:22.190158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:22.189999    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl" podUID="db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 01:39:22.980967 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:22.980930    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"]
Nov 03 01:39:22.998296 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:22.998262    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"
Nov 03 01:39:23.033659 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.033525    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69356caf-0473-482d-8254-592b6f445e0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0bc4711-f87g7\" (UID: \"69356caf-0473-482d-8254-592b6f445e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"
Nov 03 01:39:23.033659 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.033626    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"]
Nov 03 01:39:23.033901 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.033687    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69356caf-0473-482d-8254-592b6f445e0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0bc4711-f87g7\" (UID: \"69356caf-0473-482d-8254-592b6f445e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"
Nov 03 01:39:23.033901 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.033792    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2gff\" (UniqueName: \"kubernetes.io/projected/69356caf-0473-482d-8254-592b6f445e0f-kube-api-access-s2gff\") pod \"maas-api-revoke-sub-a0bc4711-f87g7\" (UID: \"69356caf-0473-482d-8254-592b6f445e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"
Nov 03 01:39:23.135595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.135534    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69356caf-0473-482d-8254-592b6f445e0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0bc4711-f87g7\" (UID: \"69356caf-0473-482d-8254-592b6f445e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"
Nov 03 01:39:23.135946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.135676    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69356caf-0473-482d-8254-592b6f445e0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0bc4711-f87g7\" (UID: \"69356caf-0473-482d-8254-592b6f445e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"
Nov 03 01:39:23.135946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.135703    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2gff\" (UniqueName: \"kubernetes.io/projected/69356caf-0473-482d-8254-592b6f445e0f-kube-api-access-s2gff\") pod \"maas-api-revoke-sub-a0bc4711-f87g7\" (UID: \"69356caf-0473-482d-8254-592b6f445e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"
Nov 03 01:39:23.136615 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.136590    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69356caf-0473-482d-8254-592b6f445e0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0bc4711-f87g7\" (UID: \"69356caf-0473-482d-8254-592b6f445e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"
Nov 03 01:39:23.140186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.139091    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69356caf-0473-482d-8254-592b6f445e0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0bc4711-f87g7\" (UID: \"69356caf-0473-482d-8254-592b6f445e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"
Nov 03 01:39:23.189754 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.189719    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 01:39:23.189910 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.189779    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl" podUID="db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 01:39:23.198620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.198107    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2gff\" (UniqueName: \"kubernetes.io/projected/69356caf-0473-482d-8254-592b6f445e0f-kube-api-access-s2gff\") pod \"maas-api-revoke-sub-a0bc4711-f87g7\" (UID: \"69356caf-0473-482d-8254-592b6f445e0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"
Nov 03 01:39:23.299548 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.299489    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"]
Nov 03 01:39:23.304916 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.304897    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.309521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.309495    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"
Nov 03 01:39:23.324152 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.324133    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"
Nov 03 01:39:23.338878 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.338851    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-xks-ca\") pod \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\" (UID: \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\") "
Nov 03 01:39:23.339101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.339080    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0488b37d-35ed-4e73-9c03-4f7f733fe73e-kube-api-access-5jl9q\" (UniqueName: \"kubernetes.io/projected/0488b37d-35ed-4e73-9c03-4f7f733fe73e-kube-api-access-5jl9q\") pod \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\" (UID: \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\") "
Nov 03 01:39:23.340672 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.340654    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-openshift-ca\") pod \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\" (UID: \"0488b37d-35ed-4e73-9c03-4f7f733fe73e\") "
Nov 03 01:39:23.341126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.341099    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.343778 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.343540    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-openshift-ca" pod "0488b37d-35ed-4e73-9c03-4f7f733fe73e" (UID: "0488b37d-35ed-4e73-9c03-4f7f733fe73e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:23.348205 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.347955    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0488b37d-35ed-4e73-9c03-4f7f733fe73e-kube-api-access-5jl9q" pod "0488b37d-35ed-4e73-9c03-4f7f733fe73e" (UID: "0488b37d-35ed-4e73-9c03-4f7f733fe73e"). InnerVolumeSpecName "kube-api-access-5jl9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:23.352557 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.347915    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-xks-ca" pod "0488b37d-35ed-4e73-9c03-4f7f733fe73e" (UID: "0488b37d-35ed-4e73-9c03-4f7f733fe73e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:23.352668 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.352650    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.361518 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.361488    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.371380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.361688    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beeaa3a5-ee87-4129-a879-5f5f591ec803-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.412994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.385096    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"]
Nov 03 01:39:23.412994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.412660    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.412994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.412706    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.412994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.412741    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.412994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.412837    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.412994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.412897    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz9rf\" (UniqueName: \"kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-kube-api-access-vz9rf\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.413502 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.413018    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:23.413502 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.413034    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0488b37d-35ed-4e73-9c03-4f7f733fe73e-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:23.413502 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.413047    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jl9q\" (UniqueName: \"kubernetes.io/projected/0488b37d-35ed-4e73-9c03-4f7f733fe73e-kube-api-access-5jl9q\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:23.520008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.517121    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.520008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.517180    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beeaa3a5-ee87-4129-a879-5f5f591ec803-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.520008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.517295    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.520008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.517336    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.520008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.517404    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.520008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.517488    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.520008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.517541    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz9rf\" (UniqueName: \"kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-kube-api-access-vz9rf\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.520008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.517642    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.520008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.517667    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.520008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.518571    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.520008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.519263    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beeaa3a5-ee87-4129-a879-5f5f591ec803-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.520008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.519555    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.530299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.530251    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.531337 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.531290    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.531810 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.531788    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.644203 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.622670    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.644203 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.633674    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.644203 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.636410    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz9rf\" (UniqueName: \"kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-kube-api-access-vz9rf\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:23.838582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.834424    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"]
Nov 03 01:39:23.918792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:23.918764    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:24.105533 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.105492    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7" event={"ID":"69356caf-0473-482d-8254-592b6f445e0f","Type":"ContainerStarted","Data":"147707f0fcdfdcf14daf043339503090987b5648668047e2f8a141c882fea204"}
Nov 03 01:39:24.107137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.107110    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr" event={"ID":"0488b37d-35ed-4e73-9c03-4f7f733fe73e","Type":"ContainerDied","Data":"e6383230d12bc71f08e6b246fe8cca7970928af42538eb0de11539ca0e096aed"}
Nov 03 01:39:24.107251 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.107147    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"
Nov 03 01:39:24.107251 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.107158    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6383230d12bc71f08e6b246fe8cca7970928af42538eb0de11539ca0e096aed"
Nov 03 01:39:24.189838 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.189802    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 01:39:24.190323 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.189852    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl" podUID="db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 01:39:24.246388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.246340    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"]
Nov 03 01:39:24.247012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.246987    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0488b37d-35ed-4e73-9c03-4f7f733fe73e" containerName="revoke-keys"
Nov 03 01:39:24.247012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.247010    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0488b37d-35ed-4e73-9c03-4f7f733fe73e" containerName="revoke-keys"
Nov 03 01:39:24.247216 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.247140    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0488b37d-35ed-4e73-9c03-4f7f733fe73e" containerName="revoke-keys"
Nov 03 01:39:24.257395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.253708    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"
Nov 03 01:39:24.307390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.307362    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"]
Nov 03 01:39:24.341669 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.341635    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvbml\" (UniqueName: \"kubernetes.io/projected/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-kube-api-access-mvbml\") pod \"maas-api-revoke-sub-9fb19f13-rwf8c\" (UID: \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"
Nov 03 01:39:24.341825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.341678    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fb19f13-rwf8c\" (UID: \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"
Nov 03 01:39:24.341886 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.341863    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fb19f13-rwf8c\" (UID: \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"
Nov 03 01:39:24.380469 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.380418    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"]
Nov 03 01:39:24.380590 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:39:24.380475    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeeaa3a5_ee87_4129_a879_5f5f591ec803.slice/crio-6cddde5f1b97bb83151927ea1539054dd4562fbcdfb6b85fb5ddbb51bcf4448f WatchSource:0}: Error finding container 6cddde5f1b97bb83151927ea1539054dd4562fbcdfb6b85fb5ddbb51bcf4448f: Status 404 returned error can't find the container with id 6cddde5f1b97bb83151927ea1539054dd4562fbcdfb6b85fb5ddbb51bcf4448f
Nov 03 01:39:24.383548 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.383513    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:24.383623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.383591    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:24.383658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.383630    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:24.443219 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.443194    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fb19f13-rwf8c\" (UID: \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"
Nov 03 01:39:24.443325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.443285    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvbml\" (UniqueName: \"kubernetes.io/projected/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-kube-api-access-mvbml\") pod \"maas-api-revoke-sub-9fb19f13-rwf8c\" (UID: \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"
Nov 03 01:39:24.443325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.443310    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fb19f13-rwf8c\" (UID: \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"
Nov 03 01:39:24.443952 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.443930    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fb19f13-rwf8c\" (UID: \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"
Nov 03 01:39:24.445260 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.445242    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fb19f13-rwf8c\" (UID: \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"
Nov 03 01:39:24.500265 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.500236    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvbml\" (UniqueName: \"kubernetes.io/projected/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-kube-api-access-mvbml\") pod \"maas-api-revoke-sub-9fb19f13-rwf8c\" (UID: \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"
Nov 03 01:39:24.586316 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.586269    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"
Nov 03 01:39:24.995535 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:24.995406    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"]
Nov 03 01:39:25.073878 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.073834    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"]
Nov 03 01:39:25.087475 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.087448    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.115891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.115852    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn" event={"ID":"beeaa3a5-ee87-4129-a879-5f5f591ec803","Type":"ContainerStarted","Data":"848e6c08d801240b3d6542b1e3444d5806eec2310d5da171c771356535c16dfa"}
Nov 03 01:39:25.116039 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.115917    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn" event={"ID":"beeaa3a5-ee87-4129-a879-5f5f591ec803","Type":"ContainerStarted","Data":"6cddde5f1b97bb83151927ea1539054dd4562fbcdfb6b85fb5ddbb51bcf4448f"}
Nov 03 01:39:25.123432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.123401    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"]
Nov 03 01:39:25.123635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.123610    5508 generic.go:356] "Generic (PLEG): container finished" podID="69356caf-0473-482d-8254-592b6f445e0f" containerID="6ae38348099f20c71bac316a02d9c1026b09342a50b4ebf769c0312a2d201537" exitCode=0
Nov 03 01:39:25.123741 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.123718    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7" event={"ID":"69356caf-0473-482d-8254-592b6f445e0f","Type":"ContainerDied","Data":"6ae38348099f20c71bac316a02d9c1026b09342a50b4ebf769c0312a2d201537"}
Nov 03 01:39:25.125618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.125584    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c" event={"ID":"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1","Type":"ContainerStarted","Data":"ba3dcd6bad52a2657c9162224caf0579e9e180c4f0b7328a6d5425b32ce84548"}
Nov 03 01:39:25.157567 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.157528    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.157765 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.157747    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f4321-ff64-453e-a103-29676b99d594-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.199987 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.199942    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f599q\" (UniqueName: \"kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-kube-api-access-f599q\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.206700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.206670    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.206926 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.206907    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.207076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.207039    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.207211 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.207195    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.207404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.207380    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f4321-ff64-453e-a103-29676b99d594-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.207553 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.207537    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.215694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.215665    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 01:39:25.215832 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.215710    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl" podUID="db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 01:39:25.308968 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.308925    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.309145 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.308981    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.309145 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.309006    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f4321-ff64-453e-a103-29676b99d594-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.309145 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.309075    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f599q\" (UniqueName: \"kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-kube-api-access-f599q\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.309145 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.309131    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.309389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.309186    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.309389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.309227    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.309389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.309255    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.309389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.309308    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f4321-ff64-453e-a103-29676b99d594-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.310421 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.310150    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.310563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.310451    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.310563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.310488    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f4321-ff64-453e-a103-29676b99d594-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.310563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.310496    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.310563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.310528    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.312738 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.312715    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.312918 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.312894    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f4321-ff64-453e-a103-29676b99d594-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.333387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.333287    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn" podStartSLOduration=2.333269633 podStartE2EDuration="2.333269633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:25.328988253 +0000 UTC m=+1244.754594736" watchObservedRunningTime="2026-11-03 01:39:25.333269633 +0000 UTC m=+1244.758876099"
Nov 03 01:39:25.383064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.383031    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f599q\" (UniqueName: \"kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-kube-api-access-f599q\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.383259 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.383111    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.402057 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.402029    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:25.810060 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.810017    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"]
Nov 03 01:39:25.819761 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.819718    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:25.819865 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.819804    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:25.819865 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.819844    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:25.920251 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.919429    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:25.920496 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.920470    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 01:39:25.920580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:25.920550    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn" podUID="beeaa3a5-ee87-4129-a879-5f5f591ec803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 01:39:26.138925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.137249    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn" event={"ID":"a07f4321-ff64-453e-a103-29676b99d594","Type":"ContainerStarted","Data":"5e5ae1738d5390a7c547061d3fcf18423e85f6d8a2377ad45fe0e232b3896f06"}
Nov 03 01:39:26.138925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.137334    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn" event={"ID":"a07f4321-ff64-453e-a103-29676b99d594","Type":"ContainerStarted","Data":"e6144ce799a400f42af171fc7401b6e43420b5f3d6ad0f0801e6930f268b8109"}
Nov 03 01:39:26.141122 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.141092    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c" event={"ID":"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1","Type":"ContainerStarted","Data":"6028e9394bc4069e737c5fe2011a983516e0ccdcffef8805422e55f8bdb65ce2"}
Nov 03 01:39:26.205789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.205750    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"]
Nov 03 01:39:26.212764 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.210972    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"
Nov 03 01:39:26.283427 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.243018    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"]
Nov 03 01:39:26.332608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.332566    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26506e6f-nsx49\" (UID: \"c556e115-0844-4a98-804c-e6a3283855ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"
Nov 03 01:39:26.332780 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.332672    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26506e6f-nsx49\" (UID: \"c556e115-0844-4a98-804c-e6a3283855ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"
Nov 03 01:39:26.332780 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.332724    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8drl\" (UniqueName: \"kubernetes.io/projected/c556e115-0844-4a98-804c-e6a3283855ce-kube-api-access-x8drl\") pod \"maas-api-revoke-sub-26506e6f-nsx49\" (UID: \"c556e115-0844-4a98-804c-e6a3283855ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"
Nov 03 01:39:26.407404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.406777    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:26.414292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.414261    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 01:39:26.414535 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.414437    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn" podUID="a07f4321-ff64-453e-a103-29676b99d594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 01:39:26.440175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.439151    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26506e6f-nsx49\" (UID: \"c556e115-0844-4a98-804c-e6a3283855ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"
Nov 03 01:39:26.440175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.439227    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26506e6f-nsx49\" (UID: \"c556e115-0844-4a98-804c-e6a3283855ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"
Nov 03 01:39:26.440175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.439265    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8drl\" (UniqueName: \"kubernetes.io/projected/c556e115-0844-4a98-804c-e6a3283855ce-kube-api-access-x8drl\") pod \"maas-api-revoke-sub-26506e6f-nsx49\" (UID: \"c556e115-0844-4a98-804c-e6a3283855ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"
Nov 03 01:39:26.441002 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.440971    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26506e6f-nsx49\" (UID: \"c556e115-0844-4a98-804c-e6a3283855ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"
Nov 03 01:39:26.471732 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.471691    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26506e6f-nsx49\" (UID: \"c556e115-0844-4a98-804c-e6a3283855ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"
Nov 03 01:39:26.474579 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.474514    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn" podStartSLOduration=1.474495149 podStartE2EDuration="1.474495149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:26.334430789 +0000 UTC m=+1245.760037279" watchObservedRunningTime="2026-11-03 01:39:26.474495149 +0000 UTC m=+1245.900101615"
Nov 03 01:39:26.515923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.515854    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8drl\" (UniqueName: \"kubernetes.io/projected/c556e115-0844-4a98-804c-e6a3283855ce-kube-api-access-x8drl\") pod \"maas-api-revoke-sub-26506e6f-nsx49\" (UID: \"c556e115-0844-4a98-804c-e6a3283855ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"
Nov 03 01:39:26.604962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.604788    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c" podStartSLOduration=2.604766467 podStartE2EDuration="2.604766467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:26.581896156 +0000 UTC m=+1246.007502623" watchObservedRunningTime="2026-11-03 01:39:26.604766467 +0000 UTC m=+1246.030372932"
Nov 03 01:39:26.643651 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.643624    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"
Nov 03 01:39:26.643849 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.643826    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"
Nov 03 01:39:26.645496 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.645477    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:26.646276 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.646257    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:26.652368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.648889    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:39:26.742949 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.742901    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69356caf-0473-482d-8254-592b6f445e0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69356caf-0473-482d-8254-592b6f445e0f-maas-api-openshift-ca\") pod \"69356caf-0473-482d-8254-592b6f445e0f\" (UID: \"69356caf-0473-482d-8254-592b6f445e0f\") "
Nov 03 01:39:26.743151 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.743038    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69356caf-0473-482d-8254-592b6f445e0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69356caf-0473-482d-8254-592b6f445e0f-maas-api-xks-ca\") pod \"69356caf-0473-482d-8254-592b6f445e0f\" (UID: \"69356caf-0473-482d-8254-592b6f445e0f\") "
Nov 03 01:39:26.743151 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.743114    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69356caf-0473-482d-8254-592b6f445e0f-kube-api-access-s2gff\" (UniqueName: \"kubernetes.io/projected/69356caf-0473-482d-8254-592b6f445e0f-kube-api-access-s2gff\") pod \"69356caf-0473-482d-8254-592b6f445e0f\" (UID: \"69356caf-0473-482d-8254-592b6f445e0f\") "
Nov 03 01:39:26.743645 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.743381    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69356caf-0473-482d-8254-592b6f445e0f-maas-api-openshift-ca" pod "69356caf-0473-482d-8254-592b6f445e0f" (UID: "69356caf-0473-482d-8254-592b6f445e0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:26.743970 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.743949    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69356caf-0473-482d-8254-592b6f445e0f-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:26.746715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.746694    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69356caf-0473-482d-8254-592b6f445e0f-maas-api-xks-ca" pod "69356caf-0473-482d-8254-592b6f445e0f" (UID: "69356caf-0473-482d-8254-592b6f445e0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:26.747841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.747821    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69356caf-0473-482d-8254-592b6f445e0f-kube-api-access-s2gff" pod "69356caf-0473-482d-8254-592b6f445e0f" (UID: "69356caf-0473-482d-8254-592b6f445e0f"). InnerVolumeSpecName "kube-api-access-s2gff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:26.893384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.850038    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69356caf-0473-482d-8254-592b6f445e0f-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:26.893384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.850078    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2gff\" (UniqueName: \"kubernetes.io/projected/69356caf-0473-482d-8254-592b6f445e0f-kube-api-access-s2gff\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:26.923720 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.923672    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 01:39:26.925144 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:26.925112    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn" podUID="beeaa3a5-ee87-4129-a879-5f5f591ec803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 01:39:27.104675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:27.104589    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"]
Nov 03 01:39:27.110453 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:27.110418    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf38d6a-z5dv6"]
Nov 03 01:39:27.184378 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:27.182473    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7" event={"ID":"69356caf-0473-482d-8254-592b6f445e0f","Type":"ContainerDied","Data":"147707f0fcdfdcf14daf043339503090987b5648668047e2f8a141c882fea204"}
Nov 03 01:39:27.184378 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:27.182579    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="147707f0fcdfdcf14daf043339503090987b5648668047e2f8a141c882fea204"
Nov 03 01:39:27.184378 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:27.182493    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"
Nov 03 01:39:27.205381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:27.191980    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"]
Nov 03 01:39:27.210128 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:27.209322    5508 generic.go:356] "Generic (PLEG): container finished" podID="0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1" containerID="6028e9394bc4069e737c5fe2011a983516e0ccdcffef8805422e55f8bdb65ce2" exitCode=0
Nov 03 01:39:27.210128 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:27.209409    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c" event={"ID":"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1","Type":"ContainerDied","Data":"6028e9394bc4069e737c5fe2011a983516e0ccdcffef8805422e55f8bdb65ce2"}
Nov 03 01:39:27.212831 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:39:27.212809    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc556e115_0844_4a98_804c_e6a3283855ce.slice/crio-f20c393e5a80598064b9920b3402c00c526c5c5db1e121d20e58f7f719c4d138 WatchSource:0}: Error finding container f20c393e5a80598064b9920b3402c00c526c5c5db1e121d20e58f7f719c4d138: Status 404 returned error can't find the container with id f20c393e5a80598064b9920b3402c00c526c5c5db1e121d20e58f7f719c4d138
Nov 03 01:39:27.405265 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:27.405237    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 01:39:27.405797 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:27.405760    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn" podUID="a07f4321-ff64-453e-a103-29676b99d594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 01:39:27.922372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:27.921677    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 01:39:27.922372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:27.921740    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn" podUID="beeaa3a5-ee87-4129-a879-5f5f591ec803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 01:39:28.220258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.220215    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49" event={"ID":"c556e115-0844-4a98-804c-e6a3283855ce","Type":"ContainerStarted","Data":"25eca983c72efcacb5f6293eb2bb77a5b911088d4d7fcd9994b1ca36a7f09fa9"}
Nov 03 01:39:28.220786 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.220277    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49" event={"ID":"c556e115-0844-4a98-804c-e6a3283855ce","Type":"ContainerStarted","Data":"f20c393e5a80598064b9920b3402c00c526c5c5db1e121d20e58f7f719c4d138"}
Nov 03 01:39:28.333647 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.333593    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49" podStartSLOduration=2.333578549 podStartE2EDuration="2.333578549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:28.329581956 +0000 UTC m=+1247.755188438" watchObservedRunningTime="2026-11-03 01:39:28.333578549 +0000 UTC m=+1247.759185013"
Nov 03 01:39:28.429382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.428779    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 01:39:28.429382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.428858    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn" podUID="a07f4321-ff64-453e-a103-29676b99d594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 01:39:28.623564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.623536    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"
Nov 03 01:39:28.710980 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.710935    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de781385-68e0-42c5-869b-b00a6a15dd3c" path="/var/lib/kubelet/pods/de781385-68e0-42c5-869b-b00a6a15dd3c/volumes"
Nov 03 01:39:28.734326 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.734286    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-xks-ca\") pod \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\" (UID: \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\") "
Nov 03 01:39:28.734564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.734396    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-openshift-ca\") pod \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\" (UID: \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\") "
Nov 03 01:39:28.734564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.734464    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-kube-api-access-mvbml\" (UniqueName: \"kubernetes.io/projected/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-kube-api-access-mvbml\") pod \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\" (UID: \"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1\") "
Nov 03 01:39:28.734801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.734777    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-openshift-ca" pod "0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1" (UID: "0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:28.734948 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.734929    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:28.737029 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.736988    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-xks-ca" pod "0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1" (UID: "0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:28.737630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.737603    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-kube-api-access-mvbml" pod "0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1" (UID: "0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1"). InnerVolumeSpecName "kube-api-access-mvbml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:28.835974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.835878    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:28.836157 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.836142    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvbml\" (UniqueName: \"kubernetes.io/projected/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1-kube-api-access-mvbml\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:28.919221 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.919188    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 01:39:28.919405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:28.919247    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn" podUID="beeaa3a5-ee87-4129-a879-5f5f591ec803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 01:39:29.228480 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:29.228452    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"
Nov 03 01:39:29.228957 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:29.228938    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c" event={"ID":"0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1","Type":"ContainerDied","Data":"ba3dcd6bad52a2657c9162224caf0579e9e180c4f0b7328a6d5425b32ce84548"}
Nov 03 01:39:29.229070 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:29.229060    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba3dcd6bad52a2657c9162224caf0579e9e180c4f0b7328a6d5425b32ce84548"
Nov 03 01:39:29.230384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:29.230360    5508 generic.go:356] "Generic (PLEG): container finished" podID="c556e115-0844-4a98-804c-e6a3283855ce" containerID="25eca983c72efcacb5f6293eb2bb77a5b911088d4d7fcd9994b1ca36a7f09fa9" exitCode=0
Nov 03 01:39:29.230474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:29.230420    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49" event={"ID":"c556e115-0844-4a98-804c-e6a3283855ce","Type":"ContainerDied","Data":"25eca983c72efcacb5f6293eb2bb77a5b911088d4d7fcd9994b1ca36a7f09fa9"}
Nov 03 01:39:29.406386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:29.405733    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 01:39:29.406386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:29.405795    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn" podUID="a07f4321-ff64-453e-a103-29676b99d594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 01:39:29.919494 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:29.919452    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 01:39:29.919690 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:29.919515    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn" podUID="beeaa3a5-ee87-4129-a879-5f5f591ec803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 01:39:30.425387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.424774    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 01:39:30.425387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.424836    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn" podUID="a07f4321-ff64-453e-a103-29676b99d594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 01:39:30.612619 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.612592    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"
Nov 03 01:39:30.726844 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.726805    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-xks-ca\") pod \"c556e115-0844-4a98-804c-e6a3283855ce\" (UID: \"c556e115-0844-4a98-804c-e6a3283855ce\") "
Nov 03 01:39:30.727048 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.727010    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c556e115-0844-4a98-804c-e6a3283855ce-kube-api-access-x8drl\" (UniqueName: \"kubernetes.io/projected/c556e115-0844-4a98-804c-e6a3283855ce-kube-api-access-x8drl\") pod \"c556e115-0844-4a98-804c-e6a3283855ce\" (UID: \"c556e115-0844-4a98-804c-e6a3283855ce\") "
Nov 03 01:39:30.727123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.727054    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-openshift-ca\") pod \"c556e115-0844-4a98-804c-e6a3283855ce\" (UID: \"c556e115-0844-4a98-804c-e6a3283855ce\") "
Nov 03 01:39:30.727454 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.727425    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-openshift-ca" pod "c556e115-0844-4a98-804c-e6a3283855ce" (UID: "c556e115-0844-4a98-804c-e6a3283855ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:30.728922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.728895    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-xks-ca" pod "c556e115-0844-4a98-804c-e6a3283855ce" (UID: "c556e115-0844-4a98-804c-e6a3283855ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:30.729121 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.729094    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c556e115-0844-4a98-804c-e6a3283855ce-kube-api-access-x8drl" pod "c556e115-0844-4a98-804c-e6a3283855ce" (UID: "c556e115-0844-4a98-804c-e6a3283855ce"). InnerVolumeSpecName "kube-api-access-x8drl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:30.828739 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.828701    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8drl\" (UniqueName: \"kubernetes.io/projected/c556e115-0844-4a98-804c-e6a3283855ce-kube-api-access-x8drl\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:30.828739 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.828738    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:30.828739 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.828754    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c556e115-0844-4a98-804c-e6a3283855ce-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:30.919168 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.919133    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 01:39:30.919371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:30.919187    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn" podUID="beeaa3a5-ee87-4129-a879-5f5f591ec803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 01:39:31.246220 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:31.246186    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"
Nov 03 01:39:31.246433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:31.246191    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49" event={"ID":"c556e115-0844-4a98-804c-e6a3283855ce","Type":"ContainerDied","Data":"f20c393e5a80598064b9920b3402c00c526c5c5db1e121d20e58f7f719c4d138"}
Nov 03 01:39:31.246433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:31.246314    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f20c393e5a80598064b9920b3402c00c526c5c5db1e121d20e58f7f719c4d138"
Nov 03 01:39:31.402592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:31.402540    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 01:39:31.402796 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:31.402604    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn" podUID="a07f4321-ff64-453e-a103-29676b99d594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 01:39:32.170378 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:32.170169    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:32.171957 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:32.171886    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:32.176870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:32.176826    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:39:32.403223 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:32.403191    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 03 01:39:32.403503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:32.403464    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn" podUID="a07f4321-ff64-453e-a103-29676b99d594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 03 01:39:33.581176 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.581139    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:33.581766 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.581666    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:33.583393 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.583368    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:39:33.729071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.729035    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"]
Nov 03 01:39:33.729788 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.729758    5508 cpu_manager.go:414] "containerMap: removing container" podUID="69356caf-0473-482d-8254-592b6f445e0f" containerName="revoke-keys"
Nov 03 01:39:33.729788 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.729785    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69356caf-0473-482d-8254-592b6f445e0f" containerName="revoke-keys"
Nov 03 01:39:33.730012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.729835    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1" containerName="revoke-keys"
Nov 03 01:39:33.730012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.729843    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1" containerName="revoke-keys"
Nov 03 01:39:33.730012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.729878    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c556e115-0844-4a98-804c-e6a3283855ce" containerName="revoke-keys"
Nov 03 01:39:33.730012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.729886    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c556e115-0844-4a98-804c-e6a3283855ce" containerName="revoke-keys"
Nov 03 01:39:33.730012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.730005    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c556e115-0844-4a98-804c-e6a3283855ce" containerName="revoke-keys"
Nov 03 01:39:33.730303 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.730017    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="69356caf-0473-482d-8254-592b6f445e0f" containerName="revoke-keys"
Nov 03 01:39:33.730303 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.730032    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1" containerName="revoke-keys"
Nov 03 01:39:33.734080 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.734052    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"
Nov 03 01:39:33.778576 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.778534    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9645045f-svvqt\" (UID: \"9164b080-801e-41b3-bcca-5b7d42ca6204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"
Nov 03 01:39:33.778730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.778669    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxcsh\" (UniqueName: \"kubernetes.io/projected/9164b080-801e-41b3-bcca-5b7d42ca6204-kube-api-access-cxcsh\") pod \"maas-api-revoke-sub-9645045f-svvqt\" (UID: \"9164b080-801e-41b3-bcca-5b7d42ca6204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"
Nov 03 01:39:33.778771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.778735    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9645045f-svvqt\" (UID: \"9164b080-801e-41b3-bcca-5b7d42ca6204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"
Nov 03 01:39:33.787081 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.787049    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"]
Nov 03 01:39:33.787578 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.787252    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:39:33.787578 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.787326    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:39:33.787578 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.787381    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:39:33.787578 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.787458    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:39:33.881009 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.880038    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9645045f-svvqt\" (UID: \"9164b080-801e-41b3-bcca-5b7d42ca6204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"
Nov 03 01:39:33.881009 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.880239    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9645045f-svvqt\" (UID: \"9164b080-801e-41b3-bcca-5b7d42ca6204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"
Nov 03 01:39:33.881009 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.880363    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxcsh\" (UniqueName: \"kubernetes.io/projected/9164b080-801e-41b3-bcca-5b7d42ca6204-kube-api-access-cxcsh\") pod \"maas-api-revoke-sub-9645045f-svvqt\" (UID: \"9164b080-801e-41b3-bcca-5b7d42ca6204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"
Nov 03 01:39:33.881306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.881106    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9645045f-svvqt\" (UID: \"9164b080-801e-41b3-bcca-5b7d42ca6204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"
Nov 03 01:39:33.885571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.885537    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9645045f-svvqt\" (UID: \"9164b080-801e-41b3-bcca-5b7d42ca6204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"
Nov 03 01:39:33.938004 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:33.937970    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxcsh\" (UniqueName: \"kubernetes.io/projected/9164b080-801e-41b3-bcca-5b7d42ca6204-kube-api-access-cxcsh\") pod \"maas-api-revoke-sub-9645045f-svvqt\" (UID: \"9164b080-801e-41b3-bcca-5b7d42ca6204\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"
Nov 03 01:39:34.045183 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:34.045141    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:39:34.064849 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:34.064811    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"
Nov 03 01:39:34.534974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:34.534824    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"]
Nov 03 01:39:35.354068 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:35.353979    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt" event={"ID":"9164b080-801e-41b3-bcca-5b7d42ca6204","Type":"ContainerStarted","Data":"c1ad170def8c713b9c894e94998efb73bd99bc82091259e9c493c2ccfe6c0891"}
Nov 03 01:39:36.090375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:36.089434    5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:36.145373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:36.142656    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:36.179117 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:36.178587    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:36.420374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:36.402633    5508 generic.go:356] "Generic (PLEG): container finished" podID="9164b080-801e-41b3-bcca-5b7d42ca6204" containerID="40f9f6a3322d15e852fee36860c32216de4cb36f4b6d8e59be452c6ae0092fe6" exitCode=0
Nov 03 01:39:36.420374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:36.402908    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt" event={"ID":"9164b080-801e-41b3-bcca-5b7d42ca6204","Type":"ContainerDied","Data":"40f9f6a3322d15e852fee36860c32216de4cb36f4b6d8e59be452c6ae0092fe6"}
Nov 03 01:39:36.602375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:36.591462    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:36.623590 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:36.623538    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 01:39:36.963496 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:36.963445    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:37.148377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:37.147742    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 01:39:37.152373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:37.148723    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:39:37.181375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:37.180476    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:37.181375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:37.180942    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:39:37.608581 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:37.608533    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:37.609558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:37.609532    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:39:37.921282 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:37.895788    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 01:39:37.921282 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:37.918137    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:39:38.157374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:38.149549    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:38.207433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:38.192042    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:38.433391 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:38.400113    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"]
Nov 03 01:39:38.433391 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:38.400187    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b02a3ac-ktv8h"]
Nov 03 01:39:38.662149 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:38.662102    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:38.911530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:38.911477    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 01:39:39.147430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.133750    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4569eb3c-cd20-4d31-9567-54b06c1e80ca" path="/var/lib/kubelet/pods/4569eb3c-cd20-4d31-9567-54b06c1e80ca/volumes"
Nov 03 01:39:39.231420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.192503    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:39.239378 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.231775    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"]
Nov 03 01:39:39.249094 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.231164    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:39.339375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.333910    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.431441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.430278    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"]
Nov 03 01:39:39.449160 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.449126    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"]
Nov 03 01:39:39.450777 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.450742    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99" podUID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerName="istio-proxy" containerID="cri-o://5e91e1f2cbf4db6236ff6b554283818e007ca6ebdd27cf723e04a4296fc1c5b9" gracePeriod=30
Nov 03 01:39:39.551844 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.551797    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.552213 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.552187    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.562496 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.562460    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.562767 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.562749    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.563255 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.563231    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53333dac-248a-4cf1-ac30-c1962d83d8b9-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.572583 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.572536    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.572867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.572830    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.573021 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.573002    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.573262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.573243    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-624p8\" (UniqueName: \"kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-kube-api-access-624p8\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.720425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.694800    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:39.755843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.714410    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-624p8\" (UniqueName: \"kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-kube-api-access-624p8\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.755843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.721688    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.755843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.721721    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.755843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.721763    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.755843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.721803    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.755843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.721921    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53333dac-248a-4cf1-ac30-c1962d83d8b9-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.755843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.722048    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.755843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.722080    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.755843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.722116    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.755843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.723052    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.755843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.723313    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.757278 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.757231    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.770164 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.768792    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.770164 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.769475    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53333dac-248a-4cf1-ac30-c1962d83d8b9-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.934617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.919445    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:39.934617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.929872    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-624p8\" (UniqueName: \"kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-kube-api-access-624p8\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:39.956967 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:39.956921    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:40.009394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.003240    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:40.012194 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.012128    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:40.070041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.069949    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:40.140582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.139477    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:39:40.169319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.169261    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"
Nov 03 01:39:40.217024 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.212244    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-xks-ca\") pod \"9164b080-801e-41b3-bcca-5b7d42ca6204\" (UID: \"9164b080-801e-41b3-bcca-5b7d42ca6204\") "
Nov 03 01:39:40.217024 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.212462    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-openshift-ca\") pod \"9164b080-801e-41b3-bcca-5b7d42ca6204\" (UID: \"9164b080-801e-41b3-bcca-5b7d42ca6204\") "
Nov 03 01:39:40.217024 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.212517    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9164b080-801e-41b3-bcca-5b7d42ca6204-kube-api-access-cxcsh\" (UniqueName: \"kubernetes.io/projected/9164b080-801e-41b3-bcca-5b7d42ca6204-kube-api-access-cxcsh\") pod \"9164b080-801e-41b3-bcca-5b7d42ca6204\" (UID: \"9164b080-801e-41b3-bcca-5b7d42ca6204\") "
Nov 03 01:39:40.272420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.253522    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:40.272420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.253972    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 01:39:40.272420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.254467    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-xks-ca" pod "9164b080-801e-41b3-bcca-5b7d42ca6204" (UID: "9164b080-801e-41b3-bcca-5b7d42ca6204"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:40.373374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.345922    5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:39:40.373374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.360931    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-openshift-ca" pod "9164b080-801e-41b3-bcca-5b7d42ca6204" (UID: "9164b080-801e-41b3-bcca-5b7d42ca6204"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:40.397843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.391265    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:40.397843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.391305    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164b080-801e-41b3-bcca-5b7d42ca6204-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:40.404171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.404149    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"
Nov 03 01:39:40.404493 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.404108    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt" event={"ID":"9164b080-801e-41b3-bcca-5b7d42ca6204","Type":"ContainerDied","Data":"c1ad170def8c713b9c894e94998efb73bd99bc82091259e9c493c2ccfe6c0891"}
Nov 03 01:39:40.404664 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.404651    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1ad170def8c713b9c894e94998efb73bd99bc82091259e9c493c2ccfe6c0891"
Nov 03 01:39:40.410930 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.410889    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9164b080-801e-41b3-bcca-5b7d42ca6204-kube-api-access-cxcsh" pod "9164b080-801e-41b3-bcca-5b7d42ca6204" (UID: "9164b080-801e-41b3-bcca-5b7d42ca6204"). InnerVolumeSpecName "kube-api-access-cxcsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:40.538448 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.533737    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:39:40.538448 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.535732    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxcsh\" (UniqueName: \"kubernetes.io/projected/9164b080-801e-41b3-bcca-5b7d42ca6204-kube-api-access-cxcsh\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:40.559379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.549276    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:39:40.612708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.585691    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:39:40.961833 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.961793    5508 scope.go:122] "RemoveContainer" containerID="607878e4c2921c2b0a4302461785d5bfe52561f7d28c8d54af5e8f171bf7d3d7"
Nov 03 01:39:40.975213 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:40.975188    5508 scope.go:122] "RemoveContainer" containerID="e80c2ee021c84ac12b2580f38ebf3ebf4c0b4eef77f329a1c0da8ad685edba3b"
Nov 03 01:39:41.016942 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.016914    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"]
Nov 03 01:39:41.017740 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.017720    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9164b080-801e-41b3-bcca-5b7d42ca6204" containerName="revoke-keys"
Nov 03 01:39:41.017799 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.017745    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9164b080-801e-41b3-bcca-5b7d42ca6204" containerName="revoke-keys"
Nov 03 01:39:41.017900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.017890    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9164b080-801e-41b3-bcca-5b7d42ca6204" containerName="revoke-keys"
Nov 03 01:39:41.042510 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.042480    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"
Nov 03 01:39:41.054602 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.054576    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e9ea64f-pwn4f\" (UID: \"259fb926-d52a-42c5-8e29-ccd173550559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"
Nov 03 01:39:41.054840 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.054823    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e9ea64f-pwn4f\" (UID: \"259fb926-d52a-42c5-8e29-ccd173550559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"
Nov 03 01:39:41.054946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.054930    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bnn7\" (UniqueName: \"kubernetes.io/projected/259fb926-d52a-42c5-8e29-ccd173550559-kube-api-access-7bnn7\") pod \"maas-api-revoke-sub-5e9ea64f-pwn4f\" (UID: \"259fb926-d52a-42c5-8e29-ccd173550559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"
Nov 03 01:39:41.076172 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.076089    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"]
Nov 03 01:39:41.091868 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.091827    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:39:41.092505 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.092474    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:39:41.092625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.092527    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:39:41.092625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.092563    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:39:41.109640 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.108482    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"]
Nov 03 01:39:41.111640 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:39:41.111611    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53333dac_248a_4cf1_ac30_c1962d83d8b9.slice/crio-227b4f001481fe2acd8293b5e3e69d4b9b103ec7f14c7f57e0a89513f9a9ea23 WatchSource:0}: Error finding container 227b4f001481fe2acd8293b5e3e69d4b9b103ec7f14c7f57e0a89513f9a9ea23: Status 404 returned error can't find the container with id 227b4f001481fe2acd8293b5e3e69d4b9b103ec7f14c7f57e0a89513f9a9ea23
Nov 03 01:39:41.114442 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.114403    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:41.114566 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.114493    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:41.114566 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.114541    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:41.156866 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.156825    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e9ea64f-pwn4f\" (UID: \"259fb926-d52a-42c5-8e29-ccd173550559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"
Nov 03 01:39:41.157074 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.156918    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e9ea64f-pwn4f\" (UID: \"259fb926-d52a-42c5-8e29-ccd173550559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"
Nov 03 01:39:41.157074 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.156942    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bnn7\" (UniqueName: \"kubernetes.io/projected/259fb926-d52a-42c5-8e29-ccd173550559-kube-api-access-7bnn7\") pod \"maas-api-revoke-sub-5e9ea64f-pwn4f\" (UID: \"259fb926-d52a-42c5-8e29-ccd173550559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"
Nov 03 01:39:41.157615 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.157591    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e9ea64f-pwn4f\" (UID: \"259fb926-d52a-42c5-8e29-ccd173550559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"
Nov 03 01:39:41.159758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.159651    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e9ea64f-pwn4f\" (UID: \"259fb926-d52a-42c5-8e29-ccd173550559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"
Nov 03 01:39:41.226046 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.226007    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bnn7\" (UniqueName: \"kubernetes.io/projected/259fb926-d52a-42c5-8e29-ccd173550559-kube-api-access-7bnn7\") pod \"maas-api-revoke-sub-5e9ea64f-pwn4f\" (UID: \"259fb926-d52a-42c5-8e29-ccd173550559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"
Nov 03 01:39:41.366773 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.366683    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"
Nov 03 01:39:41.435299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.434673    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" event={"ID":"53333dac-248a-4cf1-ac30-c1962d83d8b9","Type":"ContainerStarted","Data":"88ccdcfb3405094953e7c5668e14fb2e542c46a8fabd1d61e969b8331d437ded"}
Nov 03 01:39:41.435299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.434726    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" event={"ID":"53333dac-248a-4cf1-ac30-c1962d83d8b9","Type":"ContainerStarted","Data":"227b4f001481fe2acd8293b5e3e69d4b9b103ec7f14c7f57e0a89513f9a9ea23"}
Nov 03 01:39:41.550942 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.548955    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" podStartSLOduration=3.548935331 podStartE2EDuration="3.548935331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:41.548197452 +0000 UTC m=+1260.973803933" watchObservedRunningTime="2026-11-03 01:39:41.548935331 +0000 UTC m=+1260.974541811"
Nov 03 01:39:41.815610 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:39:41.815579    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259fb926_d52a_42c5_8e29_ccd173550559.slice/crio-6f14ed3de1bfa41e57679fbe6fa9b1a645eef9a37c2a8dfcaa203f6c24752397 WatchSource:0}: Error finding container 6f14ed3de1bfa41e57679fbe6fa9b1a645eef9a37c2a8dfcaa203f6c24752397: Status 404 returned error can't find the container with id 6f14ed3de1bfa41e57679fbe6fa9b1a645eef9a37c2a8dfcaa203f6c24752397
Nov 03 01:39:41.836529 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:41.836498    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"]
Nov 03 01:39:42.048684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.048636    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"]
Nov 03 01:39:42.060811 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.060779    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.077335 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.075701    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:42.078583 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.078051    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 01:39:42.078583 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.078103    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" podUID="53333dac-248a-4cf1-ac30-c1962d83d8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 01:39:42.084468 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.084390    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-plugins-config-volume\") pod \"payload-processing-external-model-88bf9945c-bx7lg\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.085230 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.084765    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-routing-overlay-volume\") pod \"payload-processing-external-model-88bf9945c-bx7lg\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.085230 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.084807    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87m6x\" (UniqueName: \"kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-kube-api-access-87m6x\") pod \"payload-processing-external-model-88bf9945c-bx7lg\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.085230 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.085052    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-provider-credentials\") pod \"payload-processing-external-model-88bf9945c-bx7lg\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.107093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.107059    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"]
Nov 03 01:39:42.187151 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.187118    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-provider-credentials\") pod \"payload-processing-external-model-88bf9945c-bx7lg\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.187282 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.187215    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-plugins-config-volume\") pod \"payload-processing-external-model-88bf9945c-bx7lg\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.187282 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.187255    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-routing-overlay-volume\") pod \"payload-processing-external-model-88bf9945c-bx7lg\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.187282 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.187275    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87m6x\" (UniqueName: \"kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-kube-api-access-87m6x\") pod \"payload-processing-external-model-88bf9945c-bx7lg\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.189014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.188947    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-routing-overlay-volume\") pod \"payload-processing-external-model-88bf9945c-bx7lg\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.189014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.188950    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-plugins-config-volume\") pod \"payload-processing-external-model-88bf9945c-bx7lg\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.190659 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.190633    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-provider-credentials\") pod \"payload-processing-external-model-88bf9945c-bx7lg\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.245041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.245009    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87m6x\" (UniqueName: \"kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-kube-api-access-87m6x\") pod \"payload-processing-external-model-88bf9945c-bx7lg\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") " pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.399783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.399754    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:42.446411 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.446321    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f" event={"ID":"259fb926-d52a-42c5-8e29-ccd173550559","Type":"ContainerStarted","Data":"523e48ed71911e11cdca79d41c6073e5d764d4fdadce898e6d524c396a6c9820"}
Nov 03 01:39:42.446411 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.446385    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f" event={"ID":"259fb926-d52a-42c5-8e29-ccd173550559","Type":"ContainerStarted","Data":"6f14ed3de1bfa41e57679fbe6fa9b1a645eef9a37c2a8dfcaa203f6c24752397"}
Nov 03 01:39:42.806249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.806132    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f" podStartSLOduration=2.806111411 podStartE2EDuration="2.806111411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:42.586087544 +0000 UTC m=+1262.011694038" watchObservedRunningTime="2026-11-03 01:39:42.806111411 +0000 UTC m=+1262.231717877"
Nov 03 01:39:42.811423 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:42.811369    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"]
Nov 03 01:39:42.815052 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:39:42.815021    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec0e8c5_9468_4573_a617_f91b24b36d1a.slice/crio-3fe933fc898b02b07e54eb138cd7e8bfab336c07dee3beb74dec5b4afc9e7952 WatchSource:0}: Error finding container 3fe933fc898b02b07e54eb138cd7e8bfab336c07dee3beb74dec5b4afc9e7952: Status 404 returned error can't find the container with id 3fe933fc898b02b07e54eb138cd7e8bfab336c07dee3beb74dec5b4afc9e7952
Nov 03 01:39:43.071997 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:43.071901    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 01:39:43.072625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:43.072582    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" podUID="53333dac-248a-4cf1-ac30-c1962d83d8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 01:39:43.493336 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:43.493285    5508 generic.go:356] "Generic (PLEG): container finished" podID="259fb926-d52a-42c5-8e29-ccd173550559" containerID="523e48ed71911e11cdca79d41c6073e5d764d4fdadce898e6d524c396a6c9820" exitCode=0
Nov 03 01:39:43.493734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:43.493398    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f" event={"ID":"259fb926-d52a-42c5-8e29-ccd173550559","Type":"ContainerDied","Data":"523e48ed71911e11cdca79d41c6073e5d764d4fdadce898e6d524c396a6c9820"}
Nov 03 01:39:43.495706 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:43.495618    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg" event={"ID":"0ec0e8c5-9468-4573-a617-f91b24b36d1a","Type":"ContainerStarted","Data":"94a6e44ecd98f848a2d9d9e4eaaa74c0cc701fa44e01acf771604293bfead77f"}
Nov 03 01:39:43.495706 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:43.495664    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg" event={"ID":"0ec0e8c5-9468-4573-a617-f91b24b36d1a","Type":"ContainerStarted","Data":"3fe933fc898b02b07e54eb138cd7e8bfab336c07dee3beb74dec5b4afc9e7952"}
Nov 03 01:39:43.495706 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:43.495686    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:43.716425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:43.716253    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg" podStartSLOduration=1.7162299920000001 podStartE2EDuration="1.716229992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:43.712164118 +0000 UTC m=+1263.137770582" watchObservedRunningTime="2026-11-03 01:39:43.716229992 +0000 UTC m=+1263.141836456"
Nov 03 01:39:43.845928 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:39:43.845836    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:39:44.073900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.073852    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 01:39:44.074340 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.073930    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" podUID="53333dac-248a-4cf1-ac30-c1962d83d8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 01:39:44.788909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.788829    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"
Nov 03 01:39:44.835662 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.835633    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-openshift-ca\") pod \"259fb926-d52a-42c5-8e29-ccd173550559\" (UID: \"259fb926-d52a-42c5-8e29-ccd173550559\") "
Nov 03 01:39:44.835825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.835747    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-xks-ca\") pod \"259fb926-d52a-42c5-8e29-ccd173550559\" (UID: \"259fb926-d52a-42c5-8e29-ccd173550559\") "
Nov 03 01:39:44.835825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.835786    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259fb926-d52a-42c5-8e29-ccd173550559-kube-api-access-7bnn7\" (UniqueName: \"kubernetes.io/projected/259fb926-d52a-42c5-8e29-ccd173550559-kube-api-access-7bnn7\") pod \"259fb926-d52a-42c5-8e29-ccd173550559\" (UID: \"259fb926-d52a-42c5-8e29-ccd173550559\") "
Nov 03 01:39:44.836097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.836053    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-openshift-ca" pod "259fb926-d52a-42c5-8e29-ccd173550559" (UID: "259fb926-d52a-42c5-8e29-ccd173550559"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:44.836540 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.836514    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:44.838153 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.838126    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259fb926-d52a-42c5-8e29-ccd173550559-kube-api-access-7bnn7" pod "259fb926-d52a-42c5-8e29-ccd173550559" (UID: "259fb926-d52a-42c5-8e29-ccd173550559"). InnerVolumeSpecName "kube-api-access-7bnn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:44.838272 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.838194    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-xks-ca" pod "259fb926-d52a-42c5-8e29-ccd173550559" (UID: "259fb926-d52a-42c5-8e29-ccd173550559"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:44.843198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.843173    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:39:44.937381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.937300    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5da42a7e-c55b-451e-b887-ea9d8461f23b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da42a7e-c55b-451e-b887-ea9d8461f23b-istiod-ca-cert\") pod \"5da42a7e-c55b-451e-b887-ea9d8461f23b\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") "
Nov 03 01:39:44.937381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.937383    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-credential-socket\") pod \"5da42a7e-c55b-451e-b887-ea9d8461f23b\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") "
Nov 03 01:39:44.937635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.937423    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-token\" (UniqueName: \"kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-token\") pod \"5da42a7e-c55b-451e-b887-ea9d8461f23b\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") "
Nov 03 01:39:44.937635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.937476    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-kube-api-access-2mn46\" (UniqueName: \"kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-kube-api-access-2mn46\") pod \"5da42a7e-c55b-451e-b887-ea9d8461f23b\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") "
Nov 03 01:39:44.937635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.937524    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-certs\") pod \"5da42a7e-c55b-451e-b887-ea9d8461f23b\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") "
Nov 03 01:39:44.937635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.937554    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-podinfo\") pod \"5da42a7e-c55b-451e-b887-ea9d8461f23b\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") "
Nov 03 01:39:44.937635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.937606    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-socket\") pod \"5da42a7e-c55b-451e-b887-ea9d8461f23b\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") "
Nov 03 01:39:44.937908 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.937892    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-envoy\") pod \"5da42a7e-c55b-451e-b887-ea9d8461f23b\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") "
Nov 03 01:39:44.937982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.937924    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-data\") pod \"5da42a7e-c55b-451e-b887-ea9d8461f23b\" (UID: \"5da42a7e-c55b-451e-b887-ea9d8461f23b\") "
Nov 03 01:39:44.938407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.938380    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/259fb926-d52a-42c5-8e29-ccd173550559-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:44.938407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.938407    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bnn7\" (UniqueName: \"kubernetes.io/projected/259fb926-d52a-42c5-8e29-ccd173550559-kube-api-access-7bnn7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:44.938768 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.938689    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-data" pod "5da42a7e-c55b-451e-b887-ea9d8461f23b" (UID: "5da42a7e-c55b-451e-b887-ea9d8461f23b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:39:44.939232 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.938837    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-credential-socket" pod "5da42a7e-c55b-451e-b887-ea9d8461f23b" (UID: "5da42a7e-c55b-451e-b887-ea9d8461f23b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:39:44.939306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.939228    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-certs" pod "5da42a7e-c55b-451e-b887-ea9d8461f23b" (UID: "5da42a7e-c55b-451e-b887-ea9d8461f23b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:39:44.939306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.939166    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-socket" pod "5da42a7e-c55b-451e-b887-ea9d8461f23b" (UID: "5da42a7e-c55b-451e-b887-ea9d8461f23b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:39:44.939306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.939197    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da42a7e-c55b-451e-b887-ea9d8461f23b-istiod-ca-cert" pod "5da42a7e-c55b-451e-b887-ea9d8461f23b" (UID: "5da42a7e-c55b-451e-b887-ea9d8461f23b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:44.941752 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.941713    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-token" pod "5da42a7e-c55b-451e-b887-ea9d8461f23b" (UID: "5da42a7e-c55b-451e-b887-ea9d8461f23b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:44.941868 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.941763    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-podinfo" pod "5da42a7e-c55b-451e-b887-ea9d8461f23b" (UID: "5da42a7e-c55b-451e-b887-ea9d8461f23b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:39:44.941868 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.941778    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-envoy" pod "5da42a7e-c55b-451e-b887-ea9d8461f23b" (UID: "5da42a7e-c55b-451e-b887-ea9d8461f23b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:39:44.942100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:44.942082    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-kube-api-access-2mn46" pod "5da42a7e-c55b-451e-b887-ea9d8461f23b" (UID: "5da42a7e-c55b-451e-b887-ea9d8461f23b"). InnerVolumeSpecName "kube-api-access-2mn46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:45.044994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.044885    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:45.044994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.044937    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:45.044994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.044955    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5da42a7e-c55b-451e-b887-ea9d8461f23b-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:45.044994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.044968    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:45.044994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.044982    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:45.045298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.044998    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mn46\" (UniqueName: \"kubernetes.io/projected/5da42a7e-c55b-451e-b887-ea9d8461f23b-kube-api-access-2mn46\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:45.045298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.045012    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:45.045298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.045025    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5da42a7e-c55b-451e-b887-ea9d8461f23b-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:45.045298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.045038    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5da42a7e-c55b-451e-b887-ea9d8461f23b-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:45.077410 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.073839    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 01:39:45.077410 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.073908    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" podUID="53333dac-248a-4cf1-ac30-c1962d83d8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 01:39:45.282294 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.282250    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"]
Nov 03 01:39:45.283061 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.283041    5508 cpu_manager.go:414] "containerMap: removing container" podUID="259fb926-d52a-42c5-8e29-ccd173550559" containerName="revoke-keys"
Nov 03 01:39:45.283061 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.283060    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259fb926-d52a-42c5-8e29-ccd173550559" containerName="revoke-keys"
Nov 03 01:39:45.283191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.283084    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerName="istio-proxy"
Nov 03 01:39:45.283191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.283092    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerName="istio-proxy"
Nov 03 01:39:45.283191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.283141    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerName="istio-proxy"
Nov 03 01:39:45.283191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.283149    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerName="istio-proxy"
Nov 03 01:39:45.283317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.283259    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerName="istio-proxy"
Nov 03 01:39:45.283317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.283272    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="259fb926-d52a-42c5-8e29-ccd173550559" containerName="revoke-keys"
Nov 03 01:39:45.283317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.283283    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerName="istio-proxy"
Nov 03 01:39:45.287645 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.287623    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"
Nov 03 01:39:45.326472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.326441    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"]
Nov 03 01:39:45.384007 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.383966    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjdn\" (UniqueName: \"kubernetes.io/projected/0fe508df-0678-4535-9d11-506497d147cf-kube-api-access-nbjdn\") pod \"maas-api-revoke-sub-5067c679-nzcfp\" (UID: \"0fe508df-0678-4535-9d11-506497d147cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"
Nov 03 01:39:45.384322 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.384129    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe508df-0678-4535-9d11-506497d147cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5067c679-nzcfp\" (UID: \"0fe508df-0678-4535-9d11-506497d147cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"
Nov 03 01:39:45.384322 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.384181    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe508df-0678-4535-9d11-506497d147cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5067c679-nzcfp\" (UID: \"0fe508df-0678-4535-9d11-506497d147cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"
Nov 03 01:39:45.486068 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.486021    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjdn\" (UniqueName: \"kubernetes.io/projected/0fe508df-0678-4535-9d11-506497d147cf-kube-api-access-nbjdn\") pod \"maas-api-revoke-sub-5067c679-nzcfp\" (UID: \"0fe508df-0678-4535-9d11-506497d147cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"
Nov 03 01:39:45.486405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.486113    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe508df-0678-4535-9d11-506497d147cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5067c679-nzcfp\" (UID: \"0fe508df-0678-4535-9d11-506497d147cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"
Nov 03 01:39:45.486405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.486143    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe508df-0678-4535-9d11-506497d147cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5067c679-nzcfp\" (UID: \"0fe508df-0678-4535-9d11-506497d147cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"
Nov 03 01:39:45.486886 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.486866    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe508df-0678-4535-9d11-506497d147cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5067c679-nzcfp\" (UID: \"0fe508df-0678-4535-9d11-506497d147cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"
Nov 03 01:39:45.488524 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.488502    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe508df-0678-4535-9d11-506497d147cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5067c679-nzcfp\" (UID: \"0fe508df-0678-4535-9d11-506497d147cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"
Nov 03 01:39:45.515163 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.515132    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"
Nov 03 01:39:45.515342 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.515160    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f" event={"ID":"259fb926-d52a-42c5-8e29-ccd173550559","Type":"ContainerDied","Data":"6f14ed3de1bfa41e57679fbe6fa9b1a645eef9a37c2a8dfcaa203f6c24752397"}
Nov 03 01:39:45.515342 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.515209    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f14ed3de1bfa41e57679fbe6fa9b1a645eef9a37c2a8dfcaa203f6c24752397"
Nov 03 01:39:45.517126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.517096    5508 generic.go:356] "Generic (PLEG): container finished" podID="5da42a7e-c55b-451e-b887-ea9d8461f23b" containerID="5e91e1f2cbf4db6236ff6b554283818e007ca6ebdd27cf723e04a4296fc1c5b9" exitCode=0
Nov 03 01:39:45.517235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.517140    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99" event={"ID":"5da42a7e-c55b-451e-b887-ea9d8461f23b","Type":"ContainerDied","Data":"5e91e1f2cbf4db6236ff6b554283818e007ca6ebdd27cf723e04a4296fc1c5b9"}
Nov 03 01:39:45.517235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.517156    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"
Nov 03 01:39:45.517235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.517189    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99" event={"ID":"5da42a7e-c55b-451e-b887-ea9d8461f23b","Type":"ContainerDied","Data":"798c1af3b8a070362122af39cea15db70ce5cd83cca7f4d09f5440dbf9e9a153"}
Nov 03 01:39:45.517235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.517219    5508 scope.go:122] "RemoveContainer" containerID="5e91e1f2cbf4db6236ff6b554283818e007ca6ebdd27cf723e04a4296fc1c5b9"
Nov 03 01:39:45.536904 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.536883    5508 scope.go:122] "RemoveContainer" containerID="b948d0ea76b602a2a348cb871d767262befb20cf54791f26704861f3560c8fd0"
Nov 03 01:39:45.543621 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.543597    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjdn\" (UniqueName: \"kubernetes.io/projected/0fe508df-0678-4535-9d11-506497d147cf-kube-api-access-nbjdn\") pod \"maas-api-revoke-sub-5067c679-nzcfp\" (UID: \"0fe508df-0678-4535-9d11-506497d147cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"
Nov 03 01:39:45.547667 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.547643    5508 scope.go:122] "RemoveContainer" containerID="5e91e1f2cbf4db6236ff6b554283818e007ca6ebdd27cf723e04a4296fc1c5b9"
Nov 03 01:39:45.548051 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:39:45.547991    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e91e1f2cbf4db6236ff6b554283818e007ca6ebdd27cf723e04a4296fc1c5b9\": container with ID starting with 5e91e1f2cbf4db6236ff6b554283818e007ca6ebdd27cf723e04a4296fc1c5b9 not found: ID does not exist" containerID="5e91e1f2cbf4db6236ff6b554283818e007ca6ebdd27cf723e04a4296fc1c5b9"
Nov 03 01:39:45.548051 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.548035    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e91e1f2cbf4db6236ff6b554283818e007ca6ebdd27cf723e04a4296fc1c5b9"} err="failed to get container status \"5e91e1f2cbf4db6236ff6b554283818e007ca6ebdd27cf723e04a4296fc1c5b9\": rpc error: code = NotFound desc = could not find container \"5e91e1f2cbf4db6236ff6b554283818e007ca6ebdd27cf723e04a4296fc1c5b9\": container with ID starting with 5e91e1f2cbf4db6236ff6b554283818e007ca6ebdd27cf723e04a4296fc1c5b9 not found: ID does not exist"
Nov 03 01:39:45.548188 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.548054    5508 scope.go:122] "RemoveContainer" containerID="b948d0ea76b602a2a348cb871d767262befb20cf54791f26704861f3560c8fd0"
Nov 03 01:39:45.548305 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:39:45.548287    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b948d0ea76b602a2a348cb871d767262befb20cf54791f26704861f3560c8fd0\": container with ID starting with b948d0ea76b602a2a348cb871d767262befb20cf54791f26704861f3560c8fd0 not found: ID does not exist" containerID="b948d0ea76b602a2a348cb871d767262befb20cf54791f26704861f3560c8fd0"
Nov 03 01:39:45.548379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.548313    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b948d0ea76b602a2a348cb871d767262befb20cf54791f26704861f3560c8fd0"} err="failed to get container status \"b948d0ea76b602a2a348cb871d767262befb20cf54791f26704861f3560c8fd0\": rpc error: code = NotFound desc = could not find container \"b948d0ea76b602a2a348cb871d767262befb20cf54791f26704861f3560c8fd0\": container with ID starting with b948d0ea76b602a2a348cb871d767262befb20cf54791f26704861f3560c8fd0 not found: ID does not exist"
Nov 03 01:39:45.606773 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.606742    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"
Nov 03 01:39:45.679132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.678805    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"]
Nov 03 01:39:45.683151 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:45.683069    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-jqq99"]
Nov 03 01:39:46.009721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:46.009682    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"]
Nov 03 01:39:46.015284 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:39:46.015252    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe508df_0678_4535_9d11_506497d147cf.slice/crio-8c5cd42c9ab5baf4173e7c29a1a318d4f3cb224f43ae64105b010e5c2b73c466 WatchSource:0}: Error finding container 8c5cd42c9ab5baf4173e7c29a1a318d4f3cb224f43ae64105b010e5c2b73c466: Status 404 returned error can't find the container with id 8c5cd42c9ab5baf4173e7c29a1a318d4f3cb224f43ae64105b010e5c2b73c466
Nov 03 01:39:46.073410 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:46.070917    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 01:39:46.073410 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:46.070975    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" podUID="53333dac-248a-4cf1-ac30-c1962d83d8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 01:39:46.527087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:46.527049    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp" event={"ID":"0fe508df-0678-4535-9d11-506497d147cf","Type":"ContainerStarted","Data":"104a0f259c0f7a018ac257a74f33323e3d42bf8252e5252ca6b1f4abd4cc2f0c"}
Nov 03 01:39:46.527570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:46.527141    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp" event={"ID":"0fe508df-0678-4535-9d11-506497d147cf","Type":"ContainerStarted","Data":"8c5cd42c9ab5baf4173e7c29a1a318d4f3cb224f43ae64105b010e5c2b73c466"}
Nov 03 01:39:46.643075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:46.642967    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp" podStartSLOduration=1.642952484 podStartE2EDuration="1.642952484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:46.640057617 +0000 UTC m=+1266.065664102" watchObservedRunningTime="2026-11-03 01:39:46.642952484 +0000 UTC m=+1266.068558945"
Nov 03 01:39:46.693489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:46.693454    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5da42a7e-c55b-451e-b887-ea9d8461f23b" path="/var/lib/kubelet/pods/5da42a7e-c55b-451e-b887-ea9d8461f23b/volumes"
Nov 03 01:39:47.073744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:47.073709    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 01:39:47.074058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:47.074023    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" podUID="53333dac-248a-4cf1-ac30-c1962d83d8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 01:39:47.541252 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:47.541213    5508 generic.go:356] "Generic (PLEG): container finished" podID="0fe508df-0678-4535-9d11-506497d147cf" containerID="104a0f259c0f7a018ac257a74f33323e3d42bf8252e5252ca6b1f4abd4cc2f0c" exitCode=0
Nov 03 01:39:47.541801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:47.541298    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp" event={"ID":"0fe508df-0678-4535-9d11-506497d147cf","Type":"ContainerDied","Data":"104a0f259c0f7a018ac257a74f33323e3d42bf8252e5252ca6b1f4abd4cc2f0c"}
Nov 03 01:39:48.071413 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:48.071367    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 01:39:48.071604 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:48.071442    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" podUID="53333dac-248a-4cf1-ac30-c1962d83d8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 01:39:48.835951 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:48.835923    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"
Nov 03 01:39:48.942962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:48.942929    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe508df-0678-4535-9d11-506497d147cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe508df-0678-4535-9d11-506497d147cf-maas-api-openshift-ca\") pod \"0fe508df-0678-4535-9d11-506497d147cf\" (UID: \"0fe508df-0678-4535-9d11-506497d147cf\") "
Nov 03 01:39:48.943187 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:48.943166    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe508df-0678-4535-9d11-506497d147cf-kube-api-access-nbjdn\" (UniqueName: \"kubernetes.io/projected/0fe508df-0678-4535-9d11-506497d147cf-kube-api-access-nbjdn\") pod \"0fe508df-0678-4535-9d11-506497d147cf\" (UID: \"0fe508df-0678-4535-9d11-506497d147cf\") "
Nov 03 01:39:48.943241 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:48.943220    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe508df-0678-4535-9d11-506497d147cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe508df-0678-4535-9d11-506497d147cf-maas-api-xks-ca\") pod \"0fe508df-0678-4535-9d11-506497d147cf\" (UID: \"0fe508df-0678-4535-9d11-506497d147cf\") "
Nov 03 01:39:48.943382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:48.943339    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe508df-0678-4535-9d11-506497d147cf-maas-api-openshift-ca" pod "0fe508df-0678-4535-9d11-506497d147cf" (UID: "0fe508df-0678-4535-9d11-506497d147cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:48.943679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:48.943658    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe508df-0678-4535-9d11-506497d147cf-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:48.945792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:48.945760    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe508df-0678-4535-9d11-506497d147cf-maas-api-xks-ca" pod "0fe508df-0678-4535-9d11-506497d147cf" (UID: "0fe508df-0678-4535-9d11-506497d147cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:48.959615 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:48.959532    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe508df-0678-4535-9d11-506497d147cf-kube-api-access-nbjdn" pod "0fe508df-0678-4535-9d11-506497d147cf" (UID: "0fe508df-0678-4535-9d11-506497d147cf"). InnerVolumeSpecName "kube-api-access-nbjdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:49.045252 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:49.045213    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbjdn\" (UniqueName: \"kubernetes.io/projected/0fe508df-0678-4535-9d11-506497d147cf-kube-api-access-nbjdn\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:49.045252 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:49.045253    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe508df-0678-4535-9d11-506497d147cf-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:49.076394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:49.076332    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:49.076889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:49.076864    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:49.077819 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:49.077794    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:39:49.609339 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:49.609288    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"
Nov 03 01:39:49.609339 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:49.609322    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp" event={"ID":"0fe508df-0678-4535-9d11-506497d147cf","Type":"ContainerDied","Data":"8c5cd42c9ab5baf4173e7c29a1a318d4f3cb224f43ae64105b010e5c2b73c466"}
Nov 03 01:39:49.609637 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:49.609405    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c5cd42c9ab5baf4173e7c29a1a318d4f3cb224f43ae64105b010e5c2b73c466"
Nov 03 01:39:50.118174 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.118138    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"]
Nov 03 01:39:50.118867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.118848    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0fe508df-0678-4535-9d11-506497d147cf" containerName="revoke-keys"
Nov 03 01:39:50.118912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.118871    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe508df-0678-4535-9d11-506497d147cf" containerName="revoke-keys"
Nov 03 01:39:50.118997 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.118987    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe508df-0678-4535-9d11-506497d147cf" containerName="revoke-keys"
Nov 03 01:39:50.122922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.122899    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"
Nov 03 01:39:50.171519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.171489    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:39:50.179249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.179209    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"]
Nov 03 01:39:50.217777 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.217743    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:39:50.217958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.217823    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:39:50.218111 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.218083    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:39:50.265736 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.265704    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06c2f5cb-lrqth\" (UID: \"04370439-8d45-472c-8af2-c69eca905ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"
Nov 03 01:39:50.265921 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.265819    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06c2f5cb-lrqth\" (UID: \"04370439-8d45-472c-8af2-c69eca905ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"
Nov 03 01:39:50.265991 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.265973    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcxhb\" (UniqueName: \"kubernetes.io/projected/04370439-8d45-472c-8af2-c69eca905ea4-kube-api-access-vcxhb\") pod \"maas-api-revoke-sub-06c2f5cb-lrqth\" (UID: \"04370439-8d45-472c-8af2-c69eca905ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"
Nov 03 01:39:50.367641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.367589    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06c2f5cb-lrqth\" (UID: \"04370439-8d45-472c-8af2-c69eca905ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"
Nov 03 01:39:50.367798 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.367780    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcxhb\" (UniqueName: \"kubernetes.io/projected/04370439-8d45-472c-8af2-c69eca905ea4-kube-api-access-vcxhb\") pod \"maas-api-revoke-sub-06c2f5cb-lrqth\" (UID: \"04370439-8d45-472c-8af2-c69eca905ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"
Nov 03 01:39:50.367887 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.367869    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06c2f5cb-lrqth\" (UID: \"04370439-8d45-472c-8af2-c69eca905ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"
Nov 03 01:39:50.368427 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.368324    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06c2f5cb-lrqth\" (UID: \"04370439-8d45-472c-8af2-c69eca905ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"
Nov 03 01:39:50.370321 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.370295    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06c2f5cb-lrqth\" (UID: \"04370439-8d45-472c-8af2-c69eca905ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"
Nov 03 01:39:50.425611 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.425579    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcxhb\" (UniqueName: \"kubernetes.io/projected/04370439-8d45-472c-8af2-c69eca905ea4-kube-api-access-vcxhb\") pod \"maas-api-revoke-sub-06c2f5cb-lrqth\" (UID: \"04370439-8d45-472c-8af2-c69eca905ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"
Nov 03 01:39:50.437473 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.437438    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"
Nov 03 01:39:50.820366 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:50.820310    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"]
Nov 03 01:39:50.826368 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:39:50.826316    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04370439_8d45_472c_8af2_c69eca905ea4.slice/crio-ff29b0a0b3eb0d3d02239e1ad25ba3d92559a2baabc2ed86a9c8d4e7a305cb2d WatchSource:0}: Error finding container ff29b0a0b3eb0d3d02239e1ad25ba3d92559a2baabc2ed86a9c8d4e7a305cb2d: Status 404 returned error can't find the container with id ff29b0a0b3eb0d3d02239e1ad25ba3d92559a2baabc2ed86a9c8d4e7a305cb2d
Nov 03 01:39:51.626391 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:51.626287    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth" event={"ID":"04370439-8d45-472c-8af2-c69eca905ea4","Type":"ContainerStarted","Data":"9d27ad3774f857bc2c4cfa3759d1aefa44e83b9601156a63dae3fcbd665919df"}
Nov 03 01:39:51.626391 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:51.626384    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth" event={"ID":"04370439-8d45-472c-8af2-c69eca905ea4","Type":"ContainerStarted","Data":"ff29b0a0b3eb0d3d02239e1ad25ba3d92559a2baabc2ed86a9c8d4e7a305cb2d"}
Nov 03 01:39:51.736867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:51.736811    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth" podStartSLOduration=1.736793967 podStartE2EDuration="1.736793967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:51.73459342 +0000 UTC m=+1271.160199883" watchObservedRunningTime="2026-11-03 01:39:51.736793967 +0000 UTC m=+1271.162400431"
Nov 03 01:39:52.635432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.635398    5508 generic.go:356] "Generic (PLEG): container finished" podID="04370439-8d45-472c-8af2-c69eca905ea4" containerID="9d27ad3774f857bc2c4cfa3759d1aefa44e83b9601156a63dae3fcbd665919df" exitCode=0
Nov 03 01:39:52.635834 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.635481    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth" event={"ID":"04370439-8d45-472c-8af2-c69eca905ea4","Type":"ContainerDied","Data":"9d27ad3774f857bc2c4cfa3759d1aefa44e83b9601156a63dae3fcbd665919df"}
Nov 03 01:39:52.661536 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.661489    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"]
Nov 03 01:39:52.666791 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.666768    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"
Nov 03 01:39:52.717430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.717390    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"]
Nov 03 01:39:52.800070 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.800025    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c92765ce-nxbg4\" (UID: \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"
Nov 03 01:39:52.800486 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.800448    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggkb\" (UniqueName: \"kubernetes.io/projected/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-kube-api-access-6ggkb\") pod \"maas-api-revoke-sub-c92765ce-nxbg4\" (UID: \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"
Nov 03 01:39:52.800624 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.800587    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c92765ce-nxbg4\" (UID: \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"
Nov 03 01:39:52.902637 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.902598    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c92765ce-nxbg4\" (UID: \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"
Nov 03 01:39:52.902859 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.902770    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggkb\" (UniqueName: \"kubernetes.io/projected/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-kube-api-access-6ggkb\") pod \"maas-api-revoke-sub-c92765ce-nxbg4\" (UID: \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"
Nov 03 01:39:52.902859 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.902832    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c92765ce-nxbg4\" (UID: \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"
Nov 03 01:39:52.903296 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.903277    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c92765ce-nxbg4\" (UID: \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"
Nov 03 01:39:52.905133 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.905115    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c92765ce-nxbg4\" (UID: \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"
Nov 03 01:39:52.960481 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.960443    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggkb\" (UniqueName: \"kubernetes.io/projected/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-kube-api-access-6ggkb\") pod \"maas-api-revoke-sub-c92765ce-nxbg4\" (UID: \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"
Nov 03 01:39:52.980808 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:52.980774    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"
Nov 03 01:39:53.287151 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:53.287063    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"]
Nov 03 01:39:53.297538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:53.297503    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"]
Nov 03 01:39:53.310700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:53.310607    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg" podUID="0ec0e8c5-9468-4573-a617-f91b24b36d1a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:39:53.602445 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:53.602365    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"]
Nov 03 01:39:53.608489 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:39:53.608461    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825d4108_3d57_4c2c_8831_ceb66e8e4ad8.slice/crio-e197b0d928af5c3a5425cbc4613d225d3fef37949a17cc43e2fd0a619b7806ed WatchSource:0}: Error finding container e197b0d928af5c3a5425cbc4613d225d3fef37949a17cc43e2fd0a619b7806ed: Status 404 returned error can't find the container with id e197b0d928af5c3a5425cbc4613d225d3fef37949a17cc43e2fd0a619b7806ed
Nov 03 01:39:53.644071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:53.644044    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4" event={"ID":"825d4108-3d57-4c2c-8831-ceb66e8e4ad8","Type":"ContainerStarted","Data":"e197b0d928af5c3a5425cbc4613d225d3fef37949a17cc43e2fd0a619b7806ed"}
Nov 03 01:39:54.232576 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.232550    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"
Nov 03 01:39:54.341827 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.341787    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-xks-ca\") pod \"04370439-8d45-472c-8af2-c69eca905ea4\" (UID: \"04370439-8d45-472c-8af2-c69eca905ea4\") "
Nov 03 01:39:54.342013 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.341873    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04370439-8d45-472c-8af2-c69eca905ea4-kube-api-access-vcxhb\" (UniqueName: \"kubernetes.io/projected/04370439-8d45-472c-8af2-c69eca905ea4-kube-api-access-vcxhb\") pod \"04370439-8d45-472c-8af2-c69eca905ea4\" (UID: \"04370439-8d45-472c-8af2-c69eca905ea4\") "
Nov 03 01:39:54.342013 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.341924    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-openshift-ca\") pod \"04370439-8d45-472c-8af2-c69eca905ea4\" (UID: \"04370439-8d45-472c-8af2-c69eca905ea4\") "
Nov 03 01:39:54.343368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.343321    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-openshift-ca" pod "04370439-8d45-472c-8af2-c69eca905ea4" (UID: "04370439-8d45-472c-8af2-c69eca905ea4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:54.344823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.344791    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:54.347909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.347874    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-xks-ca" pod "04370439-8d45-472c-8af2-c69eca905ea4" (UID: "04370439-8d45-472c-8af2-c69eca905ea4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:54.348041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.347879    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04370439-8d45-472c-8af2-c69eca905ea4-kube-api-access-vcxhb" pod "04370439-8d45-472c-8af2-c69eca905ea4" (UID: "04370439-8d45-472c-8af2-c69eca905ea4"). InnerVolumeSpecName "kube-api-access-vcxhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:54.446132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.446045    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04370439-8d45-472c-8af2-c69eca905ea4-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:54.446132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.446079    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcxhb\" (UniqueName: \"kubernetes.io/projected/04370439-8d45-472c-8af2-c69eca905ea4-kube-api-access-vcxhb\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:54.654117 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.654079    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"
Nov 03 01:39:54.654117 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.654090    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth" event={"ID":"04370439-8d45-472c-8af2-c69eca905ea4","Type":"ContainerDied","Data":"ff29b0a0b3eb0d3d02239e1ad25ba3d92559a2baabc2ed86a9c8d4e7a305cb2d"}
Nov 03 01:39:54.654611 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.654151    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff29b0a0b3eb0d3d02239e1ad25ba3d92559a2baabc2ed86a9c8d4e7a305cb2d"
Nov 03 01:39:54.655960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.655932    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4" event={"ID":"825d4108-3d57-4c2c-8831-ceb66e8e4ad8","Type":"ContainerStarted","Data":"fce1be557495f130b639e14c1e6af19dbdfe0627def3263a3cfcc6fb6720a0c3"}
Nov 03 01:39:54.766969 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:54.766867    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4" podStartSLOduration=2.766852388 podStartE2EDuration="2.766852388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:54.764135894 +0000 UTC m=+1274.189742356" watchObservedRunningTime="2026-11-03 01:39:54.766852388 +0000 UTC m=+1274.192458891"
Nov 03 01:39:55.342079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.342039    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"]
Nov 03 01:39:55.342663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.342647    5508 cpu_manager.go:414] "containerMap: removing container" podUID="04370439-8d45-472c-8af2-c69eca905ea4" containerName="revoke-keys"
Nov 03 01:39:55.342727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.342664    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04370439-8d45-472c-8af2-c69eca905ea4" containerName="revoke-keys"
Nov 03 01:39:55.342762 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.342753    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="04370439-8d45-472c-8af2-c69eca905ea4" containerName="revoke-keys"
Nov 03 01:39:55.346338 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.346318    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.448709 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.448665    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"]
Nov 03 01:39:55.459153 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.459100    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.459369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.459270    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.459369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.459308    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.459487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.459393    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.459487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.459460    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.459592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.459528    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wh89\" (UniqueName: \"kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-kube-api-access-2wh89\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.459592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.459570    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.459692 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.459627    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.459746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.459689    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561070 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561030    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561255 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561085    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561255 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561128    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561255 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561163    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561377    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wh89\" (UniqueName: \"kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-kube-api-access-2wh89\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561479 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561430    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561525 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561473    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561523    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-data\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561555    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-certs\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561642 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561560    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561718 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561699    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561803 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561785    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-credential-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561868 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561848    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-socket\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.561954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.561930    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istiod-ca-cert\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.563926 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.563895    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-envoy\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.564215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.564195    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-podinfo\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.618344 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.618276    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-token\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.618515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.618365    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wh89\" (UniqueName: \"kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-kube-api-access-2wh89\") pod \"e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") " pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.664997 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.662358    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:55.676216 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.676176    5508 generic.go:356] "Generic (PLEG): container finished" podID="825d4108-3d57-4c2c-8831-ceb66e8e4ad8" containerID="fce1be557495f130b639e14c1e6af19dbdfe0627def3263a3cfcc6fb6720a0c3" exitCode=0
Nov 03 01:39:55.676394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.676237    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4" event={"ID":"825d4108-3d57-4c2c-8831-ceb66e8e4ad8","Type":"ContainerDied","Data":"fce1be557495f130b639e14c1e6af19dbdfe0627def3263a3cfcc6fb6720a0c3"}
Nov 03 01:39:55.992919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.992883    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"]
Nov 03 01:39:55.998311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:55.998289    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"
Nov 03 01:39:56.051650 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.051615    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"]
Nov 03 01:39:56.070583 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.070555    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8812863-c2dns\" (UID: \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"
Nov 03 01:39:56.070722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.070629    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xl9s\" (UniqueName: \"kubernetes.io/projected/63c0eb6d-2862-4fca-8d0f-a1cca074a892-kube-api-access-8xl9s\") pod \"maas-api-revoke-sub-f8812863-c2dns\" (UID: \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"
Nov 03 01:39:56.070805 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.070777    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8812863-c2dns\" (UID: \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"
Nov 03 01:39:56.103991 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.103953    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"]
Nov 03 01:39:56.109985 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:39:56.109953    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ddd202_6ce9_48df_b027_86ac0620f0c8.slice/crio-2a96336ad235ab1c25bca49b27008cc7ae11db660413d2015f64ff27093064ed WatchSource:0}: Error finding container 2a96336ad235ab1c25bca49b27008cc7ae11db660413d2015f64ff27093064ed: Status 404 returned error can't find the container with id 2a96336ad235ab1c25bca49b27008cc7ae11db660413d2015f64ff27093064ed
Nov 03 01:39:56.112387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.112339    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:56.112458 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.112420    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:56.112458 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.112454    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:39:56.172650 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.172622    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8812863-c2dns\" (UID: \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"
Nov 03 01:39:56.172801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.172697    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xl9s\" (UniqueName: \"kubernetes.io/projected/63c0eb6d-2862-4fca-8d0f-a1cca074a892-kube-api-access-8xl9s\") pod \"maas-api-revoke-sub-f8812863-c2dns\" (UID: \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"
Nov 03 01:39:56.172801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.172744    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8812863-c2dns\" (UID: \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"
Nov 03 01:39:56.173399 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.173380    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8812863-c2dns\" (UID: \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"
Nov 03 01:39:56.174846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.174827    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8812863-c2dns\" (UID: \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"
Nov 03 01:39:56.229936 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.229904    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xl9s\" (UniqueName: \"kubernetes.io/projected/63c0eb6d-2862-4fca-8d0f-a1cca074a892-kube-api-access-8xl9s\") pod \"maas-api-revoke-sub-f8812863-c2dns\" (UID: \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"
Nov 03 01:39:56.312096 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.312002    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"
Nov 03 01:39:56.695091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.695057    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" event={"ID":"c2ddd202-6ce9-48df-b027-86ac0620f0c8","Type":"ContainerStarted","Data":"671323f45e8d45b816114fc29f4aedcec34269068b5523edf5e4e1fe6cadc2fa"}
Nov 03 01:39:56.695485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.695118    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" event={"ID":"c2ddd202-6ce9-48df-b027-86ac0620f0c8","Type":"ContainerStarted","Data":"2a96336ad235ab1c25bca49b27008cc7ae11db660413d2015f64ff27093064ed"}
Nov 03 01:39:56.699460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.699434    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"]
Nov 03 01:39:56.702333 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:39:56.702306    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c0eb6d_2862_4fca_8d0f_a1cca074a892.slice/crio-95a39fc29ad5c5d8a07daeb9e870369c6200664c5487ec1d937d54ccaab6b351 WatchSource:0}: Error finding container 95a39fc29ad5c5d8a07daeb9e870369c6200664c5487ec1d937d54ccaab6b351: Status 404 returned error can't find the container with id 95a39fc29ad5c5d8a07daeb9e870369c6200664c5487ec1d937d54ccaab6b351
Nov 03 01:39:56.804633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.804570    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podStartSLOduration=1.8045479210000002 podStartE2EDuration="1.804547921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:56.800283784 +0000 UTC m=+1276.225890268" watchObservedRunningTime="2026-11-03 01:39:56.804547921 +0000 UTC m=+1276.230154385"
Nov 03 01:39:56.879294 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:56.879267    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"
Nov 03 01:39:57.010851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.010763    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-xks-ca\") pod \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\" (UID: \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\") "
Nov 03 01:39:57.010851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.010815    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-kube-api-access-6ggkb\" (UniqueName: \"kubernetes.io/projected/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-kube-api-access-6ggkb\") pod \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\" (UID: \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\") "
Nov 03 01:39:57.011081 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.010920    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-openshift-ca\") pod \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\" (UID: \"825d4108-3d57-4c2c-8831-ceb66e8e4ad8\") "
Nov 03 01:39:57.011551 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.011505    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-openshift-ca" pod "825d4108-3d57-4c2c-8831-ceb66e8e4ad8" (UID: "825d4108-3d57-4c2c-8831-ceb66e8e4ad8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:57.012227 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.012206    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:57.085190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.085158    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-xks-ca" pod "825d4108-3d57-4c2c-8831-ceb66e8e4ad8" (UID: "825d4108-3d57-4c2c-8831-ceb66e8e4ad8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:57.087798 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.085914    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-kube-api-access-6ggkb" pod "825d4108-3d57-4c2c-8831-ceb66e8e4ad8" (UID: "825d4108-3d57-4c2c-8831-ceb66e8e4ad8"). InnerVolumeSpecName "kube-api-access-6ggkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:57.113914 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.113887    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:57.113914 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.113917    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ggkb\" (UniqueName: \"kubernetes.io/projected/825d4108-3d57-4c2c-8831-ceb66e8e4ad8-kube-api-access-6ggkb\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:57.671372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.671002    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:39:57.671372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.671129    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:39:57.671372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.671171    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:39:57.698485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.698455    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"
Nov 03 01:39:57.698960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.698484    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4" event={"ID":"825d4108-3d57-4c2c-8831-ceb66e8e4ad8","Type":"ContainerDied","Data":"e197b0d928af5c3a5425cbc4613d225d3fef37949a17cc43e2fd0a619b7806ed"}
Nov 03 01:39:57.698960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.698530    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e197b0d928af5c3a5425cbc4613d225d3fef37949a17cc43e2fd0a619b7806ed"
Nov 03 01:39:57.700224 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.700199    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns" event={"ID":"63c0eb6d-2862-4fca-8d0f-a1cca074a892","Type":"ContainerStarted","Data":"dbf078c877bac15827a95997b71a25c96d3ed3bf004c451953062be4cfba81fc"}
Nov 03 01:39:57.700360 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.700244    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns" event={"ID":"63c0eb6d-2862-4fca-8d0f-a1cca074a892","Type":"ContainerStarted","Data":"95a39fc29ad5c5d8a07daeb9e870369c6200664c5487ec1d937d54ccaab6b351"}
Nov 03 01:39:57.825870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:57.825814    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns" podStartSLOduration=2.825798564 podStartE2EDuration="2.825798564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:57.823062683 +0000 UTC m=+1277.248669148" watchObservedRunningTime="2026-11-03 01:39:57.825798564 +0000 UTC m=+1277.251405029"
Nov 03 01:39:58.303521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.303472    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x" podUID="88d0256b-82df-4484-b5a7-e077303869ac" containerName="payload-processing" containerID="cri-o://5b63cb24a38890a63866d786de9331e8927e0bf7acc9931aeafa29d0464360b7" gracePeriod=25
Nov 03 01:39:58.303913 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.303885    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg" podUID="0ec0e8c5-9468-4573-a617-f91b24b36d1a" containerName="payload-processing" containerID="cri-o://94a6e44ecd98f848a2d9d9e4eaaa74c0cc701fa44e01acf771604293bfead77f" gracePeriod=25
Nov 03 01:39:58.691422 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.675770    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:39:58.691422 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.675837    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:39:58.713633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.713599    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:58.720163 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.720134    5508 generic.go:356] "Generic (PLEG): container finished" podID="63c0eb6d-2862-4fca-8d0f-a1cca074a892" containerID="dbf078c877bac15827a95997b71a25c96d3ed3bf004c451953062be4cfba81fc" exitCode=0
Nov 03 01:39:58.720440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.720417    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns" event={"ID":"63c0eb6d-2862-4fca-8d0f-a1cca074a892","Type":"ContainerDied","Data":"dbf078c877bac15827a95997b71a25c96d3ed3bf004c451953062be4cfba81fc"}
Nov 03 01:39:58.722318 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.722294    5508 generic.go:356] "Generic (PLEG): container finished" podID="88d0256b-82df-4484-b5a7-e077303869ac" containerID="5b63cb24a38890a63866d786de9331e8927e0bf7acc9931aeafa29d0464360b7" exitCode=0
Nov 03 01:39:58.722577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.722430    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x" event={"ID":"88d0256b-82df-4484-b5a7-e077303869ac","Type":"ContainerDied","Data":"5b63cb24a38890a63866d786de9331e8927e0bf7acc9931aeafa29d0464360b7"}
Nov 03 01:39:58.724558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.724536    5508 generic.go:356] "Generic (PLEG): container finished" podID="0ec0e8c5-9468-4573-a617-f91b24b36d1a" containerID="94a6e44ecd98f848a2d9d9e4eaaa74c0cc701fa44e01acf771604293bfead77f" exitCode=0
Nov 03 01:39:58.725448 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.725426    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg" event={"ID":"0ec0e8c5-9468-4573-a617-f91b24b36d1a","Type":"ContainerDied","Data":"94a6e44ecd98f848a2d9d9e4eaaa74c0cc701fa44e01acf771604293bfead77f"}
Nov 03 01:39:58.725448 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.725444    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"
Nov 03 01:39:58.725619 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.725470    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg" event={"ID":"0ec0e8c5-9468-4573-a617-f91b24b36d1a","Type":"ContainerDied","Data":"3fe933fc898b02b07e54eb138cd7e8bfab336c07dee3beb74dec5b4afc9e7952"}
Nov 03 01:39:58.725619 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.725492    5508 scope.go:122] "RemoveContainer" containerID="94a6e44ecd98f848a2d9d9e4eaaa74c0cc701fa44e01acf771604293bfead77f"
Nov 03 01:39:58.733961 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.733924    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:39:58.737766 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.737748    5508 scope.go:122] "RemoveContainer" containerID="94a6e44ecd98f848a2d9d9e4eaaa74c0cc701fa44e01acf771604293bfead77f"
Nov 03 01:39:58.738114 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:39:58.738046    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a6e44ecd98f848a2d9d9e4eaaa74c0cc701fa44e01acf771604293bfead77f\": container with ID starting with 94a6e44ecd98f848a2d9d9e4eaaa74c0cc701fa44e01acf771604293bfead77f not found: ID does not exist" containerID="94a6e44ecd98f848a2d9d9e4eaaa74c0cc701fa44e01acf771604293bfead77f"
Nov 03 01:39:58.738114 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.738088    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a6e44ecd98f848a2d9d9e4eaaa74c0cc701fa44e01acf771604293bfead77f"} err="failed to get container status \"94a6e44ecd98f848a2d9d9e4eaaa74c0cc701fa44e01acf771604293bfead77f\": rpc error: code = NotFound desc = could not find container \"94a6e44ecd98f848a2d9d9e4eaaa74c0cc701fa44e01acf771604293bfead77f\": container with ID starting with 94a6e44ecd98f848a2d9d9e4eaaa74c0cc701fa44e01acf771604293bfead77f not found: ID does not exist"
Nov 03 01:39:58.741119 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.740711    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-plugins-config-volume\") pod \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") "
Nov 03 01:39:58.741119 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.740766    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-kube-api-access-87m6x\" (UniqueName: \"kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-kube-api-access-87m6x\") pod \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") "
Nov 03 01:39:58.741119 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.740854    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-routing-overlay-volume\") pod \"88d0256b-82df-4484-b5a7-e077303869ac\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") "
Nov 03 01:39:58.741119 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.740929    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-provider-credentials\") pod \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") "
Nov 03 01:39:58.741119 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.740993    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-kube-api-access-8ld65\" (UniqueName: \"kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-kube-api-access-8ld65\") pod \"88d0256b-82df-4484-b5a7-e077303869ac\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") "
Nov 03 01:39:58.741471 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.741405    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-plugins-config-volume" pod "0ec0e8c5-9468-4573-a617-f91b24b36d1a" (UID: "0ec0e8c5-9468-4573-a617-f91b24b36d1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:58.741471 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.741437    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-provider-credentials\") pod \"88d0256b-82df-4484-b5a7-e077303869ac\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") "
Nov 03 01:39:58.741624 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.741580    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-routing-overlay-volume\") pod \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\" (UID: \"0ec0e8c5-9468-4573-a617-f91b24b36d1a\") "
Nov 03 01:39:58.742822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.741681    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-plugins-config-volume\") pod \"88d0256b-82df-4484-b5a7-e077303869ac\" (UID: \"88d0256b-82df-4484-b5a7-e077303869ac\") "
Nov 03 01:39:58.742822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.741851    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-routing-overlay-volume" pod "88d0256b-82df-4484-b5a7-e077303869ac" (UID: "88d0256b-82df-4484-b5a7-e077303869ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:58.742822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.742071    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-routing-overlay-volume" pod "0ec0e8c5-9468-4573-a617-f91b24b36d1a" (UID: "0ec0e8c5-9468-4573-a617-f91b24b36d1a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:58.742822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.742203    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-plugins-config-volume" pod "88d0256b-82df-4484-b5a7-e077303869ac" (UID: "88d0256b-82df-4484-b5a7-e077303869ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:58.742822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.742508    5508 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-routing-overlay-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:58.742822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.742530    5508 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-routing-overlay-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:58.742822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.742545    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d0256b-82df-4484-b5a7-e077303869ac-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:58.742822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.742571    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec0e8c5-9468-4573-a617-f91b24b36d1a-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:58.744539 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.744516    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-provider-credentials" pod "0ec0e8c5-9468-4573-a617-f91b24b36d1a" (UID: "0ec0e8c5-9468-4573-a617-f91b24b36d1a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:58.744711 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.744687    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-provider-credentials" pod "88d0256b-82df-4484-b5a7-e077303869ac" (UID: "88d0256b-82df-4484-b5a7-e077303869ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:58.744781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.744745    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-kube-api-access-87m6x" pod "0ec0e8c5-9468-4573-a617-f91b24b36d1a" (UID: "0ec0e8c5-9468-4573-a617-f91b24b36d1a"). InnerVolumeSpecName "kube-api-access-87m6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:58.744825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.744772    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-kube-api-access-8ld65" pod "88d0256b-82df-4484-b5a7-e077303869ac" (UID: "88d0256b-82df-4484-b5a7-e077303869ac"). InnerVolumeSpecName "kube-api-access-8ld65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:58.843744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.843657    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87m6x\" (UniqueName: \"kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-kube-api-access-87m6x\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:58.843744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.843692    5508 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec0e8c5-9468-4573-a617-f91b24b36d1a-provider-credentials\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:58.843744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.843708    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ld65\" (UniqueName: \"kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-kube-api-access-8ld65\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:58.843744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:58.843725    5508 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88d0256b-82df-4484-b5a7-e077303869ac-provider-credentials\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:39:59.237653 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.237605    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"]
Nov 03 01:39:59.241756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.241722    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-88bf9945c-bx7lg"]
Nov 03 01:39:59.666373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.665727    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:39:59.666373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.665792    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:39:59.733049 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.733019    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"
Nov 03 01:39:59.733557 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.733055    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-694f985c7-bnq2x" event={"ID":"88d0256b-82df-4484-b5a7-e077303869ac","Type":"ContainerDied","Data":"4edda2fe8f41c8dcbd0343f022d42e15a2e1439fab99490e1a38e4e444b6762e"}
Nov 03 01:39:59.733557 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.733112    5508 scope.go:122] "RemoveContainer" containerID="5b63cb24a38890a63866d786de9331e8927e0bf7acc9931aeafa29d0464360b7"
Nov 03 01:39:59.898087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.898019    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"]
Nov 03 01:39:59.904709 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.904641    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-694f985c7-bnq2x"]
Nov 03 01:39:59.945426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.945404    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"
Nov 03 01:39:59.984806 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.984764    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c0eb6d-2862-4fca-8d0f-a1cca074a892-kube-api-access-8xl9s\" (UniqueName: \"kubernetes.io/projected/63c0eb6d-2862-4fca-8d0f-a1cca074a892-kube-api-access-8xl9s\") pod \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\" (UID: \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\") "
Nov 03 01:39:59.984975 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.984850    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-xks-ca\") pod \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\" (UID: \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\") "
Nov 03 01:39:59.985061 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.985050    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-openshift-ca\") pod \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\" (UID: \"63c0eb6d-2862-4fca-8d0f-a1cca074a892\") "
Nov 03 01:39:59.985904 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.985878    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-openshift-ca" pod "63c0eb6d-2862-4fca-8d0f-a1cca074a892" (UID: "63c0eb6d-2862-4fca-8d0f-a1cca074a892"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:59.987457 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.987433    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-xks-ca" pod "63c0eb6d-2862-4fca-8d0f-a1cca074a892" (UID: "63c0eb6d-2862-4fca-8d0f-a1cca074a892"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:59.987640 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:39:59.987608    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c0eb6d-2862-4fca-8d0f-a1cca074a892-kube-api-access-8xl9s" pod "63c0eb6d-2862-4fca-8d0f-a1cca074a892" (UID: "63c0eb6d-2862-4fca-8d0f-a1cca074a892"). InnerVolumeSpecName "kube-api-access-8xl9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:00.087372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.087280    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xl9s\" (UniqueName: \"kubernetes.io/projected/63c0eb6d-2862-4fca-8d0f-a1cca074a892-kube-api-access-8xl9s\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:00.087372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.087315    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:00.087372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.087331    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c0eb6d-2862-4fca-8d0f-a1cca074a892-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:00.667374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.665750    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:40:00.667374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.665813    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:40:00.699175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.699129    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec0e8c5-9468-4573-a617-f91b24b36d1a" path="/var/lib/kubelet/pods/0ec0e8c5-9468-4573-a617-f91b24b36d1a/volumes"
Nov 03 01:40:00.699841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.699816    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d0256b-82df-4484-b5a7-e077303869ac" path="/var/lib/kubelet/pods/88d0256b-82df-4484-b5a7-e077303869ac/volumes"
Nov 03 01:40:00.742985 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.742958    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"
Nov 03 01:40:00.743431 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.742984    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns" event={"ID":"63c0eb6d-2862-4fca-8d0f-a1cca074a892","Type":"ContainerDied","Data":"95a39fc29ad5c5d8a07daeb9e870369c6200664c5487ec1d937d54ccaab6b351"}
Nov 03 01:40:00.743431 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.743036    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95a39fc29ad5c5d8a07daeb9e870369c6200664c5487ec1d937d54ccaab6b351"
Nov 03 01:40:00.870371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.868394    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"]
Nov 03 01:40:00.870371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.869204    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0ec0e8c5-9468-4573-a617-f91b24b36d1a" containerName="payload-processing"
Nov 03 01:40:00.870371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.869224    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec0e8c5-9468-4573-a617-f91b24b36d1a" containerName="payload-processing"
Nov 03 01:40:00.870371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.869278    5508 cpu_manager.go:414] "containerMap: removing container" podUID="88d0256b-82df-4484-b5a7-e077303869ac" containerName="payload-processing"
Nov 03 01:40:00.870371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.869287    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d0256b-82df-4484-b5a7-e077303869ac" containerName="payload-processing"
Nov 03 01:40:00.870371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.869298    5508 cpu_manager.go:414] "containerMap: removing container" podUID="63c0eb6d-2862-4fca-8d0f-a1cca074a892" containerName="revoke-keys"
Nov 03 01:40:00.870371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.869305    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c0eb6d-2862-4fca-8d0f-a1cca074a892" containerName="revoke-keys"
Nov 03 01:40:00.870371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.869383    5508 cpu_manager.go:414] "containerMap: removing container" podUID="825d4108-3d57-4c2c-8831-ceb66e8e4ad8" containerName="revoke-keys"
Nov 03 01:40:00.870371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.869391    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825d4108-3d57-4c2c-8831-ceb66e8e4ad8" containerName="revoke-keys"
Nov 03 01:40:00.870371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.869533    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec0e8c5-9468-4573-a617-f91b24b36d1a" containerName="payload-processing"
Nov 03 01:40:00.870371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.869551    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d0256b-82df-4484-b5a7-e077303869ac" containerName="payload-processing"
Nov 03 01:40:00.870371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.869560    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c0eb6d-2862-4fca-8d0f-a1cca074a892" containerName="revoke-keys"
Nov 03 01:40:00.870371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.869572    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="825d4108-3d57-4c2c-8831-ceb66e8e4ad8" containerName="revoke-keys"
Nov 03 01:40:00.885727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.885691    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"
Nov 03 01:40:00.916110 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.916080    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"]
Nov 03 01:40:00.935226 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.935116    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:40:00.935226 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.935123    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:40:00.935226 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.935215    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:40:00.935519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:00.935509    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:40:01.001994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.001961    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfrfm\" (UniqueName: \"kubernetes.io/projected/df935e9d-c812-4ac5-b899-c575f946827c-kube-api-access-vfrfm\") pod \"maas-api-revoke-sub-8740514f-mnp9q\" (UID: \"df935e9d-c812-4ac5-b899-c575f946827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"
Nov 03 01:40:01.002195 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.002007    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8740514f-mnp9q\" (UID: \"df935e9d-c812-4ac5-b899-c575f946827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"
Nov 03 01:40:01.002195 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.002040    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8740514f-mnp9q\" (UID: \"df935e9d-c812-4ac5-b899-c575f946827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"
Nov 03 01:40:01.103619 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.103581    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfrfm\" (UniqueName: \"kubernetes.io/projected/df935e9d-c812-4ac5-b899-c575f946827c-kube-api-access-vfrfm\") pod \"maas-api-revoke-sub-8740514f-mnp9q\" (UID: \"df935e9d-c812-4ac5-b899-c575f946827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"
Nov 03 01:40:01.103795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.103627    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8740514f-mnp9q\" (UID: \"df935e9d-c812-4ac5-b899-c575f946827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"
Nov 03 01:40:01.103795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.103758    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8740514f-mnp9q\" (UID: \"df935e9d-c812-4ac5-b899-c575f946827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"
Nov 03 01:40:01.104536 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.104507    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8740514f-mnp9q\" (UID: \"df935e9d-c812-4ac5-b899-c575f946827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"
Nov 03 01:40:01.106396 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.106371    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8740514f-mnp9q\" (UID: \"df935e9d-c812-4ac5-b899-c575f946827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"
Nov 03 01:40:01.175923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.175861    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfrfm\" (UniqueName: \"kubernetes.io/projected/df935e9d-c812-4ac5-b899-c575f946827c-kube-api-access-vfrfm\") pod \"maas-api-revoke-sub-8740514f-mnp9q\" (UID: \"df935e9d-c812-4ac5-b899-c575f946827c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"
Nov 03 01:40:01.200975 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.200895    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"
Nov 03 01:40:01.621958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.621878    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"]
Nov 03 01:40:01.630750 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:40:01.630720    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf935e9d_c812_4ac5_b899_c575f946827c.slice/crio-362dc7ea0b08a7d78a388c081dea63f5e3ffb927cf7047a523620e55af1921a4 WatchSource:0}: Error finding container 362dc7ea0b08a7d78a388c081dea63f5e3ffb927cf7047a523620e55af1921a4: Status 404 returned error can't find the container with id 362dc7ea0b08a7d78a388c081dea63f5e3ffb927cf7047a523620e55af1921a4
Nov 03 01:40:01.671928 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.671799    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:40:01.671928 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.671858    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:40:01.804369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:01.803593    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q" event={"ID":"df935e9d-c812-4ac5-b899-c575f946827c","Type":"ContainerStarted","Data":"362dc7ea0b08a7d78a388c081dea63f5e3ffb927cf7047a523620e55af1921a4"}
Nov 03 01:40:02.670384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:02.669731    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:40:02.670384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:02.669800    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:40:02.865407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:02.863709    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q" event={"ID":"df935e9d-c812-4ac5-b899-c575f946827c","Type":"ContainerStarted","Data":"47f60cabfa207eabfe30a6ae02ad120e0aef92bda9a8f6fa0f516febe34ad6e9"}
Nov 03 01:40:03.263430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:03.262446    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:03.619673 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:03.619561    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:03.788381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:03.663828    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:40:03.788731 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:03.788690    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:40:03.959374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:03.954740    5508 generic.go:356] "Generic (PLEG): container finished" podID="df935e9d-c812-4ac5-b899-c575f946827c" containerID="47f60cabfa207eabfe30a6ae02ad120e0aef92bda9a8f6fa0f516febe34ad6e9" exitCode=0
Nov 03 01:40:03.959374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:03.954893    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q" event={"ID":"df935e9d-c812-4ac5-b899-c575f946827c","Type":"ContainerDied","Data":"47f60cabfa207eabfe30a6ae02ad120e0aef92bda9a8f6fa0f516febe34ad6e9"}
Nov 03 01:40:03.959941 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:03.959634    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:04.139517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:04.139415    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"]
Nov 03 01:40:04.144803 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:04.143706    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:04.161588 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:04.161546    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ccfebe6-2j7bs"]
Nov 03 01:40:04.280440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:04.277453    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:04.280440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:04.278244    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:40:04.321459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:04.321408    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:04.719686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:04.719562    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:40:04.733485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:04.733443    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:40:04.819939 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:04.819508    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:05.011434 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:04.970545    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:05.074432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.038998    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7283494-829e-41e5-a67d-99bee0c7271e" path="/var/lib/kubelet/pods/a7283494-829e-41e5-a67d-99bee0c7271e/volumes"
Nov 03 01:40:05.074432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.039500    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:40:05.074432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.039552    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:40:05.208451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.197456    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:05.237500 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.237461    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:40:05.309446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.279453    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:05.309446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.246460    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:40:05.309446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.280291    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" podUID="53333dac-248a-4cf1-ac30-c1962d83d8b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:05.824438 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.657692    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 01:40:05.824438 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.658817    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:40:05.824438 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.671288    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:40:05.824438 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.671386    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:05.849910 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.836658    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:40:05.849910 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:05.836731    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:40:06.055500 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:06.046941    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:06.090554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:06.090438    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 03 01:40:06.118375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:06.117183    5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Nov 03 01:40:06.247376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:06.210508    5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:06.273643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:06.203592    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:06.294966 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:06.294922    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:06.718374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:06.648470    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:40:06.718374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:06.648566    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:06.718374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:06.663632    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 01:40:06.919374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:06.870176    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 01:40:06.919374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:06.870247    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 01:40:07.149435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:07.138641    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:07.178693 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:07.171590    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 03 01:40:07.273528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:07.273482    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:07.306624 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:07.306569    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:07.701507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:07.676635    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:08.042435 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:40:08.038412    5508 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.117s"
Nov 03 01:40:08.154711 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:08.146247    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:08.255399 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:08.198637    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:08.316560 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:08.316456    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 01:40:08.327404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:08.316802    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:08.702942 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:08.702895    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:08.765446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:08.763805    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:40:08.765446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:08.763865    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:08.770318 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:08.768544    5508 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6" podUID="802af463-0d24-4f35-99f8-1dd96b6ed704" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.27:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:09.085433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.035868    5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Nov 03 01:40:09.498620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.237707    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:09.525380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.319468    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 01:40:09.525380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.319527    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:09.525380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.377903    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:09.623918 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.598263    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"]
Nov 03 01:40:09.623918 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.615662    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.671827 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.671794    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"]
Nov 03 01:40:09.715977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.714043    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:09.715977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.715532    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.715977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.715612    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.715977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.715684    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.715977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.715755    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.715977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.715803    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.715977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.715866    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.715977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.715895    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5zdb\" (UniqueName: \"kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-kube-api-access-n5zdb\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.715977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.715944    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.716573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.716016    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.746177 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.745720    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"
Nov 03 01:40:09.852697 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.851568    5508 patch_prober.go:29] interesting pod/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:40:09.852697 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.851638    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:09.872003 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.871969    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df935e9d-c812-4ac5-b899-c575f946827c-kube-api-access-vfrfm\" (UniqueName: \"kubernetes.io/projected/df935e9d-c812-4ac5-b899-c575f946827c-kube-api-access-vfrfm\") pod \"df935e9d-c812-4ac5-b899-c575f946827c\" (UID: \"df935e9d-c812-4ac5-b899-c575f946827c\") "
Nov 03 01:40:09.872384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.872327    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-xks-ca\") pod \"df935e9d-c812-4ac5-b899-c575f946827c\" (UID: \"df935e9d-c812-4ac5-b899-c575f946827c\") "
Nov 03 01:40:09.872650 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.872632    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-openshift-ca\") pod \"df935e9d-c812-4ac5-b899-c575f946827c\" (UID: \"df935e9d-c812-4ac5-b899-c575f946827c\") "
Nov 03 01:40:09.873342 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.873302    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.873555 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.873513    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5zdb\" (UniqueName: \"kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-kube-api-access-n5zdb\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.911375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.874813    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-openshift-ca" pod "df935e9d-c812-4ac5-b899-c575f946827c" (UID: "df935e9d-c812-4ac5-b899-c575f946827c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:09.911751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.891579    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df935e9d-c812-4ac5-b899-c575f946827c-kube-api-access-vfrfm" pod "df935e9d-c812-4ac5-b899-c575f946827c" (UID: "df935e9d-c812-4ac5-b899-c575f946827c"). InnerVolumeSpecName "kube-api-access-vfrfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:09.911879 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.875877    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.912554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.912522    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q" event={"ID":"df935e9d-c812-4ac5-b899-c575f946827c","Type":"ContainerDied","Data":"362dc7ea0b08a7d78a388c081dea63f5e3ffb927cf7047a523620e55af1921a4"}
Nov 03 01:40:09.912755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.912730    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="362dc7ea0b08a7d78a388c081dea63f5e3ffb927cf7047a523620e55af1921a4"
Nov 03 01:40:09.913000 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.912978    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"
Nov 03 01:40:09.913725 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.913707    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.914657 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.914229    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.914915 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.914896    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.915752 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.915380    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.916008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.915987    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.923643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.923602    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.924191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.924156    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.925049 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.924782    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.930863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.930835    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.936418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.936336    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.937251 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.937231    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:09.941722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.941684    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfrfm\" (UniqueName: \"kubernetes.io/projected/df935e9d-c812-4ac5-b899-c575f946827c-kube-api-access-vfrfm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:09.942066 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.926364    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:09.960564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:09.960520    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-xks-ca" pod "df935e9d-c812-4ac5-b899-c575f946827c" (UID: "df935e9d-c812-4ac5-b899-c575f946827c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:10.002200 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.002109    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5zdb\" (UniqueName: \"kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-kube-api-access-n5zdb\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:10.021215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.021173    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:10.043726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.043692    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df935e9d-c812-4ac5-b899-c575f946827c-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:10.138255 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.138221    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:10.187399 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.174429    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:10.245557 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.245517    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:10.358400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.346090    5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:10.504370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.503822    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 03 01:40:10.538504 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.529843    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:10.538504 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.537327    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 01:40:10.590426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.588866    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:40:10.591851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.591829    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:40:10.597992 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.588113    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:10.892315 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:10.892276    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:11.322708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.322676    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:40:11.323280 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.323259    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:40:11.323468 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.323455    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"]
Nov 03 01:40:11.325005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.324983    5508 cpu_manager.go:414] "containerMap: removing container" podUID="df935e9d-c812-4ac5-b899-c575f946827c" containerName="revoke-keys"
Nov 03 01:40:11.325175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.325157    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df935e9d-c812-4ac5-b899-c575f946827c" containerName="revoke-keys"
Nov 03 01:40:11.325674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.325654    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="df935e9d-c812-4ac5-b899-c575f946827c" containerName="revoke-keys"
Nov 03 01:40:11.405222 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.405180    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:40:11.405446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.405232    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"]
Nov 03 01:40:11.405446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.405380    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:40:11.405446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.405442    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:40:11.405635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.405519    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:40:11.407226 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.407200    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.508585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.489136    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2b9f22c1-openshift-default-dockercfg-p5xl7\""
Nov 03 01:40:11.508585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.490681    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.508585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.490732    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.508585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.490884    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.508585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.490925    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.508585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.490950    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08801e9f-1be9-4deb-931a-508623c37c7a-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.508585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.491096    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqk99\" (UniqueName: \"kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-kube-api-access-lqk99\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.508585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.491170    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.508585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.491294    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.508585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.491325    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08801e9f-1be9-4deb-931a-508623c37c7a-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.534468 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.534438    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"]
Nov 03 01:40:11.572832 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.572718    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:40:11.573118 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.573092    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:40:11.600307 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.576437    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:40:11.609392 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.598517    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.610160 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.610128    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.610468 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.610445    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.610673 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.610655    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.610821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.610788    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08801e9f-1be9-4deb-931a-508623c37c7a-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.611058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.611026    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqk99\" (UniqueName: \"kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-kube-api-access-lqk99\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.611242 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.611227    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.611489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.611453    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.611694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.611655    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08801e9f-1be9-4deb-931a-508623c37c7a-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.613715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.613687    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.613848 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.598987    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.614012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.613988    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.616287 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.616095    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08801e9f-1be9-4deb-931a-508623c37c7a-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.617767 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.616585    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.617767 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.617509    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08801e9f-1be9-4deb-931a-508623c37c7a-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.646411 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.643742    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.735370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.725454    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.747381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.746723    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqk99\" (UniqueName: \"kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-kube-api-access-lqk99\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:11.832677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:11.817075    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:12.429502 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:12.429405    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb" event={"ID":"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2","Type":"ContainerStarted","Data":"48fbcb4305404989e4b5b6da794443b9f9c31a4cb3841a48851187095cea6614"}
Nov 03 01:40:12.429502 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:12.429476    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb" event={"ID":"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2","Type":"ContainerStarted","Data":"f6cd53f87c2458c32acb0c9382d49fb20e9290fe82e8c8fe06e03752762a253f"}
Nov 03 01:40:12.523927 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:40:12.523883    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08801e9f_1be9_4deb_931a_508623c37c7a.slice/crio-8d5ff2d7f3bdaad54811d0254ae66e6391452d9fb14d135d566ca6a6ad29738f WatchSource:0}: Error finding container 8d5ff2d7f3bdaad54811d0254ae66e6391452d9fb14d135d566ca6a6ad29738f: Status 404 returned error can't find the container with id 8d5ff2d7f3bdaad54811d0254ae66e6391452d9fb14d135d566ca6a6ad29738f
Nov 03 01:40:12.524330 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:12.523919    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"]
Nov 03 01:40:12.526785 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:12.526748    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:40:12.526912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:12.526832    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:40:12.526912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:12.526879    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:40:12.575295 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:12.574185    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb" podStartSLOduration=4.574159987 podStartE2EDuration="4.574159987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:12.546598921 +0000 UTC m=+1291.972205386" watchObservedRunningTime="2026-11-03 01:40:12.574159987 +0000 UTC m=+1291.999766453"
Nov 03 01:40:13.246518 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:13.246479    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:13.247907 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:13.247883    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 01:40:13.247987 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:13.247932    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb" podUID="c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 01:40:13.438420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:13.438377    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" event={"ID":"08801e9f-1be9-4deb-931a-508623c37c7a","Type":"ContainerStarted","Data":"e84add4aad2b8a95c39f4b1cdc4d9675e88f46d5097b10416da0cc6a4361aa88"}
Nov 03 01:40:13.438594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:13.438437    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" event={"ID":"08801e9f-1be9-4deb-931a-508623c37c7a","Type":"ContainerStarted","Data":"8d5ff2d7f3bdaad54811d0254ae66e6391452d9fb14d135d566ca6a6ad29738f"}
Nov 03 01:40:13.570433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:13.557081    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" podStartSLOduration=3.557050362 podStartE2EDuration="3.557050362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:13.556531297 +0000 UTC m=+1292.982137764" watchObservedRunningTime="2026-11-03 01:40:13.557050362 +0000 UTC m=+1292.982656827"
Nov 03 01:40:13.820549 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:13.820461    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:13.834047 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:13.834001    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:13.868834 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:40:13.868784    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:40:14.247021 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:14.246988    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 01:40:14.247243 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:14.247044    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb" podUID="c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 01:40:14.446254 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:14.446225    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:14.447174 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:14.447154    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:40:15.246513 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:15.246484    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 01:40:15.246877 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:15.246549    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb" podUID="c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 01:40:16.246567 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:16.246520    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 01:40:16.246967 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:16.246590    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb" podUID="c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 01:40:17.247055 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.247023    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 01:40:17.247455 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.247081    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb" podUID="c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 01:40:17.545448 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.545364    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"]
Nov 03 01:40:17.584065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.584020    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"
Nov 03 01:40:17.593111 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.592894    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6w64\" (UniqueName: \"kubernetes.io/projected/af6d4708-bf8d-4c36-955d-0f63b572b5af-kube-api-access-k6w64\") pod \"maas-api-revoke-sub-fdb86700-8p4d2\" (UID: \"af6d4708-bf8d-4c36-955d-0f63b572b5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"
Nov 03 01:40:17.593111 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.592975    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb86700-8p4d2\" (UID: \"af6d4708-bf8d-4c36-955d-0f63b572b5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"
Nov 03 01:40:17.593111 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.593010    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb86700-8p4d2\" (UID: \"af6d4708-bf8d-4c36-955d-0f63b572b5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"
Nov 03 01:40:17.603558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.603529    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"]
Nov 03 01:40:17.652985 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.648398    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:40:17.690418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.679555    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:40:17.691299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.691275    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:40:17.691429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.691397    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:40:17.693848 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.693826    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6w64\" (UniqueName: \"kubernetes.io/projected/af6d4708-bf8d-4c36-955d-0f63b572b5af-kube-api-access-k6w64\") pod \"maas-api-revoke-sub-fdb86700-8p4d2\" (UID: \"af6d4708-bf8d-4c36-955d-0f63b572b5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"
Nov 03 01:40:17.693950 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.693874    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb86700-8p4d2\" (UID: \"af6d4708-bf8d-4c36-955d-0f63b572b5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"
Nov 03 01:40:17.693950 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.693898    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb86700-8p4d2\" (UID: \"af6d4708-bf8d-4c36-955d-0f63b572b5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"
Nov 03 01:40:17.694706 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.694682    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb86700-8p4d2\" (UID: \"af6d4708-bf8d-4c36-955d-0f63b572b5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"
Nov 03 01:40:17.698860 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.698836    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb86700-8p4d2\" (UID: \"af6d4708-bf8d-4c36-955d-0f63b572b5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"
Nov 03 01:40:17.777037 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.776999    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6w64\" (UniqueName: \"kubernetes.io/projected/af6d4708-bf8d-4c36-955d-0f63b572b5af-kube-api-access-k6w64\") pod \"maas-api-revoke-sub-fdb86700-8p4d2\" (UID: \"af6d4708-bf8d-4c36-955d-0f63b572b5af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"
Nov 03 01:40:17.901559 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:17.901524    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"
Nov 03 01:40:18.246744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:18.246656    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 01:40:18.246744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:18.246720    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb" podUID="c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 01:40:18.302744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:18.302706    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"]
Nov 03 01:40:18.309493 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:40:18.309466    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6d4708_bf8d_4c36_955d_0f63b572b5af.slice/crio-7224f4b83bbcc6303ff168329fc7d1815395f51276a2854fabc17158ed5beb69 WatchSource:0}: Error finding container 7224f4b83bbcc6303ff168329fc7d1815395f51276a2854fabc17158ed5beb69: Status 404 returned error can't find the container with id 7224f4b83bbcc6303ff168329fc7d1815395f51276a2854fabc17158ed5beb69
Nov 03 01:40:18.499830 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:18.499741    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2" event={"ID":"af6d4708-bf8d-4c36-955d-0f63b572b5af","Type":"ContainerStarted","Data":"7224f4b83bbcc6303ff168329fc7d1815395f51276a2854fabc17158ed5beb69"}
Nov 03 01:40:19.521032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:19.520983    5508 generic.go:356] "Generic (PLEG): container finished" podID="af6d4708-bf8d-4c36-955d-0f63b572b5af" containerID="b2db03d474eeb41f8e23113c59ad748eb7e51a7c734cbb56ed086bf78652bef8" exitCode=0
Nov 03 01:40:19.521635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:19.521604    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2" event={"ID":"af6d4708-bf8d-4c36-955d-0f63b572b5af","Type":"ContainerDied","Data":"b2db03d474eeb41f8e23113c59ad748eb7e51a7c734cbb56ed086bf78652bef8"}
Nov 03 01:40:19.601949 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:19.601736    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:19.603563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:19.603528    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:19.604817 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:19.604794    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:40:20.309561 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.309518    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"]
Nov 03 01:40:20.334371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.314713    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"
Nov 03 01:40:20.334371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.320685    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"]
Nov 03 01:40:20.380402 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.367670    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2b9f22c1-dockercfg-xnk5r\""
Nov 03 01:40:20.380402 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.368336    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2b9f22c1\""
Nov 03 01:40:20.422418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.410211    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2qx9\" (UniqueName: \"kubernetes.io/projected/59e1dcd4-ac74-4991-a93e-546e3f5e1231-kube-api-access-d2qx9\") pod \"payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx\" (UID: \"59e1dcd4-ac74-4991-a93e-546e3f5e1231\") " pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"
Nov 03 01:40:20.422418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.410362    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e1dcd4-ac74-4991-a93e-546e3f5e1231-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx\" (UID: \"59e1dcd4-ac74-4991-a93e-546e3f5e1231\") " pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"
Nov 03 01:40:20.512216 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.512166    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2qx9\" (UniqueName: \"kubernetes.io/projected/59e1dcd4-ac74-4991-a93e-546e3f5e1231-kube-api-access-d2qx9\") pod \"payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx\" (UID: \"59e1dcd4-ac74-4991-a93e-546e3f5e1231\") " pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"
Nov 03 01:40:20.512438 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.512277    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e1dcd4-ac74-4991-a93e-546e3f5e1231-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx\" (UID: \"59e1dcd4-ac74-4991-a93e-546e3f5e1231\") " pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"
Nov 03 01:40:20.513106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.513085    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e1dcd4-ac74-4991-a93e-546e3f5e1231-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx\" (UID: \"59e1dcd4-ac74-4991-a93e-546e3f5e1231\") " pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"
Nov 03 01:40:20.570124 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.570050    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2qx9\" (UniqueName: \"kubernetes.io/projected/59e1dcd4-ac74-4991-a93e-546e3f5e1231-kube-api-access-d2qx9\") pod \"payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx\" (UID: \"59e1dcd4-ac74-4991-a93e-546e3f5e1231\") " pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"
Nov 03 01:40:20.702619 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.702595    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"
Nov 03 01:40:20.714809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.714778    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6d4708-bf8d-4c36-955d-0f63b572b5af-kube-api-access-k6w64\" (UniqueName: \"kubernetes.io/projected/af6d4708-bf8d-4c36-955d-0f63b572b5af-kube-api-access-k6w64\") pod \"af6d4708-bf8d-4c36-955d-0f63b572b5af\" (UID: \"af6d4708-bf8d-4c36-955d-0f63b572b5af\") "
Nov 03 01:40:20.714970 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.714955    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-xks-ca\") pod \"af6d4708-bf8d-4c36-955d-0f63b572b5af\" (UID: \"af6d4708-bf8d-4c36-955d-0f63b572b5af\") "
Nov 03 01:40:20.715064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.715030    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-openshift-ca\") pod \"af6d4708-bf8d-4c36-955d-0f63b572b5af\" (UID: \"af6d4708-bf8d-4c36-955d-0f63b572b5af\") "
Nov 03 01:40:20.715397 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.715374    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-openshift-ca" pod "af6d4708-bf8d-4c36-955d-0f63b572b5af" (UID: "af6d4708-bf8d-4c36-955d-0f63b572b5af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:20.715554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.715535    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:20.716866 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.716840    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6d4708-bf8d-4c36-955d-0f63b572b5af-kube-api-access-k6w64" pod "af6d4708-bf8d-4c36-955d-0f63b572b5af" (UID: "af6d4708-bf8d-4c36-955d-0f63b572b5af"). InnerVolumeSpecName "kube-api-access-k6w64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:20.716962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.716908    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-xks-ca" pod "af6d4708-bf8d-4c36-955d-0f63b572b5af" (UID: "af6d4708-bf8d-4c36-955d-0f63b572b5af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:20.728984 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.728962    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"
Nov 03 01:40:20.817538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.817501    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6w64\" (UniqueName: \"kubernetes.io/projected/af6d4708-bf8d-4c36-955d-0f63b572b5af-kube-api-access-k6w64\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:20.817538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:20.817538    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af6d4708-bf8d-4c36-955d-0f63b572b5af-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:40:21.103555 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:21.103520    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"]
Nov 03 01:40:21.108986 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:40:21.108962    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e1dcd4_ac74_4991_a93e_546e3f5e1231.slice/crio-3d5e782b524a66cf0cbf5a1d025850b939f46943b40c33c20826dff013d0cda6 WatchSource:0}: Error finding container 3d5e782b524a66cf0cbf5a1d025850b939f46943b40c33c20826dff013d0cda6: Status 404 returned error can't find the container with id 3d5e782b524a66cf0cbf5a1d025850b939f46943b40c33c20826dff013d0cda6
Nov 03 01:40:21.539859 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:21.539824    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"
Nov 03 01:40:21.539859 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:21.539829    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2" event={"ID":"af6d4708-bf8d-4c36-955d-0f63b572b5af","Type":"ContainerDied","Data":"7224f4b83bbcc6303ff168329fc7d1815395f51276a2854fabc17158ed5beb69"}
Nov 03 01:40:21.540107 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:21.539897    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7224f4b83bbcc6303ff168329fc7d1815395f51276a2854fabc17158ed5beb69"
Nov 03 01:40:21.541522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:21.541495    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx" event={"ID":"59e1dcd4-ac74-4991-a93e-546e3f5e1231","Type":"ContainerStarted","Data":"2d2ffa070f2daa4d1864aca1838bd37798fae8ba5daf676b2f8d9f4e98800682"}
Nov 03 01:40:21.541640 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:21.541545    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx" event={"ID":"59e1dcd4-ac74-4991-a93e-546e3f5e1231","Type":"ContainerStarted","Data":"3d5e782b524a66cf0cbf5a1d025850b939f46943b40c33c20826dff013d0cda6"}
Nov 03 01:40:21.541870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:21.541849    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"
Nov 03 01:40:21.657185 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:21.656909    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx" podStartSLOduration=1.6568899240000001 podStartE2EDuration="1.656889924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:21.650430082 +0000 UTC m=+1301.076036570" watchObservedRunningTime="2026-11-03 01:40:21.656889924 +0000 UTC m=+1301.082496388"
Nov 03 01:40:26.384511 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:26.384464    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:40:28.973508 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:28.973451    5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Nov 03 01:40:29.231571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:29.231451    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:29.306530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:29.306483    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:29.407600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:29.407550    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:29.668277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:29.668233    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:30.055543 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:30.055437    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Nov 03 01:40:30.091532 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:30.091485    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-qbvm6" podUID="802af463-0d24-4f35-99f8-1dd96b6ed704" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:30.264371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:30.263673    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:30.264673 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:30.264648    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:40:30.323563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:30.323450    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:30.324245 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:30.324221    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:40:30.361373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:30.350572    5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:30.427499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:30.427447    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:30.428084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:30.428062    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:40:30.729374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:30.705992    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:31.077136 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:31.077043    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-s9q4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:40:31.084780 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:31.084738    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:31.290400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:31.188283    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:40:31.290740 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:31.290715    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:40:31.299373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:31.276642    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 03 01:40:31.450374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:31.436280    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:31.742435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:31.736885    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:32.069983 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:32.068310    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:32.193374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:32.184812    5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:32.284483 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:32.284434    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:32.321541 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:32.321434    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:32.335767 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:32.335723    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 01:40:32.362002 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:32.361976    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 01:40:32.569516 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:32.569489    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"
Nov 03 01:40:33.078372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:33.069747    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:33.144481 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:33.102840    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:40:33.334252 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:33.334138    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:33.613548 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:33.613444    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:40:33.684371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:33.684132    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 01:40:41.020098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:41.020066    5508 scope.go:122] "RemoveContainer" containerID="ecad5c82051500b6dbacf5c6180e4617983eb6093bc4a1463063c49659ee35f8"
Nov 03 01:40:41.030558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:41.030527    5508 scope.go:122] "RemoveContainer" containerID="cd76470a1275cb93a2377fa3b440747c5859bc1271c2db09fe623f2c3b734a96"
Nov 03 01:40:41.040494 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:41.040472    5508 scope.go:122] "RemoveContainer" containerID="e41745219f24904bb3db7de8d615d8911ca02eb32af8715781c7a9cb0d4d3a15"
Nov 03 01:40:41.051691 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:41.051671    5508 scope.go:122] "RemoveContainer" containerID="5a6f7d6cb9e6793676b63c91f759026ddc20624ea610eaa0feffc9d7ed000f37"
Nov 03 01:40:41.060946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:41.060923    5508 scope.go:122] "RemoveContainer" containerID="39462141eb5bcec093c48a6a1468f45415f039004b421fc4f35ad4b24ddfeea3"
Nov 03 01:40:43.811842 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:40:43.811804    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:40:55.809580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:55.809498    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:55.988519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:40:55.988471    5508 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Nov 03 01:41:07.361091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:07.361039    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:41:13.814471 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:41:13.814414    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:41:15.487615 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:15.487566    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:41:15.488153 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:15.487641    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:41:16.993709 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:16.993672    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"]
Nov 03 01:41:16.994549 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:16.994528    5508 cpu_manager.go:414] "containerMap: removing container" podUID="af6d4708-bf8d-4c36-955d-0f63b572b5af" containerName="revoke-keys"
Nov 03 01:41:16.994622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:16.994551    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af6d4708-bf8d-4c36-955d-0f63b572b5af" containerName="revoke-keys"
Nov 03 01:41:16.994743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:16.994731    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="af6d4708-bf8d-4c36-955d-0f63b572b5af" containerName="revoke-keys"
Nov 03 01:41:16.998795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:16.998770    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"
Nov 03 01:41:17.046729 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.046689    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a13fd92-gfwn4\" (UID: \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"
Nov 03 01:41:17.046958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.046751    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nzwb\" (UniqueName: \"kubernetes.io/projected/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-kube-api-access-2nzwb\") pod \"maas-api-revoke-sub-9a13fd92-gfwn4\" (UID: \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"
Nov 03 01:41:17.046958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.046845    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a13fd92-gfwn4\" (UID: \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"
Nov 03 01:41:17.049746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.049693    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"]
Nov 03 01:41:17.090815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.090779    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:41:17.091062 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.090841    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:41:17.091232 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.090881    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:41:17.091387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.090949    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:41:17.147964 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.147926    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a13fd92-gfwn4\" (UID: \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"
Nov 03 01:41:17.148184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.147983    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nzwb\" (UniqueName: \"kubernetes.io/projected/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-kube-api-access-2nzwb\") pod \"maas-api-revoke-sub-9a13fd92-gfwn4\" (UID: \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"
Nov 03 01:41:17.148184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.148036    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a13fd92-gfwn4\" (UID: \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"
Nov 03 01:41:17.148889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.148864    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a13fd92-gfwn4\" (UID: \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"
Nov 03 01:41:17.151622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.151594    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a13fd92-gfwn4\" (UID: \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"
Nov 03 01:41:17.205728 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.205692    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nzwb\" (UniqueName: \"kubernetes.io/projected/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-kube-api-access-2nzwb\") pod \"maas-api-revoke-sub-9a13fd92-gfwn4\" (UID: \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"
Nov 03 01:41:17.313335 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.313200    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"
Nov 03 01:41:17.705896 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:17.705854    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"]
Nov 03 01:41:17.712202 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:41:17.712164    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53364c1e_cdcc_4f5f_9f62_434b6823b4ef.slice/crio-1c29fd3a1dc2d372ff66158786911ecbb637c9d8e057eef62ce2df49121dd483 WatchSource:0}: Error finding container 1c29fd3a1dc2d372ff66158786911ecbb637c9d8e057eef62ce2df49121dd483: Status 404 returned error can't find the container with id 1c29fd3a1dc2d372ff66158786911ecbb637c9d8e057eef62ce2df49121dd483
Nov 03 01:41:18.195141 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:18.195108    5508 generic.go:356] "Generic (PLEG): container finished" podID="08801e9f-1be9-4deb-931a-508623c37c7a" containerID="e84add4aad2b8a95c39f4b1cdc4d9675e88f46d5097b10416da0cc6a4361aa88" exitCode=0
Nov 03 01:41:18.195591 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:18.195191    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" event={"ID":"08801e9f-1be9-4deb-931a-508623c37c7a","Type":"ContainerDied","Data":"e84add4aad2b8a95c39f4b1cdc4d9675e88f46d5097b10416da0cc6a4361aa88"}
Nov 03 01:41:18.195761 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:18.195741    5508 scope.go:122] "RemoveContainer" containerID="e84add4aad2b8a95c39f4b1cdc4d9675e88f46d5097b10416da0cc6a4361aa88"
Nov 03 01:41:18.197003 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:18.196976    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4" event={"ID":"53364c1e-cdcc-4f5f-9f62-434b6823b4ef","Type":"ContainerStarted","Data":"8686b1361ed9cabb9e736730ecda84e3f6696a6a23915ab331edf5741967ad21"}
Nov 03 01:41:18.197097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:18.197024    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4" event={"ID":"53364c1e-cdcc-4f5f-9f62-434b6823b4ef","Type":"ContainerStarted","Data":"1c29fd3a1dc2d372ff66158786911ecbb637c9d8e057eef62ce2df49121dd483"}
Nov 03 01:41:18.197383 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:18.197332    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:41:18.206488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:18.197903    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:41:18.206488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:18.198163    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:41:18.490611 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:18.465134    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4" podStartSLOduration=2.465114942 podStartE2EDuration="2.465114942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:18.46262968 +0000 UTC m=+1357.888236145" watchObservedRunningTime="2026-11-03 01:41:18.465114942 +0000 UTC m=+1357.890721407"
Nov 03 01:41:19.090186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.089983    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"]
Nov 03 01:41:19.104054 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.104024    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"]
Nov 03 01:41:19.105775 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.104186    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.211074 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.211038    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" event={"ID":"08801e9f-1be9-4deb-931a-508623c37c7a","Type":"ContainerStarted","Data":"601e7028cc51463414bbf875d5369d43cb482e96064708209e3a3d73c73345ab"}
Nov 03 01:41:19.212924 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.212895    5508 generic.go:356] "Generic (PLEG): container finished" podID="53364c1e-cdcc-4f5f-9f62-434b6823b4ef" containerID="8686b1361ed9cabb9e736730ecda84e3f6696a6a23915ab331edf5741967ad21" exitCode=0
Nov 03 01:41:19.213073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.212974    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4" event={"ID":"53364c1e-cdcc-4f5f-9f62-434b6823b4ef","Type":"ContainerDied","Data":"8686b1361ed9cabb9e736730ecda84e3f6696a6a23915ab331edf5741967ad21"}
Nov 03 01:41:19.275375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.273164    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.275375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.273219    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.275375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.273305    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2nbd\" (UniqueName: \"kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-kube-api-access-d2nbd\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.275375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.273373    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.275375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.273397    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.275375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.273455    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.275375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.273482    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.275375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.273549    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895db3e8-7f0d-40fd-9572-66659bc5887e-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.275375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.273581    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.375406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.375253    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2nbd\" (UniqueName: \"kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-kube-api-access-d2nbd\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.375406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.375328    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.375406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.375369    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.375733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.375432    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.375733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.375459    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.375733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.375555    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895db3e8-7f0d-40fd-9572-66659bc5887e-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.375733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.375587    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.375954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.375844    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.375954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.375876    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.375954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.375934    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.376428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.375921    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.376872 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.376846    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.376971 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.376918    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.377585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.377560    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895db3e8-7f0d-40fd-9572-66659bc5887e-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.378897 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.378875    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.380342 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.380323    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.433332 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.433298    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.433497 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.433307    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2nbd\" (UniqueName: \"kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-kube-api-access-d2nbd\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.726215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.726178    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:19.818783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.818758    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:41:19.818905 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.818796    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 03 01:41:19.818905 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:19.818854    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 03 01:41:20.192301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.192271    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"]
Nov 03 01:41:20.197995 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:41:20.197964    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895db3e8_7f0d_40fd_9572_66659bc5887e.slice/crio-34af4f095287b57befec88241c0963ff70341dff1399f6335c5c4d09eaa29b00 WatchSource:0}: Error finding container 34af4f095287b57befec88241c0963ff70341dff1399f6335c5c4d09eaa29b00: Status 404 returned error can't find the container with id 34af4f095287b57befec88241c0963ff70341dff1399f6335c5c4d09eaa29b00
Nov 03 01:41:20.200675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.200642    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:41:20.200767 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.200710    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:41:20.200767 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.200747    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:41:20.224196 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.224167    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5" event={"ID":"895db3e8-7f0d-40fd-9572-66659bc5887e","Type":"ContainerStarted","Data":"34af4f095287b57befec88241c0963ff70341dff1399f6335c5c4d09eaa29b00"}
Nov 03 01:41:20.494390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.494204    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"
Nov 03 01:41:20.597835 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.597784    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-xks-ca\") pod \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\" (UID: \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\") "
Nov 03 01:41:20.598076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.598046    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-openshift-ca\") pod \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\" (UID: \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\") "
Nov 03 01:41:20.598148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.598100    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-kube-api-access-2nzwb\" (UniqueName: \"kubernetes.io/projected/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-kube-api-access-2nzwb\") pod \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\" (UID: \"53364c1e-cdcc-4f5f-9f62-434b6823b4ef\") "
Nov 03 01:41:20.598591 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.598563    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-openshift-ca" pod "53364c1e-cdcc-4f5f-9f62-434b6823b4ef" (UID: "53364c1e-cdcc-4f5f-9f62-434b6823b4ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:41:20.598783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.598765    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:20.607397 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.602529    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-kube-api-access-2nzwb" pod "53364c1e-cdcc-4f5f-9f62-434b6823b4ef" (UID: "53364c1e-cdcc-4f5f-9f62-434b6823b4ef"). InnerVolumeSpecName "kube-api-access-2nzwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:20.607397 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.602886    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-xks-ca" pod "53364c1e-cdcc-4f5f-9f62-434b6823b4ef" (UID: "53364c1e-cdcc-4f5f-9f62-434b6823b4ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:20.711053 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.710952    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:20.711053 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.710998    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nzwb\" (UniqueName: \"kubernetes.io/projected/53364c1e-cdcc-4f5f-9f62-434b6823b4ef-kube-api-access-2nzwb\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:41:20.818666 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.818622    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 03 01:41:20.818836 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:20.818672    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 03 01:41:21.237557 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:21.237516    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5" event={"ID":"895db3e8-7f0d-40fd-9572-66659bc5887e","Type":"ContainerStarted","Data":"d2010e4421c35147212188611d8eb5df1513bb770214fbe52fb94138d04aa9c2"}
Nov 03 01:41:21.240233 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:21.240205    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4" event={"ID":"53364c1e-cdcc-4f5f-9f62-434b6823b4ef","Type":"ContainerDied","Data":"1c29fd3a1dc2d372ff66158786911ecbb637c9d8e057eef62ce2df49121dd483"}
Nov 03 01:41:21.240396 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:21.240259    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c29fd3a1dc2d372ff66158786911ecbb637c9d8e057eef62ce2df49121dd483"
Nov 03 01:41:21.240396 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:21.240340    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"
Nov 03 01:41:21.628419 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:21.628296    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5" podStartSLOduration=3.628281448 podStartE2EDuration="3.628281448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:21.406462047 +0000 UTC m=+1360.832068540" watchObservedRunningTime="2026-11-03 01:41:21.628281448 +0000 UTC m=+1361.053887912"
Nov 03 01:41:21.726425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:21.726395    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:21.728226 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:21.728199    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 01:41:21.728391 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:21.728242    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5" podUID="895db3e8-7f0d-40fd-9572-66659bc5887e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 01:41:21.818883 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:21.818845    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 03 01:41:21.819056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:21.818899    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 03 01:41:22.726695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:22.726658    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 01:41:22.727169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:22.726719    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5" podUID="895db3e8-7f0d-40fd-9572-66659bc5887e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 01:41:22.818885 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:22.818848    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 03 01:41:22.819045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:22.818906    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 03 01:41:23.727629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:23.727598    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 01:41:23.728102 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:23.728078    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5" podUID="895db3e8-7f0d-40fd-9572-66659bc5887e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 01:41:23.818826 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:23.818794    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Nov 03 01:41:23.819010 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:23.818842    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Nov 03 01:41:24.726701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:24.726654    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 01:41:24.726897 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:24.726739    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5" podUID="895db3e8-7f0d-40fd-9572-66659bc5887e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 01:41:24.822437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:24.822407    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:41:24.822934 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:24.822914    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:41:24.823546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:24.823523    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:41:25.726729 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:25.726694    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 01:41:25.726935 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:25.726783    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5" podUID="895db3e8-7f0d-40fd-9572-66659bc5887e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 01:41:26.727070 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:26.727028    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 01:41:26.727510 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:26.727106    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5" podUID="895db3e8-7f0d-40fd-9572-66659bc5887e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 01:41:27.731478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:27.731449    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:27.731943 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:27.731922    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:27.732795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:27.732773    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:41:30.695649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.695614    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"]
Nov 03 01:41:30.696437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.696409    5508 cpu_manager.go:414] "containerMap: removing container" podUID="53364c1e-cdcc-4f5f-9f62-434b6823b4ef" containerName="revoke-keys"
Nov 03 01:41:30.696437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.696431    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53364c1e-cdcc-4f5f-9f62-434b6823b4ef" containerName="revoke-keys"
Nov 03 01:41:30.696649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.696572    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="53364c1e-cdcc-4f5f-9f62-434b6823b4ef" containerName="revoke-keys"
Nov 03 01:41:30.700447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.700425    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.760629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.760600    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-51bcf3-openshift-default-dockercfg-dvvhc\""
Nov 03 01:41:30.802119 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.802080    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"]
Nov 03 01:41:30.813974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.813941    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83390757-f37d-4198-b4ac-bc05f260fa82-istio-podinfo\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.814329 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.814305    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-credential-socket\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.814484 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.814454    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-envoy\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.814668 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.814651    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-socket\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.814827 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.814812    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-certs\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.814958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.814942    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-data\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.815164 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.815136    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83390757-f37d-4198-b4ac-bc05f260fa82-istiod-ca-cert\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.815219 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.815186    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-istio-token\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.815270 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.815218    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flhmd\" (UniqueName: \"kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-kube-api-access-flhmd\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.931536 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.931503    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flhmd\" (UniqueName: \"kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-kube-api-access-flhmd\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.931898 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.931877    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83390757-f37d-4198-b4ac-bc05f260fa82-istio-podinfo\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.932094 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.932079    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-credential-socket\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.932196 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.932182    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-envoy\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.932379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.932364    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-socket\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.932522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.932508    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-certs\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.932646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.932631    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-data\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.932805 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.932790    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83390757-f37d-4198-b4ac-bc05f260fa82-istiod-ca-cert\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.932913 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.932900    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-istio-token\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.937663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.937035    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83390757-f37d-4198-b4ac-bc05f260fa82-istio-podinfo\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.937663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.937368    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-credential-socket\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.954404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.954269    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-certs\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.956132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.955027    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-data\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.957026 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.957004    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83390757-f37d-4198-b4ac-bc05f260fa82-istiod-ca-cert\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:30.957477 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:30.957453    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-socket\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:31.010390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:31.000547    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-envoy\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:31.076371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:31.029653    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-istio-token\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:31.080419 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:31.077376    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flhmd\" (UniqueName: \"kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-kube-api-access-flhmd\") pod \"e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") " pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:31.313840 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:31.313750    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:31.765202 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:31.760281    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"]
Nov 03 01:41:31.774536 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:41:31.774484    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83390757_f37d_4198_b4ac_bc05f260fa82.slice/crio-eeb5b2167e05ba219ea238932d1bcec1a1a7598011995cac35d72e47ebedca2f WatchSource:0}: Error finding container eeb5b2167e05ba219ea238932d1bcec1a1a7598011995cac35d72e47ebedca2f: Status 404 returned error can't find the container with id eeb5b2167e05ba219ea238932d1bcec1a1a7598011995cac35d72e47ebedca2f
Nov 03 01:41:31.776852 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:31.776818    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:41:31.776941 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:31.776883    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:41:31.776941 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:31.776909    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:41:32.413899 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:32.413862    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" event={"ID":"83390757-f37d-4198-b4ac-bc05f260fa82","Type":"ContainerStarted","Data":"54e9673589e42ee976169781ee949a322cdda03c97a34c0236981a90569c255c"}
Nov 03 01:41:32.414094 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:32.413931    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" event={"ID":"83390757-f37d-4198-b4ac-bc05f260fa82","Type":"ContainerStarted","Data":"eeb5b2167e05ba219ea238932d1bcec1a1a7598011995cac35d72e47ebedca2f"}
Nov 03 01:41:32.575872 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:32.575820    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" podStartSLOduration=2.575804114 podStartE2EDuration="2.575804114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:32.572901147 +0000 UTC m=+1371.998507612" watchObservedRunningTime="2026-11-03 01:41:32.575804114 +0000 UTC m=+1372.001410577"
Nov 03 01:41:33.314246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:33.314208    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:33.319661 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:33.319637    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:33.421222 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:33.421186    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:33.422130 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:33.422114    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:41:35.604257 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:35.604222    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:41:39.069168 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.069137    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"]
Nov 03 01:41:39.073641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.073618    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"
Nov 03 01:41:39.095239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.095213    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85txs\" (UniqueName: \"kubernetes.io/projected/69203466-b06c-468f-850e-eec3a5cb0dfb-kube-api-access-85txs\") pod \"payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r\" (UID: \"69203466-b06c-468f-850e-eec3a5cb0dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"
Nov 03 01:41:39.095407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.095380    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69203466-b06c-468f-850e-eec3a5cb0dfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r\" (UID: \"69203466-b06c-468f-850e-eec3a5cb0dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"
Nov 03 01:41:39.128359 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.128317    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"]
Nov 03 01:41:39.177371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.171655    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-51bcf3\""
Nov 03 01:41:39.177371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.171848    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-51bcf3-dockercfg-dcnk6\""
Nov 03 01:41:39.232374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.219037    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85txs\" (UniqueName: \"kubernetes.io/projected/69203466-b06c-468f-850e-eec3a5cb0dfb-kube-api-access-85txs\") pod \"payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r\" (UID: \"69203466-b06c-468f-850e-eec3a5cb0dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"
Nov 03 01:41:39.232374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.219147    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69203466-b06c-468f-850e-eec3a5cb0dfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r\" (UID: \"69203466-b06c-468f-850e-eec3a5cb0dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"
Nov 03 01:41:39.232374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.220159    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69203466-b06c-468f-850e-eec3a5cb0dfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r\" (UID: \"69203466-b06c-468f-850e-eec3a5cb0dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"
Nov 03 01:41:39.428372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.422689    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85txs\" (UniqueName: \"kubernetes.io/projected/69203466-b06c-468f-850e-eec3a5cb0dfb-kube-api-access-85txs\") pod \"payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r\" (UID: \"69203466-b06c-468f-850e-eec3a5cb0dfb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"
Nov 03 01:41:39.428372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.423601    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"
Nov 03 01:41:39.484368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.469090    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"]
Nov 03 01:41:39.506134 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.506102    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"]
Nov 03 01:41:39.507945 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.506409    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"
Nov 03 01:41:39.581018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.580935    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6k64\" (UniqueName: \"kubernetes.io/projected/99a5b548-b10a-42f7-9194-b185d7408231-kube-api-access-z6k64\") pod \"payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p\" (UID: \"99a5b548-b10a-42f7-9194-b185d7408231\") " pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"
Nov 03 01:41:39.581018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.580990    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a5b548-b10a-42f7-9194-b185d7408231-plugins-config-volume\") pod \"payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p\" (UID: \"99a5b548-b10a-42f7-9194-b185d7408231\") " pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"
Nov 03 01:41:39.682469 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.682377    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6k64\" (UniqueName: \"kubernetes.io/projected/99a5b548-b10a-42f7-9194-b185d7408231-kube-api-access-z6k64\") pod \"payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p\" (UID: \"99a5b548-b10a-42f7-9194-b185d7408231\") " pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"
Nov 03 01:41:39.682614 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.682481    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a5b548-b10a-42f7-9194-b185d7408231-plugins-config-volume\") pod \"payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p\" (UID: \"99a5b548-b10a-42f7-9194-b185d7408231\") " pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"
Nov 03 01:41:39.683412 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.683388    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a5b548-b10a-42f7-9194-b185d7408231-plugins-config-volume\") pod \"payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p\" (UID: \"99a5b548-b10a-42f7-9194-b185d7408231\") " pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"
Nov 03 01:41:39.742429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.742400    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6k64\" (UniqueName: \"kubernetes.io/projected/99a5b548-b10a-42f7-9194-b185d7408231-kube-api-access-z6k64\") pod \"payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p\" (UID: \"99a5b548-b10a-42f7-9194-b185d7408231\") " pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"
Nov 03 01:41:39.744861 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.744838    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"]
Nov 03 01:41:39.749495 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.749474    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.802568 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.802542    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-41aa5f-openshift-default-dockercfg-g26tf\""
Nov 03 01:41:39.808462 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.808439    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"]
Nov 03 01:41:39.829077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.829051    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"
Nov 03 01:41:39.876371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.875044    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"]
Nov 03 01:41:39.885202 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.885180    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.885299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.885229    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.885371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.885341    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.885427 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.885392    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.885606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.885558    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.885656 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.885638    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.885698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.885689    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.885785 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.885772    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnnxq\" (UniqueName: \"kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-kube-api-access-bnnxq\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.885844 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.885793    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.986831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.986800    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.986960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.986866    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.986960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.986898    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.987041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.986963    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnnxq\" (UniqueName: \"kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-kube-api-access-bnnxq\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.987041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.986993    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.987137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.987104    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.987137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.987130    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.987216 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.987180    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.987216 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.987211    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.987281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.987254    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-certs\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.987606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.987567    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-data\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.987732 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.987620    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istiod-ca-cert\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.987732 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.987628    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.987931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.987858    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-credential-socket\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.989697 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.989675    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-envoy\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:39.989886 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:39.989865    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-podinfo\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:40.045279 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.045228    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-token\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:40.045419 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.045339    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnnxq\" (UniqueName: \"kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-kube-api-access-bnnxq\") pod \"e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") " pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:40.068677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.068649    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:40.218676 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.218608    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"]
Nov 03 01:41:40.223260 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:41:40.223234    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a5b548_b10a_42f7_9194_b185d7408231.slice/crio-88ddd2b774666a45cf6aa4eabc3ff47e561963ed3913ea1f9b7fc2f659adaa61 WatchSource:0}: Error finding container 88ddd2b774666a45cf6aa4eabc3ff47e561963ed3913ea1f9b7fc2f659adaa61: Status 404 returned error can't find the container with id 88ddd2b774666a45cf6aa4eabc3ff47e561963ed3913ea1f9b7fc2f659adaa61
Nov 03 01:41:40.454542 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.454510    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"]
Nov 03 01:41:40.457915 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:41:40.457885    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3fb1ba8_ef8c_4ef2_a78d_2f01d9ea0857.slice/crio-5b68817c1bfbac10dbdeecc6d11303ea2ce9b142d53bb1bab69f6d0557314f5d WatchSource:0}: Error finding container 5b68817c1bfbac10dbdeecc6d11303ea2ce9b142d53bb1bab69f6d0557314f5d: Status 404 returned error can't find the container with id 5b68817c1bfbac10dbdeecc6d11303ea2ce9b142d53bb1bab69f6d0557314f5d
Nov 03 01:41:40.460671 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.460639    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:41:40.460780 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.460712    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:41:40.460780 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.460759    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:41:40.514137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.514109    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p" event={"ID":"99a5b548-b10a-42f7-9194-b185d7408231","Type":"ContainerStarted","Data":"57bc060e184be6c18c3b1f42ef014981f9838d576344d121f0162d14c9632cb4"}
Nov 03 01:41:40.514277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.514167    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"
Nov 03 01:41:40.514277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.514185    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p" event={"ID":"99a5b548-b10a-42f7-9194-b185d7408231","Type":"ContainerStarted","Data":"88ddd2b774666a45cf6aa4eabc3ff47e561963ed3913ea1f9b7fc2f659adaa61"}
Nov 03 01:41:40.515461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.515436    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f" event={"ID":"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857","Type":"ContainerStarted","Data":"5b68817c1bfbac10dbdeecc6d11303ea2ce9b142d53bb1bab69f6d0557314f5d"}
Nov 03 01:41:40.516758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.516736    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r" event={"ID":"69203466-b06c-468f-850e-eec3a5cb0dfb","Type":"ContainerStarted","Data":"5555481c265536bcc012730a6f83ea6a9db5f4d223c2bf2412bb82a6ed216527"}
Nov 03 01:41:40.516852 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.516773    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r" event={"ID":"69203466-b06c-468f-850e-eec3a5cb0dfb","Type":"ContainerStarted","Data":"4698dba44592c6007e4f5d9cbb58b67e792b19748d39176341f91417eb67732f"}
Nov 03 01:41:40.516852 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.516791    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"
Nov 03 01:41:40.625614 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.625547    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p" podStartSLOduration=1.625528009 podStartE2EDuration="1.625528009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:40.622957831 +0000 UTC m=+1380.048564294" watchObservedRunningTime="2026-11-03 01:41:40.625528009 +0000 UTC m=+1380.051134474"
Nov 03 01:41:40.759468 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:40.754904    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r" podStartSLOduration=1.7548829910000001 podStartE2EDuration="1.754882991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:40.731491296 +0000 UTC m=+1380.157097780" watchObservedRunningTime="2026-11-03 01:41:40.754882991 +0000 UTC m=+1380.180489456"
Nov 03 01:41:41.120140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:41.120060    5508 scope.go:122] "RemoveContainer" containerID="3e725af250f317c38645e9d9deb5ff9495d390366210c5d7b8ed0faf5d040a76"
Nov 03 01:41:41.525822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:41.525786    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f" event={"ID":"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857","Type":"ContainerStarted","Data":"f753c15fa19c15434f0cc885f7250947c1fa3b125d55d6fe3f103c3f0f9c6e12"}
Nov 03 01:41:41.640681 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:41.640629    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f" podStartSLOduration=2.640615553 podStartE2EDuration="2.640615553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:41.638271285 +0000 UTC m=+1381.063877770" watchObservedRunningTime="2026-11-03 01:41:41.640615553 +0000 UTC m=+1381.066222036"
Nov 03 01:41:42.075400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:42.074740    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:42.079485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:42.079451    5508 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 01:41:42.079709 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:42.079663    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f" podUID="e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 01:41:43.072813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:43.072779    5508 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 01:41:43.073838 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:43.073428    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f" podUID="e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 01:41:43.813976 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:41:43.813937    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:41:44.075803 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:44.075713    5508 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 01:41:44.076448 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:44.076417    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f" podUID="e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 01:41:45.071919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:45.071463    5508 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 01:41:45.071919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:45.071520    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f" podUID="e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 01:41:46.073724 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:46.073686    5508 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 01:41:46.074196 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:46.073753    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f" podUID="e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 01:41:47.069301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:47.069263    5508 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 03 01:41:47.069476 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:47.069326    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f" podUID="e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 03 01:41:48.074225 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:48.074193    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:48.074698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:48.074680    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:48.075304 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:48.075288    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:41:51.529082 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:51.529047    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"
Nov 03 01:41:51.529563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:51.529109    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"
Nov 03 01:41:59.295894 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.295859    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"]
Nov 03 01:41:59.300092 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.300074    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"
Nov 03 01:41:59.396440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.396403    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:41:59.396667 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.396523    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:41:59.396667 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.396585    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:41:59.396667 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.396651    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:41:59.401153 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.401127    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"]
Nov 03 01:41:59.419401 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.419373    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e5a21f-qqmpq\" (UID: \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"
Nov 03 01:41:59.419563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.419492    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwv6h\" (UniqueName: \"kubernetes.io/projected/fb153706-24e6-4f66-bd5b-c45f1c25de1c-kube-api-access-wwv6h\") pod \"maas-api-revoke-sub-06e5a21f-qqmpq\" (UID: \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"
Nov 03 01:41:59.419625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.419565    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e5a21f-qqmpq\" (UID: \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"
Nov 03 01:41:59.520933 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.520892    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e5a21f-qqmpq\" (UID: \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"
Nov 03 01:41:59.521123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.521026    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwv6h\" (UniqueName: \"kubernetes.io/projected/fb153706-24e6-4f66-bd5b-c45f1c25de1c-kube-api-access-wwv6h\") pod \"maas-api-revoke-sub-06e5a21f-qqmpq\" (UID: \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"
Nov 03 01:41:59.521123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.521072    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e5a21f-qqmpq\" (UID: \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"
Nov 03 01:41:59.521618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.521592    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06e5a21f-qqmpq\" (UID: \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"
Nov 03 01:41:59.523626 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.523600    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06e5a21f-qqmpq\" (UID: \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"
Nov 03 01:41:59.579266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.579184    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwv6h\" (UniqueName: \"kubernetes.io/projected/fb153706-24e6-4f66-bd5b-c45f1c25de1c-kube-api-access-wwv6h\") pod \"maas-api-revoke-sub-06e5a21f-qqmpq\" (UID: \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"
Nov 03 01:41:59.614100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.614068    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"
Nov 03 01:41:59.997490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:41:59.997454    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"]
Nov 03 01:42:00.001929 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:42:00.001900    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb153706_24e6_4f66_bd5b_c45f1c25de1c.slice/crio-737d701a765725e76b5344dae2dc0d6aabc3b0307dd2c0208fb6e3eb298abf29 WatchSource:0}: Error finding container 737d701a765725e76b5344dae2dc0d6aabc3b0307dd2c0208fb6e3eb298abf29: Status 404 returned error can't find the container with id 737d701a765725e76b5344dae2dc0d6aabc3b0307dd2c0208fb6e3eb298abf29
Nov 03 01:42:00.727972 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:00.727938    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq" event={"ID":"fb153706-24e6-4f66-bd5b-c45f1c25de1c","Type":"ContainerStarted","Data":"d95369dca4cd8c680bc40497f0916ad4218de6fa2394ed314974bdb7e216284c"}
Nov 03 01:42:00.728325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:00.727985    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq" event={"ID":"fb153706-24e6-4f66-bd5b-c45f1c25de1c","Type":"ContainerStarted","Data":"737d701a765725e76b5344dae2dc0d6aabc3b0307dd2c0208fb6e3eb298abf29"}
Nov 03 01:42:00.837561 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:00.837511    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq" podStartSLOduration=1.837496872 podStartE2EDuration="1.837496872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:00.8357721 +0000 UTC m=+1400.261378582" watchObservedRunningTime="2026-11-03 01:42:00.837496872 +0000 UTC m=+1400.263103337"
Nov 03 01:42:01.737536 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:01.737492    5508 generic.go:356] "Generic (PLEG): container finished" podID="fb153706-24e6-4f66-bd5b-c45f1c25de1c" containerID="d95369dca4cd8c680bc40497f0916ad4218de6fa2394ed314974bdb7e216284c" exitCode=0
Nov 03 01:42:01.738036 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:01.737567    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq" event={"ID":"fb153706-24e6-4f66-bd5b-c45f1c25de1c","Type":"ContainerDied","Data":"d95369dca4cd8c680bc40497f0916ad4218de6fa2394ed314974bdb7e216284c"}
Nov 03 01:42:02.921193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:02.921171    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"
Nov 03 01:42:03.126503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:03.116809    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-xks-ca\") pod \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\" (UID: \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\") "
Nov 03 01:42:03.126503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:03.117013    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-openshift-ca\") pod \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\" (UID: \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\") "
Nov 03 01:42:03.126503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:03.117298    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb153706-24e6-4f66-bd5b-c45f1c25de1c-kube-api-access-wwv6h\" (UniqueName: \"kubernetes.io/projected/fb153706-24e6-4f66-bd5b-c45f1c25de1c-kube-api-access-wwv6h\") pod \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\" (UID: \"fb153706-24e6-4f66-bd5b-c45f1c25de1c\") "
Nov 03 01:42:03.147780 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:03.147724    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-openshift-ca" pod "fb153706-24e6-4f66-bd5b-c45f1c25de1c" (UID: "fb153706-24e6-4f66-bd5b-c45f1c25de1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:42:03.154079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:03.154025    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb153706-24e6-4f66-bd5b-c45f1c25de1c-kube-api-access-wwv6h" pod "fb153706-24e6-4f66-bd5b-c45f1c25de1c" (UID: "fb153706-24e6-4f66-bd5b-c45f1c25de1c"). InnerVolumeSpecName "kube-api-access-wwv6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:03.186543 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:03.186487    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-xks-ca" pod "fb153706-24e6-4f66-bd5b-c45f1c25de1c" (UID: "fb153706-24e6-4f66-bd5b-c45f1c25de1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:42:03.219259 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:03.219195    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:03.219259 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:03.219225    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwv6h\" (UniqueName: \"kubernetes.io/projected/fb153706-24e6-4f66-bd5b-c45f1c25de1c-kube-api-access-wwv6h\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:03.219259 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:03.219237    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb153706-24e6-4f66-bd5b-c45f1c25de1c-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:03.756386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:03.756319    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq" event={"ID":"fb153706-24e6-4f66-bd5b-c45f1c25de1c","Type":"ContainerDied","Data":"737d701a765725e76b5344dae2dc0d6aabc3b0307dd2c0208fb6e3eb298abf29"}
Nov 03 01:42:03.756606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:03.756395    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="737d701a765725e76b5344dae2dc0d6aabc3b0307dd2c0208fb6e3eb298abf29"
Nov 03 01:42:03.756606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:03.756426    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"
Nov 03 01:42:13.813170 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:42:13.813136    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:42:24.733550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:24.733516    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:42:27.238372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:27.237481    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:42:27.305462 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:27.305418    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 01:42:27.615550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:27.615449    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:42:28.058656 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.058599    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"]
Nov 03 01:42:28.060104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.060075    5508 cpu_manager.go:414] "containerMap: removing container" podUID="fb153706-24e6-4f66-bd5b-c45f1c25de1c" containerName="revoke-keys"
Nov 03 01:42:28.060291 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.060273    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb153706-24e6-4f66-bd5b-c45f1c25de1c" containerName="revoke-keys"
Nov 03 01:42:28.060669 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.060646    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb153706-24e6-4f66-bd5b-c45f1c25de1c" containerName="revoke-keys"
Nov 03 01:42:28.076916 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.076883    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.153375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.127447    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-948470-openshift-default-dockercfg-2d8gr\""
Nov 03 01:42:28.278969 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.229183    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9618bd52-f012-4dbb-b707-53a666005df7-istio-podinfo\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.299587 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.299497    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-data\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.299897 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.299876    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npkcg\" (UniqueName: \"kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-kube-api-access-npkcg\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.310417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.300136    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-istio-token\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.347520 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.347481    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-socket\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.347754 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.347735    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-credential-socket\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.347955 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.347935    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-certs\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.354477 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.354432    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9618bd52-f012-4dbb-b707-53a666005df7-istiod-ca-cert\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.354800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.354779    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-envoy\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.361535 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.278486    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:42:28.369372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.311081    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"]
Nov 03 01:42:28.378381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.371462    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:42:28.413204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.413176    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:42:28.486831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.475566    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-data\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.486831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.475620    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npkcg\" (UniqueName: \"kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-kube-api-access-npkcg\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.486831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.475745    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-istio-token\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.486831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.475807    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-socket\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.486831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.475834    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-credential-socket\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.486831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.475916    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-certs\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.486831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.475947    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9618bd52-f012-4dbb-b707-53a666005df7-istiod-ca-cert\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.486831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.476078    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-envoy\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.486831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.476112    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9618bd52-f012-4dbb-b707-53a666005df7-istio-podinfo\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.543368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.520120    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-socket\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.544162 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.526771    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-credential-socket\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.544331 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.527047    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-certs\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.544459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.527686    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9618bd52-f012-4dbb-b707-53a666005df7-istiod-ca-cert\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.544547 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.540975    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9618bd52-f012-4dbb-b707-53a666005df7-istio-podinfo\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.568214 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.567466    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-data\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.628781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.615973    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded"
Nov 03 01:42:28.628781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.627897    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:42:28.753377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.723907    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-envoy\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.828470 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.754873    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:42:28.974454 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.974420    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npkcg\" (UniqueName: \"kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-kube-api-access-npkcg\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:28.994372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:28.979768    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-istio-token\") pod \"e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") " pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:29.021234 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:29.020955    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:29.346549 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:29.346503    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:42:29.443290 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:29.443204    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 01:42:29.886872 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:29.886839    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"]
Nov 03 01:42:29.888311 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:42:29.888273    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9618bd52_f012_4dbb_b707_53a666005df7.slice/crio-2feeace6974a6f910691b1dd4668fbbcb27c2b2b3044e0fefab02ebabc46ab5e WatchSource:0}: Error finding container 2feeace6974a6f910691b1dd4668fbbcb27c2b2b3044e0fefab02ebabc46ab5e: Status 404 returned error can't find the container with id 2feeace6974a6f910691b1dd4668fbbcb27c2b2b3044e0fefab02ebabc46ab5e
Nov 03 01:42:29.891463 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:29.891421    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:42:29.891548 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:29.891510    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:42:29.891587 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:29.891552    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:42:30.214796 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:30.214759    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" event={"ID":"9618bd52-f012-4dbb-b707-53a666005df7","Type":"ContainerStarted","Data":"0dd8244730903977d68c22882139c63e3b29674e456ada8b20bd9529a59dd183"}
Nov 03 01:42:30.215017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:30.214812    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" event={"ID":"9618bd52-f012-4dbb-b707-53a666005df7","Type":"ContainerStarted","Data":"2feeace6974a6f910691b1dd4668fbbcb27c2b2b3044e0fefab02ebabc46ab5e"}
Nov 03 01:42:30.333467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:30.333416    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" podStartSLOduration=3.333401553 podStartE2EDuration="3.333401553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:30.328415508 +0000 UTC m=+1429.754021992" watchObservedRunningTime="2026-11-03 01:42:30.333401553 +0000 UTC m=+1429.759008018"
Nov 03 01:42:31.028625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:31.028534    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:31.034006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:31.033983    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:31.222405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:31.222377    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:31.223263 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:31.223241    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:42:34.464526 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:34.464478    5508 patch_prober.go:29] interesting pod/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:42:34.465020 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:34.464599    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:42:37.289243 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:37.289199    5508 generic.go:356] "Generic (PLEG): container finished" podID="83390757-f37d-4198-b4ac-bc05f260fa82" containerID="54e9673589e42ee976169781ee949a322cdda03c97a34c0236981a90569c255c" exitCode=0
Nov 03 01:42:37.289823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:37.289299    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" event={"ID":"83390757-f37d-4198-b4ac-bc05f260fa82","Type":"ContainerDied","Data":"54e9673589e42ee976169781ee949a322cdda03c97a34c0236981a90569c255c"}
Nov 03 01:42:37.289908 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:37.289890    5508 scope.go:122] "RemoveContainer" containerID="54e9673589e42ee976169781ee949a322cdda03c97a34c0236981a90569c255c"
Nov 03 01:42:37.291525 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:37.291500    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:42:37.291617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:37.291568    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:42:37.291617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:37.291602    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:42:38.300662 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:38.300627    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" event={"ID":"83390757-f37d-4198-b4ac-bc05f260fa82","Type":"ContainerStarted","Data":"375c972a106dfcba9f7f0db96a0e09179e38c055747bfe9777ee711d6a708467"}
Nov 03 01:42:38.314434 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:38.314404    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:42:38.314598 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:38.314575    5508 patch_prober.go:29] interesting pod/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 01:42:38.314640 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:38.314617    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 01:42:39.314795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.314758    5508 patch_prober.go:29] interesting pod/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 01:42:39.315311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.314819    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 01:42:39.781304 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.781245    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"]
Nov 03 01:42:39.786692 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.786666    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"
Nov 03 01:42:39.828685 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.828655    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"]
Nov 03 01:42:39.836917 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.835832    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:42:39.870636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.870603    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:42:39.870993 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.870972    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:42:39.871227 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.871211    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:42:39.895549 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.895510    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gvg6\" (UniqueName: \"kubernetes.io/projected/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-kube-api-access-2gvg6\") pod \"maas-api-revoke-sub-573493fe-hqvn5\" (UID: \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"
Nov 03 01:42:39.895924 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.895903    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-573493fe-hqvn5\" (UID: \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"
Nov 03 01:42:39.896110 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.896095    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-573493fe-hqvn5\" (UID: \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"
Nov 03 01:42:39.997905 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.997870    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gvg6\" (UniqueName: \"kubernetes.io/projected/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-kube-api-access-2gvg6\") pod \"maas-api-revoke-sub-573493fe-hqvn5\" (UID: \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"
Nov 03 01:42:39.998108 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.998039    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-573493fe-hqvn5\" (UID: \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"
Nov 03 01:42:39.998400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.998371    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-573493fe-hqvn5\" (UID: \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"
Nov 03 01:42:39.999213 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:39.999185    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-573493fe-hqvn5\" (UID: \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"
Nov 03 01:42:40.000842 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:40.000820    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-573493fe-hqvn5\" (UID: \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"
Nov 03 01:42:40.080109 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:40.080027    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gvg6\" (UniqueName: \"kubernetes.io/projected/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-kube-api-access-2gvg6\") pod \"maas-api-revoke-sub-573493fe-hqvn5\" (UID: \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"
Nov 03 01:42:40.104595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:40.103657    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"
Nov 03 01:42:40.316493 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:40.314858    5508 patch_prober.go:29] interesting pod/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 01:42:40.316493 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:40.314919    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 01:42:40.520440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:40.520327    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"]
Nov 03 01:42:40.523446 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:42:40.523410    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211dc2ea_ab3c_4f44_9b46_b4d5daa1fcb5.slice/crio-cbe5a3d6bbdad423e88e7cf936dfa2e0e375a752e62c3b7bacddeeae23b92b94 WatchSource:0}: Error finding container cbe5a3d6bbdad423e88e7cf936dfa2e0e375a752e62c3b7bacddeeae23b92b94: Status 404 returned error can't find the container with id cbe5a3d6bbdad423e88e7cf936dfa2e0e375a752e62c3b7bacddeeae23b92b94
Nov 03 01:42:41.218940 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.187576    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"]
Nov 03 01:42:41.267388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.267251    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"
Nov 03 01:42:41.341371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.316502    5508 patch_prober.go:29] interesting pod/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 01:42:41.446089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.442633    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 01:42:41.446393 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.421697    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"]
Nov 03 01:42:41.446574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.354555    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f34bcde-6db6-468d-b473-4f41de31149a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25\" (UID: \"1f34bcde-6db6-468d-b473-4f41de31149a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"
Nov 03 01:42:41.447063 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.447037    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr4jw\" (UniqueName: \"kubernetes.io/projected/1f34bcde-6db6-468d-b473-4f41de31149a-kube-api-access-mr4jw\") pod \"payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25\" (UID: \"1f34bcde-6db6-468d-b473-4f41de31149a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"
Nov 03 01:42:41.455873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.455829    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-948470-dockercfg-vk7hk\""
Nov 03 01:42:41.576406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.459024    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-948470\""
Nov 03 01:42:41.576406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.572149    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"]
Nov 03 01:42:41.576406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.574382    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr4jw\" (UniqueName: \"kubernetes.io/projected/1f34bcde-6db6-468d-b473-4f41de31149a-kube-api-access-mr4jw\") pod \"payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25\" (UID: \"1f34bcde-6db6-468d-b473-4f41de31149a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"
Nov 03 01:42:41.576406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.574519    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f34bcde-6db6-468d-b473-4f41de31149a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25\" (UID: \"1f34bcde-6db6-468d-b473-4f41de31149a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"
Nov 03 01:42:41.576406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.575688    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f34bcde-6db6-468d-b473-4f41de31149a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25\" (UID: \"1f34bcde-6db6-468d-b473-4f41de31149a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"
Nov 03 01:42:41.584656 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.582758    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"]
Nov 03 01:42:41.584656 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.582865    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"
Nov 03 01:42:41.601825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.601787    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5" event={"ID":"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5","Type":"ContainerStarted","Data":"cbe5a3d6bbdad423e88e7cf936dfa2e0e375a752e62c3b7bacddeeae23b92b94"}
Nov 03 01:42:41.634824 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.634795    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr4jw\" (UniqueName: \"kubernetes.io/projected/1f34bcde-6db6-468d-b473-4f41de31149a-kube-api-access-mr4jw\") pod \"payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25\" (UID: \"1f34bcde-6db6-468d-b473-4f41de31149a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"
Nov 03 01:42:41.675772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.675725    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgd62\" (UniqueName: \"kubernetes.io/projected/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-kube-api-access-qgd62\") pod \"payload-processing-e2e-api-a-948470-cb959cb87-d2w72\" (UID: \"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e\") " pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"
Nov 03 01:42:41.676238 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.676210    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-948470-cb959cb87-d2w72\" (UID: \"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e\") " pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"
Nov 03 01:42:41.748184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.748036    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5" podStartSLOduration=2.748023581 podStartE2EDuration="2.748023581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:41.747527399 +0000 UTC m=+1441.173133864" watchObservedRunningTime="2026-11-03 01:42:41.748023581 +0000 UTC m=+1441.173630043"
Nov 03 01:42:41.782637 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.782596    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgd62\" (UniqueName: \"kubernetes.io/projected/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-kube-api-access-qgd62\") pod \"payload-processing-e2e-api-a-948470-cb959cb87-d2w72\" (UID: \"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e\") " pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"
Nov 03 01:42:41.782814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.782779    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-948470-cb959cb87-d2w72\" (UID: \"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e\") " pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"
Nov 03 01:42:41.783758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.783733    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-948470-cb959cb87-d2w72\" (UID: \"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e\") " pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"
Nov 03 01:42:41.847945 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.847909    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgd62\" (UniqueName: \"kubernetes.io/projected/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-kube-api-access-qgd62\") pod \"payload-processing-e2e-api-a-948470-cb959cb87-d2w72\" (UID: \"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e\") " pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"
Nov 03 01:42:41.905891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.905861    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"
Nov 03 01:42:41.915038 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:41.915000    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"
Nov 03 01:42:42.315853 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:42.315770    5508 patch_prober.go:29] interesting pod/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 01:42:42.315853 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:42.315829    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 01:42:42.316039 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:42.315875    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"]
Nov 03 01:42:42.380412 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:42.380179    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"]
Nov 03 01:42:42.381629 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:42:42.381588    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf383e9_e6fa_4cd8_b46a_8b3924c9fb8e.slice/crio-5cd01ff7136250f9af5a24b57c5dd843d258d160a5221faeebd0f440a903d46d WatchSource:0}: Error finding container 5cd01ff7136250f9af5a24b57c5dd843d258d160a5221faeebd0f440a903d46d: Status 404 returned error can't find the container with id 5cd01ff7136250f9af5a24b57c5dd843d258d160a5221faeebd0f440a903d46d
Nov 03 01:42:42.702575 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:42.701776    5508 generic.go:356] "Generic (PLEG): container finished" podID="211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5" containerID="9d7d1c3a7d7ab22a27d1ee921cc579f206be6628736bb7a70848cbc6fe0e709a" exitCode=0
Nov 03 01:42:42.707099 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:42.707059    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5" event={"ID":"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5","Type":"ContainerDied","Data":"9d7d1c3a7d7ab22a27d1ee921cc579f206be6628736bb7a70848cbc6fe0e709a"}
Nov 03 01:42:42.707235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:42.707124    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72" event={"ID":"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e","Type":"ContainerStarted","Data":"5cd01ff7136250f9af5a24b57c5dd843d258d160a5221faeebd0f440a903d46d"}
Nov 03 01:42:42.708087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:42.708066    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25" event={"ID":"1f34bcde-6db6-468d-b473-4f41de31149a","Type":"ContainerStarted","Data":"b636508f7957f5d24223181640a06f603c6ae0f90ab733190908d5757cd9c767"}
Nov 03 01:42:42.708286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:42.708271    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"
Nov 03 01:42:42.991202 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:42.991128    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25" podStartSLOduration=2.991106053 podStartE2EDuration="2.991106053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:42.983450233 +0000 UTC m=+1442.409056714" watchObservedRunningTime="2026-11-03 01:42:42.991106053 +0000 UTC m=+1442.416712517"
Nov 03 01:42:43.314222 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:43.314137    5508 patch_prober.go:29] interesting pod/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Nov 03 01:42:43.314222 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:43.314190    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Nov 03 01:42:43.721626 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:43.721588    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72" event={"ID":"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e","Type":"ContainerStarted","Data":"500a916a59c09aa7f38de8cc54a793d1ac67697eaf6a89708336d70fca2fbbd4"}
Nov 03 01:42:43.722121 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:43.721749    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"
Nov 03 01:42:43.723460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:43.723407    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25" event={"ID":"1f34bcde-6db6-468d-b473-4f41de31149a","Type":"ContainerStarted","Data":"50b78491a51e210ca463c1223b765ff9bacf4f4ca4466c3c79fbd6803a9e93dd"}
Nov 03 01:42:43.828810 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:42:43.828770    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:42:43.868490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:43.868438    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72" podStartSLOduration=2.868423107 podStartE2EDuration="2.868423107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:43.864689113 +0000 UTC m=+1443.290295578" watchObservedRunningTime="2026-11-03 01:42:43.868423107 +0000 UTC m=+1443.294029570"
Nov 03 01:42:43.907044 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:43.907021    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"
Nov 03 01:42:43.981285 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:43.978506    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-xks-ca\") pod \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\" (UID: \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\") "
Nov 03 01:42:43.981285 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:43.978615    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-openshift-ca\") pod \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\" (UID: \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\") "
Nov 03 01:42:43.981285 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:43.978764    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-kube-api-access-2gvg6\" (UniqueName: \"kubernetes.io/projected/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-kube-api-access-2gvg6\") pod \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\" (UID: \"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5\") "
Nov 03 01:42:43.992597 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:43.985703    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-openshift-ca" pod "211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5" (UID: "211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:42:44.026570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:44.025529    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-xks-ca" pod "211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5" (UID: "211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:42:44.026570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:44.025667    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-kube-api-access-2gvg6" pod "211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5" (UID: "211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5"). InnerVolumeSpecName "kube-api-access-2gvg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:42:44.080800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:44.080762    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:44.080800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:44.080800    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:44.081047 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:44.080817    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gvg6\" (UniqueName: \"kubernetes.io/projected/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5-kube-api-access-2gvg6\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:42:44.319157 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:44.319071    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:42:44.319489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:44.319465    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:42:44.320074 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:44.320055    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:42:44.732896 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:44.732867    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"
Nov 03 01:42:44.733462 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:44.732909    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5" event={"ID":"211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5","Type":"ContainerDied","Data":"cbe5a3d6bbdad423e88e7cf936dfa2e0e375a752e62c3b7bacddeeae23b92b94"}
Nov 03 01:42:44.733462 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:44.732949    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbe5a3d6bbdad423e88e7cf936dfa2e0e375a752e62c3b7bacddeeae23b92b94"
Nov 03 01:42:50.802584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:50.802505    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:42:54.736144 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:54.736114    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"
Nov 03 01:42:54.736539 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:54.736187    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"
Nov 03 01:42:58.347395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.347360    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"]
Nov 03 01:42:58.348134 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.348114    5508 cpu_manager.go:414] "containerMap: removing container" podUID="211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5" containerName="revoke-keys"
Nov 03 01:42:58.348134 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.348134    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5" containerName="revoke-keys"
Nov 03 01:42:58.348287 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.348255    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5" containerName="revoke-keys"
Nov 03 01:42:58.352006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.351989    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.440327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.440283    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-socket\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.440672 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.440652    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-credential-socket\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.457401 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.457367    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-certs\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.457750 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.457730    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5j4s\" (UniqueName: \"kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-kube-api-access-c5j4s\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.457900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.457883    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-istio-token\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.461370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.458019    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916e646a-6298-47da-a869-a863a7c410fc-istiod-ca-cert\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.484543 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.484506    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-envoy\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.484873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.484854    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-data\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.485148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.485122    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916e646a-6298-47da-a869-a863a7c410fc-istio-podinfo\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.516373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.508718    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-012d31-openshift-default-dockercfg-5bbdv\""
Nov 03 01:42:58.581813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.581778    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"]
Nov 03 01:42:58.589091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.587199    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-envoy\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.589091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.587310    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-data\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.589091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.587435    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916e646a-6298-47da-a869-a863a7c410fc-istio-podinfo\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.589091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.587511    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-socket\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.589091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.587534    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-credential-socket\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.589091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.587563    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-certs\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.589091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.587625    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5j4s\" (UniqueName: \"kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-kube-api-access-c5j4s\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.589091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.587670    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-istio-token\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.589091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.587710    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916e646a-6298-47da-a869-a863a7c410fc-istiod-ca-cert\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.589091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.588501    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916e646a-6298-47da-a869-a863a7c410fc-istiod-ca-cert\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.591314 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.590209    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-credential-socket\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.591314 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.590603    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-data\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.603171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.591904    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-socket\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.614445 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.604099    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-certs\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.699510 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.697459    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-istio-token\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.699510 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.697672    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-envoy\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.699510 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.698223    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916e646a-6298-47da-a869-a863a7c410fc-istio-podinfo\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.708444 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.708333    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5j4s\" (UniqueName: \"kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-kube-api-access-c5j4s\") pod \"e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") " pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:58.971493 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:58.969777    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:42:59.412187 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:59.412156    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"]
Nov 03 01:42:59.416457 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:42:59.416411    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916e646a_6298_47da_a869_a863a7c410fc.slice/crio-7ca1b3805eb01abf8a0d21852148d2c248d88faf94794b26cb401bd92832ed87 WatchSource:0}: Error finding container 7ca1b3805eb01abf8a0d21852148d2c248d88faf94794b26cb401bd92832ed87: Status 404 returned error can't find the container with id 7ca1b3805eb01abf8a0d21852148d2c248d88faf94794b26cb401bd92832ed87
Nov 03 01:42:59.419805 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:59.419771    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:42:59.419885 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:59.419838    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:42:59.419885 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:59.419872    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:42:59.843021 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:59.842942    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"]
Nov 03 01:42:59.848172 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:59.848141    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"
Nov 03 01:42:59.897617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:59.897581    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5e048b\""
Nov 03 01:42:59.900732 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:59.900703    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"]
Nov 03 01:42:59.954381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:59.930161    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d65sd\" (UniqueName: \"kubernetes.io/projected/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-kube-api-access-d65sd\") pod \"payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb\" (UID: \"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"
Nov 03 01:42:59.962375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:59.954787    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb\" (UID: \"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"
Nov 03 01:42:59.989373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:42:59.978205    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5e048b-dockercfg-z6n4d\""
Nov 03 01:43:00.113491 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.084806    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb\" (UID: \"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"
Nov 03 01:43:00.272601 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.272557    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d65sd\" (UniqueName: \"kubernetes.io/projected/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-kube-api-access-d65sd\") pod \"payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb\" (UID: \"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"
Nov 03 01:43:00.273002 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.245030    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" event={"ID":"916e646a-6298-47da-a869-a863a7c410fc","Type":"ContainerStarted","Data":"368a31001287facf53ada9a06e66ae576581db3f0d1627395416ad4f393644e6"}
Nov 03 01:43:00.273199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.273176    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" event={"ID":"916e646a-6298-47da-a869-a863a7c410fc","Type":"ContainerStarted","Data":"7ca1b3805eb01abf8a0d21852148d2c248d88faf94794b26cb401bd92832ed87"}
Nov 03 01:43:00.273387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.273369    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"]
Nov 03 01:43:00.273618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.135165    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb\" (UID: \"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"
Nov 03 01:43:00.279941 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.279918    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"]
Nov 03 01:43:00.280047 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.280034    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"
Nov 03 01:43:00.330950 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.330908    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d65sd\" (UniqueName: \"kubernetes.io/projected/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-kube-api-access-d65sd\") pod \"payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb\" (UID: \"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"
Nov 03 01:43:00.360598 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.360549    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" podStartSLOduration=2.36053373 podStartE2EDuration="2.36053373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:00.358205232 +0000 UTC m=+1459.783811716" watchObservedRunningTime="2026-11-03 01:43:00.36053373 +0000 UTC m=+1459.786140193"
Nov 03 01:43:00.374108 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.374020    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkqfr\" (UniqueName: \"kubernetes.io/projected/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-kube-api-access-jkqfr\") pod \"payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn\" (UID: \"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"
Nov 03 01:43:00.374334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.374309    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn\" (UID: \"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"
Nov 03 01:43:00.464288 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.464250    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"
Nov 03 01:43:00.501373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.490922    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkqfr\" (UniqueName: \"kubernetes.io/projected/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-kube-api-access-jkqfr\") pod \"payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn\" (UID: \"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"
Nov 03 01:43:00.501373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.491117    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn\" (UID: \"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"
Nov 03 01:43:00.524371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.513917    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn\" (UID: \"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"
Nov 03 01:43:00.581191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.581151    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkqfr\" (UniqueName: \"kubernetes.io/projected/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-kube-api-access-jkqfr\") pod \"payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn\" (UID: \"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"
Nov 03 01:43:00.600106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.599554    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"
Nov 03 01:43:00.971103 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.970985    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:43:00.971300 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.971262    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"]
Nov 03 01:43:00.976521 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:43:00.976496    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49255ba1_cf88_4f8a_8d5f_84f260eb2d9b.slice/crio-7183970f79606d633662bb4b36900eb5d6bd52aa2e61c61698332cfd1879ff3c WatchSource:0}: Error finding container 7183970f79606d633662bb4b36900eb5d6bd52aa2e61c61698332cfd1879ff3c: Status 404 returned error can't find the container with id 7183970f79606d633662bb4b36900eb5d6bd52aa2e61c61698332cfd1879ff3c
Nov 03 01:43:00.977730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:00.977695    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:43:01.024077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:01.024055    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"]
Nov 03 01:43:01.029643 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:43:01.029617    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2a3aba_9a9c_4add_a2bd_0e7a5c5b376d.slice/crio-f9c31ad5ce9bee19becabafe00a9cf067f448ffe55837f38617fc1e78578ff13 WatchSource:0}: Error finding container f9c31ad5ce9bee19becabafe00a9cf067f448ffe55837f38617fc1e78578ff13: Status 404 returned error can't find the container with id f9c31ad5ce9bee19becabafe00a9cf067f448ffe55837f38617fc1e78578ff13
Nov 03 01:43:01.255074 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:01.254982    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb" event={"ID":"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b","Type":"ContainerStarted","Data":"7a8f820afc84a7db72f64805a8fae290dd063b67efca84dfd657171cb3de73af"}
Nov 03 01:43:01.255074 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:01.255040    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb" event={"ID":"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b","Type":"ContainerStarted","Data":"7183970f79606d633662bb4b36900eb5d6bd52aa2e61c61698332cfd1879ff3c"}
Nov 03 01:43:01.255074 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:01.255069    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"
Nov 03 01:43:01.256885 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:01.256854    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn" event={"ID":"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d","Type":"ContainerStarted","Data":"90e74c0765c0fc5856bab77ba8f0aa314f89d95aa73adf6e5eb54b97bf8621e5"}
Nov 03 01:43:01.257007 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:01.256910    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn" event={"ID":"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d","Type":"ContainerStarted","Data":"f9c31ad5ce9bee19becabafe00a9cf067f448ffe55837f38617fc1e78578ff13"}
Nov 03 01:43:01.257721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:01.257120    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:43:01.258708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:01.258687    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:43:01.371803 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:01.371749    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb" podStartSLOduration=2.371733354 podStartE2EDuration="2.371733354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:01.365685268 +0000 UTC m=+1460.791291732" watchObservedRunningTime="2026-11-03 01:43:01.371733354 +0000 UTC m=+1460.797339818"
Nov 03 01:43:01.479962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:01.479899    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn" podStartSLOduration=2.479879002 podStartE2EDuration="2.479879002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:01.475988452 +0000 UTC m=+1460.901594997" watchObservedRunningTime="2026-11-03 01:43:01.479879002 +0000 UTC m=+1460.905485467"
Nov 03 01:43:02.266267 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:02.266237    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"
Nov 03 01:43:06.886371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:06.879950    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"]
Nov 03 01:43:07.134396 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.131715    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"]
Nov 03 01:43:07.134396 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.131846    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"
Nov 03 01:43:07.236019 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.231229    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-012d31\""
Nov 03 01:43:07.314446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.314407    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"]
Nov 03 01:43:07.336382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.336334    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5556c0df-cf71-4528-ab71-71ef67e9ac3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb\" (UID: \"5556c0df-cf71-4528-ab71-71ef67e9ac3e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"
Nov 03 01:43:07.336641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.336620    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rdt\" (UniqueName: \"kubernetes.io/projected/5556c0df-cf71-4528-ab71-71ef67e9ac3e-kube-api-access-g5rdt\") pod \"payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb\" (UID: \"5556c0df-cf71-4528-ab71-71ef67e9ac3e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"
Nov 03 01:43:07.352623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.351519    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-012d31-dockercfg-z4pqg\""
Nov 03 01:43:07.356061 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.355162    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"
Nov 03 01:43:07.360497 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.360478    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"]
Nov 03 01:43:07.437755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.437730    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5556c0df-cf71-4528-ab71-71ef67e9ac3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb\" (UID: \"5556c0df-cf71-4528-ab71-71ef67e9ac3e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"
Nov 03 01:43:07.437893 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.437764    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rdt\" (UniqueName: \"kubernetes.io/projected/5556c0df-cf71-4528-ab71-71ef67e9ac3e-kube-api-access-g5rdt\") pod \"payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb\" (UID: \"5556c0df-cf71-4528-ab71-71ef67e9ac3e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"
Nov 03 01:43:07.438436 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.438418    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5556c0df-cf71-4528-ab71-71ef67e9ac3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb\" (UID: \"5556c0df-cf71-4528-ab71-71ef67e9ac3e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"
Nov 03 01:43:07.495696 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.495629    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rdt\" (UniqueName: \"kubernetes.io/projected/5556c0df-cf71-4528-ab71-71ef67e9ac3e-kube-api-access-g5rdt\") pod \"payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb\" (UID: \"5556c0df-cf71-4528-ab71-71ef67e9ac3e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"
Nov 03 01:43:07.538793 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.538767    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c23153d-c0e2-4a72-a25a-67a4e61766bb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz\" (UID: \"7c23153d-c0e2-4a72-a25a-67a4e61766bb\") " pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"
Nov 03 01:43:07.538960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.538944    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljwqn\" (UniqueName: \"kubernetes.io/projected/7c23153d-c0e2-4a72-a25a-67a4e61766bb-kube-api-access-ljwqn\") pod \"payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz\" (UID: \"7c23153d-c0e2-4a72-a25a-67a4e61766bb\") " pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"
Nov 03 01:43:07.640318 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.640292    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljwqn\" (UniqueName: \"kubernetes.io/projected/7c23153d-c0e2-4a72-a25a-67a4e61766bb-kube-api-access-ljwqn\") pod \"payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz\" (UID: \"7c23153d-c0e2-4a72-a25a-67a4e61766bb\") " pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"
Nov 03 01:43:07.640481 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.640443    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c23153d-c0e2-4a72-a25a-67a4e61766bb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz\" (UID: \"7c23153d-c0e2-4a72-a25a-67a4e61766bb\") " pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"
Nov 03 01:43:07.641075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.641058    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c23153d-c0e2-4a72-a25a-67a4e61766bb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz\" (UID: \"7c23153d-c0e2-4a72-a25a-67a4e61766bb\") " pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"
Nov 03 01:43:07.655463 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.655439    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"
Nov 03 01:43:07.698178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.698147    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljwqn\" (UniqueName: \"kubernetes.io/projected/7c23153d-c0e2-4a72-a25a-67a4e61766bb-kube-api-access-ljwqn\") pod \"payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz\" (UID: \"7c23153d-c0e2-4a72-a25a-67a4e61766bb\") " pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"
Nov 03 01:43:07.975416 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:07.975381    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"
Nov 03 01:43:08.042376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:08.042313    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"]
Nov 03 01:43:08.364623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:08.364508    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb" event={"ID":"5556c0df-cf71-4528-ab71-71ef67e9ac3e","Type":"ContainerStarted","Data":"8359e5d50c7f2ed31d318e9d0995eb36bc3113b1f807c393c9184d0936a1e35a"}
Nov 03 01:43:08.364623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:08.364569    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb" event={"ID":"5556c0df-cf71-4528-ab71-71ef67e9ac3e","Type":"ContainerStarted","Data":"da6ec7000176a4abb745cb1befef4343b858c10733ecd91a3810131689e81e3a"}
Nov 03 01:43:08.364874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:08.364729    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"
Nov 03 01:43:08.427699 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:08.427672    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"]
Nov 03 01:43:08.431472 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:43:08.431445    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c23153d_c0e2_4a72_a25a_67a4e61766bb.slice/crio-ed016b52b2111c01454ca9cce329720fc966f99ae930b884f6d818da2ee37282 WatchSource:0}: Error finding container ed016b52b2111c01454ca9cce329720fc966f99ae930b884f6d818da2ee37282: Status 404 returned error can't find the container with id ed016b52b2111c01454ca9cce329720fc966f99ae930b884f6d818da2ee37282
Nov 03 01:43:08.494458 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:08.494403    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb" podStartSLOduration=2.494383402 podStartE2EDuration="2.494383402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:08.490076349 +0000 UTC m=+1467.915682815" watchObservedRunningTime="2026-11-03 01:43:08.494383402 +0000 UTC m=+1467.919989867"
Nov 03 01:43:09.374911 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:09.374877    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz" event={"ID":"7c23153d-c0e2-4a72-a25a-67a4e61766bb","Type":"ContainerStarted","Data":"f9db715e6ec6b47ef3131961c246b5ab49fc558e92b05aea28fbc208975e19a0"}
Nov 03 01:43:09.375394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:09.374938    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz" event={"ID":"7c23153d-c0e2-4a72-a25a-67a4e61766bb","Type":"ContainerStarted","Data":"ed016b52b2111c01454ca9cce329720fc966f99ae930b884f6d818da2ee37282"}
Nov 03 01:43:09.375394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:09.374985    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"
Nov 03 01:43:09.487171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:09.485312    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz" podStartSLOduration=3.485295473 podStartE2EDuration="3.485295473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:09.483093301 +0000 UTC m=+1468.908699765" watchObservedRunningTime="2026-11-03 01:43:09.485295473 +0000 UTC m=+1468.910901940"
Nov 03 01:43:12.269874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:12.269839    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"
Nov 03 01:43:13.307917 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:13.307888    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"
Nov 03 01:43:13.815061 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:43:13.815004    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:43:19.379883 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:19.379841    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"
Nov 03 01:43:20.387617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:20.387592    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"
Nov 03 01:43:24.084342 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:24.084283    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"]
Nov 03 01:43:24.088902 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:24.088871    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578a8646-qw69z"]
Nov 03 01:43:24.693125 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:24.693095    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f5b563-9f66-4f89-ba51-96a24ccdce3c" path="/var/lib/kubelet/pods/b9f5b563-9f66-4f89-ba51-96a24ccdce3c/volumes"
Nov 03 01:43:28.046435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:28.046400    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"]
Nov 03 01:43:28.143618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:28.143581    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a5bf2f3-vkgg5"]
Nov 03 01:43:28.693917 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:28.693876    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5601949d-18e6-4ec3-a012-8c8523d2379b" path="/var/lib/kubelet/pods/5601949d-18e6-4ec3-a012-8c8523d2379b/volumes"
Nov 03 01:43:35.228575 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:35.228534    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:43:39.072372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:39.069385    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"]
Nov 03 01:43:39.096249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:39.096221    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5f0be2e-r7jp7"]
Nov 03 01:43:40.698660 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:40.698617    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c5a642-782c-4f47-abae-a3f376686970" path="/var/lib/kubelet/pods/46c5a642-782c-4f47-abae-a3f376686970/volumes"
Nov 03 01:43:40.932680 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:40.932650    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:43:40.933781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:40.933760    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:43:41.046425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.046341    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"]
Nov 03 01:43:41.092464 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.092433    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee45262-htfff"]
Nov 03 01:43:41.407943 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.407912    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"]
Nov 03 01:43:41.414118 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.414096    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.445463 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.445428    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.445768 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.445749    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55a9f34-b745-4306-83d9-5460d87832e5-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.445949 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.445929    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phzpj\" (UniqueName: \"kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-kube-api-access-phzpj\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.446153 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.446136    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.446499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.446340    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.446726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.446708    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.446907 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.446891    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.447104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.447086    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55a9f34-b745-4306-83d9-5460d87832e5-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.447253 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.447239    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.478435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.478077    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f4e704-openshift-default-dockercfg-nt8mh\""
Nov 03 01:43:41.561375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.548979    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55a9f34-b745-4306-83d9-5460d87832e5-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.561375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.549059    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzpj\" (UniqueName: \"kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-kube-api-access-phzpj\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.561375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.549133    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.561375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.549194    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.561375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.549288    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.561375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.549367    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.561375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.549455    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55a9f34-b745-4306-83d9-5460d87832e5-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.561375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.549500    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.561375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.549570    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.561375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.550065    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.561375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.555250    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.580433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.571772    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.580433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.572775    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.732418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.714900    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.785959 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.785914    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55a9f34-b745-4306-83d9-5460d87832e5-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.810852 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.810818    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phzpj\" (UniqueName: \"kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-kube-api-access-phzpj\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:41.856380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.842727    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"]
Nov 03 01:43:41.969938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:41.969908    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55a9f34-b745-4306-83d9-5460d87832e5-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:42.017413 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:42.016958    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-twv4g\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:42.078156 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:42.078123    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:42.600181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:42.600108    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"]
Nov 03 01:43:42.604311 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:43:42.604286    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55a9f34_b745_4306_83d9_5460d87832e5.slice/crio-787a3db19bf3d542d642d7f74c1632991d6803174f3d0eb3977c92ea3fef8d2c WatchSource:0}: Error finding container 787a3db19bf3d542d642d7f74c1632991d6803174f3d0eb3977c92ea3fef8d2c: Status 404 returned error can't find the container with id 787a3db19bf3d542d642d7f74c1632991d6803174f3d0eb3977c92ea3fef8d2c
Nov 03 01:43:42.607512 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:42.607476    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:43:42.607616 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:42.607564    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:43:42.607616 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:42.607607    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:43:42.696478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:42.696437    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784a5f88-f63f-41be-afce-9dddb8dc2edb" path="/var/lib/kubelet/pods/784a5f88-f63f-41be-afce-9dddb8dc2edb/volumes"
Nov 03 01:43:42.886756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:42.886679    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" event={"ID":"c55a9f34-b745-4306-83d9-5460d87832e5","Type":"ContainerStarted","Data":"40be624633397e27b1ce74e2571f33917b777b75e8b286edcfa494655021de62"}
Nov 03 01:43:42.886756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:42.886743    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" event={"ID":"c55a9f34-b745-4306-83d9-5460d87832e5","Type":"ContainerStarted","Data":"787a3db19bf3d542d642d7f74c1632991d6803174f3d0eb3977c92ea3fef8d2c"}
Nov 03 01:43:43.003231 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:43.003169    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" podStartSLOduration=2.0031535 podStartE2EDuration="2.0031535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:42.999820148 +0000 UTC m=+1502.425426636" watchObservedRunningTime="2026-11-03 01:43:43.0031535 +0000 UTC m=+1502.428759963"
Nov 03 01:43:43.079106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:43.079069    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:43.085309 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:43.085285    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:43.813610 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:43:43.813560    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:43:43.895837 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:43.895799    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:43.896789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:43.896767    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:43:44.063600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:44.063537    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"]
Nov 03 01:43:44.071646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:44.071569    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62c1bd13-mswzq"]
Nov 03 01:43:44.693948 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:44.693916    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439114f0-b0e2-4932-b9c4-1f7edd63df2f" path="/var/lib/kubelet/pods/439114f0-b0e2-4932-b9c4-1f7edd63df2f/volumes"
Nov 03 01:43:46.548314 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:46.547776    5508 patch_prober.go:29] interesting pod/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": read tcp 10.133.0.2:43482->10.133.0.122:15021: read: connection reset by peer" start-of-body=
Nov 03 01:43:46.548314 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:46.547826    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": read tcp 10.133.0.2:43482->10.133.0.122:15021: read: connection reset by peer"
Nov 03 01:43:46.974038 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:46.973994    5508 generic.go:356] "Generic (PLEG): container finished" podID="9618bd52-f012-4dbb-b707-53a666005df7" containerID="0dd8244730903977d68c22882139c63e3b29674e456ada8b20bd9529a59dd183" exitCode=0
Nov 03 01:43:46.974227 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:46.974060    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" event={"ID":"9618bd52-f012-4dbb-b707-53a666005df7","Type":"ContainerDied","Data":"0dd8244730903977d68c22882139c63e3b29674e456ada8b20bd9529a59dd183"}
Nov 03 01:43:46.974583 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:46.974566    5508 scope.go:122] "RemoveContainer" containerID="0dd8244730903977d68c22882139c63e3b29674e456ada8b20bd9529a59dd183"
Nov 03 01:43:46.976397 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:46.976367    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:43:46.976474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:46.976457    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:43:46.976518 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:46.976504    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:43:47.986415 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:47.986379    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" event={"ID":"9618bd52-f012-4dbb-b707-53a666005df7","Type":"ContainerStarted","Data":"c835ef9b6656ddda6b0e8ba6a06ed2b27e4fe3b8f10a8cba8f91f8a64b7becf7"}
Nov 03 01:43:48.028092 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:48.028064    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:43:48.028374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:48.028327    5508 patch_prober.go:29] interesting pod/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 01:43:48.028426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:48.028408    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 01:43:49.028547 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:49.028515    5508 patch_prober.go:29] interesting pod/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 01:43:49.028920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:49.028562    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 01:43:50.028126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:50.028082    5508 patch_prober.go:29] interesting pod/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 01:43:50.028333 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:50.028153    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 01:43:51.036785 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.036749    5508 patch_prober.go:29] interesting pod/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 01:43:51.054805 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.054750    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 01:43:51.056929 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.056901    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"]
Nov 03 01:43:51.319686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.315730    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"]
Nov 03 01:43:51.319686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.315854    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"
Nov 03 01:43:51.319686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.315946    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"]
Nov 03 01:43:51.341786 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.341536    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"]
Nov 03 01:43:51.341786 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.341657    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"
Nov 03 01:43:51.366454 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.366420    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f4e704\""
Nov 03 01:43:51.422795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.416796    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f4e704-dockercfg-sxqlm\""
Nov 03 01:43:51.440964 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.440936    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75fl\" (UniqueName: \"kubernetes.io/projected/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-kube-api-access-g75fl\") pod \"payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7\" (UID: \"01dc3983-c36e-4e40-a7ea-a61d27f24d4a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"
Nov 03 01:43:51.441137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.441111    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrg2t\" (UniqueName: \"kubernetes.io/projected/65d323ac-8a9a-422a-b97c-26c7e9b096ca-kube-api-access-wrg2t\") pod \"payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz\" (UID: \"65d323ac-8a9a-422a-b97c-26c7e9b096ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"
Nov 03 01:43:51.441203 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.441187    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7\" (UID: \"01dc3983-c36e-4e40-a7ea-a61d27f24d4a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"
Nov 03 01:43:51.441287 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.441216    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d323ac-8a9a-422a-b97c-26c7e9b096ca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz\" (UID: \"65d323ac-8a9a-422a-b97c-26c7e9b096ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"
Nov 03 01:43:51.542863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.542428    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7\" (UID: \"01dc3983-c36e-4e40-a7ea-a61d27f24d4a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"
Nov 03 01:43:51.542863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.542479    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d323ac-8a9a-422a-b97c-26c7e9b096ca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz\" (UID: \"65d323ac-8a9a-422a-b97c-26c7e9b096ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"
Nov 03 01:43:51.542863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.542681    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75fl\" (UniqueName: \"kubernetes.io/projected/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-kube-api-access-g75fl\") pod \"payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7\" (UID: \"01dc3983-c36e-4e40-a7ea-a61d27f24d4a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"
Nov 03 01:43:51.542863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.542864    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrg2t\" (UniqueName: \"kubernetes.io/projected/65d323ac-8a9a-422a-b97c-26c7e9b096ca-kube-api-access-wrg2t\") pod \"payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz\" (UID: \"65d323ac-8a9a-422a-b97c-26c7e9b096ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"
Nov 03 01:43:51.543326 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.543306    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d323ac-8a9a-422a-b97c-26c7e9b096ca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz\" (UID: \"65d323ac-8a9a-422a-b97c-26c7e9b096ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"
Nov 03 01:43:51.543428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.543325    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7\" (UID: \"01dc3983-c36e-4e40-a7ea-a61d27f24d4a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"
Nov 03 01:43:51.600984 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.600913    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75fl\" (UniqueName: \"kubernetes.io/projected/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-kube-api-access-g75fl\") pod \"payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7\" (UID: \"01dc3983-c36e-4e40-a7ea-a61d27f24d4a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"
Nov 03 01:43:51.601155 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.601041    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrg2t\" (UniqueName: \"kubernetes.io/projected/65d323ac-8a9a-422a-b97c-26c7e9b096ca-kube-api-access-wrg2t\") pod \"payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz\" (UID: \"65d323ac-8a9a-422a-b97c-26c7e9b096ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"
Nov 03 01:43:51.739801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.739771    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"
Nov 03 01:43:51.775387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:51.772610    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"
Nov 03 01:43:52.028765 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:52.028735    5508 patch_prober.go:29] interesting pod/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 01:43:52.028910 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:52.028787    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 01:43:52.181529 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:52.181491    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"]
Nov 03 01:43:52.198929 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:43:52.198896    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01dc3983_c36e_4e40_a7ea_a61d27f24d4a.slice/crio-cb33339271788f94493af15deaf22ca5425dd3bfadca720ce5200ace7a8261b2 WatchSource:0}: Error finding container cb33339271788f94493af15deaf22ca5425dd3bfadca720ce5200ace7a8261b2: Status 404 returned error can't find the container with id cb33339271788f94493af15deaf22ca5425dd3bfadca720ce5200ace7a8261b2
Nov 03 01:43:52.208339 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:52.208310    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"]
Nov 03 01:43:52.301857 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:52.301789    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7" event={"ID":"01dc3983-c36e-4e40-a7ea-a61d27f24d4a","Type":"ContainerStarted","Data":"cb33339271788f94493af15deaf22ca5425dd3bfadca720ce5200ace7a8261b2"}
Nov 03 01:43:52.304628 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:52.304601    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz" event={"ID":"65d323ac-8a9a-422a-b97c-26c7e9b096ca","Type":"ContainerStarted","Data":"82c5e7175ac9813b090be0772d486596f00bcc21be124417250dd6e300af1a70"}
Nov 03 01:43:53.043368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:53.034910    5508 patch_prober.go:29] interesting pod/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 03 01:43:53.043368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:53.034970    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 03 01:43:53.316335 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:53.316254    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7" event={"ID":"01dc3983-c36e-4e40-a7ea-a61d27f24d4a","Type":"ContainerStarted","Data":"f2fb342704c8e272fe2d28da8bf9e2b06a24599450f4cb66de2f72165360e9d1"}
Nov 03 01:43:53.316745 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:53.316365    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"
Nov 03 01:43:53.318134 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:53.318111    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz" event={"ID":"65d323ac-8a9a-422a-b97c-26c7e9b096ca","Type":"ContainerStarted","Data":"31a4f447fef68dc35ee9d8e4c9a8603e5ad29d7dc191a83b76ead173a3b6a124"}
Nov 03 01:43:53.318254 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:53.318176    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"
Nov 03 01:43:53.431231 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:53.431178    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7" podStartSLOduration=3.431163958 podStartE2EDuration="3.431163958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:53.427182835 +0000 UTC m=+1512.852789310" watchObservedRunningTime="2026-11-03 01:43:53.431163958 +0000 UTC m=+1512.856770401"
Nov 03 01:43:54.032370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:54.032325    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:43:54.032789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:54.032762    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:43:54.033548 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:54.033526    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:43:54.156433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:54.156369    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz" podStartSLOduration=4.156335798 podStartE2EDuration="4.156335798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:43:53.542651342 +0000 UTC m=+1512.968257808" watchObservedRunningTime="2026-11-03 01:43:54.156335798 +0000 UTC m=+1513.581942264"
Nov 03 01:43:56.063298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:56.063084    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"]
Nov 03 01:43:56.114380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:56.112879    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e67cf21-2hlq2"]
Nov 03 01:43:56.696294 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:56.696261    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a4b93a1-df1e-4b40-8dab-f2547f2a1224" path="/var/lib/kubelet/pods/2a4b93a1-df1e-4b40-8dab-f2547f2a1224/volumes"
Nov 03 01:43:58.062398 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.057693    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"]
Nov 03 01:43:58.093121 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.093092    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daa5bab2-qzk88"]
Nov 03 01:43:58.634647 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.634617    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"]
Nov 03 01:43:58.639457 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.639439    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"
Nov 03 01:43:58.700662 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.700636    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ed081b-fb60-4650-8e62-b162c9053888" path="/var/lib/kubelet/pods/30ed081b-fb60-4650-8e62-b162c9053888/volumes"
Nov 03 01:43:58.700974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.700956    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"]
Nov 03 01:43:58.732614 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.732582    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:43:58.732774 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.732729    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:43:58.732774 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.732734    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:43:58.734219 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.734201    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:43:58.764150 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.764128    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzjwc\" (UniqueName: \"kubernetes.io/projected/945531b5-dd6e-473b-a0c3-f54b71cd150a-kube-api-access-vzjwc\") pod \"maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t\" (UID: \"945531b5-dd6e-473b-a0c3-f54b71cd150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"
Nov 03 01:43:58.764288 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.764164    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t\" (UID: \"945531b5-dd6e-473b-a0c3-f54b71cd150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"
Nov 03 01:43:58.764288 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.764241    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t\" (UID: \"945531b5-dd6e-473b-a0c3-f54b71cd150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"
Nov 03 01:43:58.865932 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.865906    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzjwc\" (UniqueName: \"kubernetes.io/projected/945531b5-dd6e-473b-a0c3-f54b71cd150a-kube-api-access-vzjwc\") pod \"maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t\" (UID: \"945531b5-dd6e-473b-a0c3-f54b71cd150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"
Nov 03 01:43:58.866069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.865946    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t\" (UID: \"945531b5-dd6e-473b-a0c3-f54b71cd150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"
Nov 03 01:43:58.866069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.865990    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t\" (UID: \"945531b5-dd6e-473b-a0c3-f54b71cd150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"
Nov 03 01:43:58.866788 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.866761    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t\" (UID: \"945531b5-dd6e-473b-a0c3-f54b71cd150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"
Nov 03 01:43:58.868405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.868382    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t\" (UID: \"945531b5-dd6e-473b-a0c3-f54b71cd150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"
Nov 03 01:43:58.922762 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.922738    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzjwc\" (UniqueName: \"kubernetes.io/projected/945531b5-dd6e-473b-a0c3-f54b71cd150a-kube-api-access-vzjwc\") pod \"maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t\" (UID: \"945531b5-dd6e-473b-a0c3-f54b71cd150a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"
Nov 03 01:43:58.954597 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:58.954575    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"
Nov 03 01:43:59.339293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:59.339222    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"]
Nov 03 01:43:59.344830 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:43:59.344803    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945531b5_dd6e_473b_a0c3_f54b71cd150a.slice/crio-19bfa43d9a4e734cda0a8100b40209a917c1843c1f97365d52263d7f8ef19454 WatchSource:0}: Error finding container 19bfa43d9a4e734cda0a8100b40209a917c1843c1f97365d52263d7f8ef19454: Status 404 returned error can't find the container with id 19bfa43d9a4e734cda0a8100b40209a917c1843c1f97365d52263d7f8ef19454
Nov 03 01:43:59.454091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:43:59.454064    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t" event={"ID":"945531b5-dd6e-473b-a0c3-f54b71cd150a","Type":"ContainerStarted","Data":"19bfa43d9a4e734cda0a8100b40209a917c1843c1f97365d52263d7f8ef19454"}
Nov 03 01:44:00.465506 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:00.465472    5508 generic.go:356] "Generic (PLEG): container finished" podID="945531b5-dd6e-473b-a0c3-f54b71cd150a" containerID="08a9faa55a47add4022d5fea90f45284afe6b55c3c0064ce50393afc78e710f2" exitCode=0
Nov 03 01:44:00.465986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:00.465561    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t" event={"ID":"945531b5-dd6e-473b-a0c3-f54b71cd150a","Type":"ContainerDied","Data":"08a9faa55a47add4022d5fea90f45284afe6b55c3c0064ce50393afc78e710f2"}
Nov 03 01:44:01.644322 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:01.644284    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"
Nov 03 01:44:01.716515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:01.716487    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945531b5-dd6e-473b-a0c3-f54b71cd150a-kube-api-access-vzjwc\" (UniqueName: \"kubernetes.io/projected/945531b5-dd6e-473b-a0c3-f54b71cd150a-kube-api-access-vzjwc\") pod \"945531b5-dd6e-473b-a0c3-f54b71cd150a\" (UID: \"945531b5-dd6e-473b-a0c3-f54b71cd150a\") "
Nov 03 01:44:01.716650 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:01.716531    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-xks-ca\") pod \"945531b5-dd6e-473b-a0c3-f54b71cd150a\" (UID: \"945531b5-dd6e-473b-a0c3-f54b71cd150a\") "
Nov 03 01:44:01.716699 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:01.716679    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-openshift-ca\") pod \"945531b5-dd6e-473b-a0c3-f54b71cd150a\" (UID: \"945531b5-dd6e-473b-a0c3-f54b71cd150a\") "
Nov 03 01:44:01.717001 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:01.716978    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-openshift-ca" pod "945531b5-dd6e-473b-a0c3-f54b71cd150a" (UID: "945531b5-dd6e-473b-a0c3-f54b71cd150a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:44:01.717239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:01.717215    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:44:01.718538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:01.718518    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945531b5-dd6e-473b-a0c3-f54b71cd150a-kube-api-access-vzjwc" pod "945531b5-dd6e-473b-a0c3-f54b71cd150a" (UID: "945531b5-dd6e-473b-a0c3-f54b71cd150a"). InnerVolumeSpecName "kube-api-access-vzjwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:44:01.718640 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:01.718626    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-xks-ca" pod "945531b5-dd6e-473b-a0c3-f54b71cd150a" (UID: "945531b5-dd6e-473b-a0c3-f54b71cd150a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:44:01.818138 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:01.818109    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzjwc\" (UniqueName: \"kubernetes.io/projected/945531b5-dd6e-473b-a0c3-f54b71cd150a-kube-api-access-vzjwc\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:44:01.818138 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:01.818136    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945531b5-dd6e-473b-a0c3-f54b71cd150a-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:44:02.486826 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:02.486796    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"
Nov 03 01:44:02.486826 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:02.486817    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t" event={"ID":"945531b5-dd6e-473b-a0c3-f54b71cd150a","Type":"ContainerDied","Data":"19bfa43d9a4e734cda0a8100b40209a917c1843c1f97365d52263d7f8ef19454"}
Nov 03 01:44:02.487032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:02.486860    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19bfa43d9a4e734cda0a8100b40209a917c1843c1f97365d52263d7f8ef19454"
Nov 03 01:44:04.331930 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:04.331903    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"
Nov 03 01:44:04.332376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:04.331975    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"
Nov 03 01:44:10.052472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:10.052441    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"]
Nov 03 01:44:10.099267 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:10.099231    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"]
Nov 03 01:44:10.103653 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:10.103625    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c17edc5-85sdv"]
Nov 03 01:44:10.737335 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:10.737296    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f28066b-46f4-4ce3-8e94-76442f440e3f" path="/var/lib/kubelet/pods/6f28066b-46f4-4ce3-8e94-76442f440e3f/volumes"
Nov 03 01:44:13.051603 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:13.051562    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"]
Nov 03 01:44:13.097401 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:13.097373    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6cd7ec-gl27f"]
Nov 03 01:44:13.813375 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:44:13.813329    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:44:14.694276 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:14.694244    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b97e649a-42c7-48d0-80f5-51f24407ee8b" path="/var/lib/kubelet/pods/b97e649a-42c7-48d0-80f5-51f24407ee8b/volumes"
Nov 03 01:44:15.099663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:15.099569    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx" podUID="59e1dcd4-ac74-4991-a93e-546e3f5e1231" containerName="payload-processing" containerID="cri-o://2d2ffa070f2daa4d1864aca1838bd37798fae8ba5daf676b2f8d9f4e98800682" gracePeriod=25
Nov 03 01:44:17.682546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.682455    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"]
Nov 03 01:44:17.683236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.683169    5508 cpu_manager.go:414] "containerMap: removing container" podUID="945531b5-dd6e-473b-a0c3-f54b71cd150a" containerName="revoke-keys"
Nov 03 01:44:17.683236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.683183    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945531b5-dd6e-473b-a0c3-f54b71cd150a" containerName="revoke-keys"
Nov 03 01:44:17.683336 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.683277    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="945531b5-dd6e-473b-a0c3-f54b71cd150a" containerName="revoke-keys"
Nov 03 01:44:17.688648 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.688620    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"
Nov 03 01:44:17.741986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.741963    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"]
Nov 03 01:44:17.769012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.768982    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkmsm\" (UniqueName: \"kubernetes.io/projected/25245a3a-7fc5-440a-bc25-756977548b2d-kube-api-access-vkmsm\") pod \"maas-api-revoke-sub-b95171d6-mjcdp\" (UID: \"25245a3a-7fc5-440a-bc25-756977548b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"
Nov 03 01:44:17.769170 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.769146    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b95171d6-mjcdp\" (UID: \"25245a3a-7fc5-440a-bc25-756977548b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"
Nov 03 01:44:17.769488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.769470    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b95171d6-mjcdp\" (UID: \"25245a3a-7fc5-440a-bc25-756977548b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"
Nov 03 01:44:17.780770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.780748    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:44:17.780900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.780779    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:44:17.780900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.780796    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:44:17.784770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.784750    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:44:17.870763 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.870739    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b95171d6-mjcdp\" (UID: \"25245a3a-7fc5-440a-bc25-756977548b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"
Nov 03 01:44:17.870905 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.870803    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkmsm\" (UniqueName: \"kubernetes.io/projected/25245a3a-7fc5-440a-bc25-756977548b2d-kube-api-access-vkmsm\") pod \"maas-api-revoke-sub-b95171d6-mjcdp\" (UID: \"25245a3a-7fc5-440a-bc25-756977548b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"
Nov 03 01:44:17.870905 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.870879    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b95171d6-mjcdp\" (UID: \"25245a3a-7fc5-440a-bc25-756977548b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"
Nov 03 01:44:17.871376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.871329    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b95171d6-mjcdp\" (UID: \"25245a3a-7fc5-440a-bc25-756977548b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"
Nov 03 01:44:17.873212 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.873194    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b95171d6-mjcdp\" (UID: \"25245a3a-7fc5-440a-bc25-756977548b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"
Nov 03 01:44:17.928664 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:17.928641    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkmsm\" (UniqueName: \"kubernetes.io/projected/25245a3a-7fc5-440a-bc25-756977548b2d-kube-api-access-vkmsm\") pod \"maas-api-revoke-sub-b95171d6-mjcdp\" (UID: \"25245a3a-7fc5-440a-bc25-756977548b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"
Nov 03 01:44:18.006657 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:18.006599    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"
Nov 03 01:44:18.385121 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:18.385044    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"]
Nov 03 01:44:18.389331 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:44:18.389303    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25245a3a_7fc5_440a_bc25_756977548b2d.slice/crio-38824f8aecfb6e4fb276900e22a80576046dcd3c875888d00e664e77606255d4 WatchSource:0}: Error finding container 38824f8aecfb6e4fb276900e22a80576046dcd3c875888d00e664e77606255d4: Status 404 returned error can't find the container with id 38824f8aecfb6e4fb276900e22a80576046dcd3c875888d00e664e77606255d4
Nov 03 01:44:18.958099 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:18.958058    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp" event={"ID":"25245a3a-7fc5-440a-bc25-756977548b2d","Type":"ContainerStarted","Data":"baf9b204e6c2fa1fd7284a0c596dd804c838a449dfc4f8cd67ace1b6d5f3b31c"}
Nov 03 01:44:18.958563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:18.958126    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp" event={"ID":"25245a3a-7fc5-440a-bc25-756977548b2d","Type":"ContainerStarted","Data":"38824f8aecfb6e4fb276900e22a80576046dcd3c875888d00e664e77606255d4"}
Nov 03 01:44:19.069417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:19.069266    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp" podStartSLOduration=2.069251668 podStartE2EDuration="2.069251668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:19.065745684 +0000 UTC m=+1538.491352149" watchObservedRunningTime="2026-11-03 01:44:19.069251668 +0000 UTC m=+1538.494858131"
Nov 03 01:44:19.968084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:19.968051    5508 generic.go:356] "Generic (PLEG): container finished" podID="25245a3a-7fc5-440a-bc25-756977548b2d" containerID="baf9b204e6c2fa1fd7284a0c596dd804c838a449dfc4f8cd67ace1b6d5f3b31c" exitCode=0
Nov 03 01:44:19.968486 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:19.968132    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp" event={"ID":"25245a3a-7fc5-440a-bc25-756977548b2d","Type":"ContainerDied","Data":"baf9b204e6c2fa1fd7284a0c596dd804c838a449dfc4f8cd67ace1b6d5f3b31c"}
Nov 03 01:44:21.148827 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:21.148799    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"
Nov 03 01:44:21.223430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:21.223403    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-xks-ca\") pod \"25245a3a-7fc5-440a-bc25-756977548b2d\" (UID: \"25245a3a-7fc5-440a-bc25-756977548b2d\") "
Nov 03 01:44:21.223731 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:21.223711    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-openshift-ca\") pod \"25245a3a-7fc5-440a-bc25-756977548b2d\" (UID: \"25245a3a-7fc5-440a-bc25-756977548b2d\") "
Nov 03 01:44:21.223807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:21.223794    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25245a3a-7fc5-440a-bc25-756977548b2d-kube-api-access-vkmsm\" (UniqueName: \"kubernetes.io/projected/25245a3a-7fc5-440a-bc25-756977548b2d-kube-api-access-vkmsm\") pod \"25245a3a-7fc5-440a-bc25-756977548b2d\" (UID: \"25245a3a-7fc5-440a-bc25-756977548b2d\") "
Nov 03 01:44:21.224070 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:21.224048    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-openshift-ca" pod "25245a3a-7fc5-440a-bc25-756977548b2d" (UID: "25245a3a-7fc5-440a-bc25-756977548b2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:44:21.224416 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:21.224398    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:44:21.225504 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:21.225482    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-xks-ca" pod "25245a3a-7fc5-440a-bc25-756977548b2d" (UID: "25245a3a-7fc5-440a-bc25-756977548b2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:44:21.225793 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:21.225773    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25245a3a-7fc5-440a-bc25-756977548b2d-kube-api-access-vkmsm" pod "25245a3a-7fc5-440a-bc25-756977548b2d" (UID: "25245a3a-7fc5-440a-bc25-756977548b2d"). InnerVolumeSpecName "kube-api-access-vkmsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:44:21.326184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:21.326154    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkmsm\" (UniqueName: \"kubernetes.io/projected/25245a3a-7fc5-440a-bc25-756977548b2d-kube-api-access-vkmsm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:44:21.326184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:21.326184    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25245a3a-7fc5-440a-bc25-756977548b2d-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:44:21.990023 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:21.989990    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"
Nov 03 01:44:21.990023 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:21.990006    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp" event={"ID":"25245a3a-7fc5-440a-bc25-756977548b2d","Type":"ContainerDied","Data":"38824f8aecfb6e4fb276900e22a80576046dcd3c875888d00e664e77606255d4"}
Nov 03 01:44:21.990288 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:21.990051    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38824f8aecfb6e4fb276900e22a80576046dcd3c875888d00e664e77606255d4"
Nov 03 01:44:22.552689 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:22.552649    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx" podUID="59e1dcd4-ac74-4991-a93e-546e3f5e1231" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:44:24.046979 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:24.046946    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"]
Nov 03 01:44:24.093387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:24.093330    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b388413e-7rvzr"]
Nov 03 01:44:24.702998 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:24.702966    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0488b37d-35ed-4e73-9c03-4f7f733fe73e" path="/var/lib/kubelet/pods/0488b37d-35ed-4e73-9c03-4f7f733fe73e/volumes"
Nov 03 01:44:25.360619 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:25.360585    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"]
Nov 03 01:44:25.367334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:25.367296    5508 cpu_manager.go:414] "containerMap: removing container" podUID="25245a3a-7fc5-440a-bc25-756977548b2d" containerName="revoke-keys"
Nov 03 01:44:25.367584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:25.367567    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25245a3a-7fc5-440a-bc25-756977548b2d" containerName="revoke-keys"
Nov 03 01:44:25.368049 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:25.368029    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="25245a3a-7fc5-440a-bc25-756977548b2d" containerName="revoke-keys"
Nov 03 01:44:25.725515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:25.725483    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"
Nov 03 01:44:25.727270 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:25.727244    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"]
Nov 03 01:44:25.825665 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:25.825636    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4952b1\""
Nov 03 01:44:25.825909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:25.825894    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4952b1-dockercfg-w4r69\""
Nov 03 01:44:25.901225 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:25.901191    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpccw\" (UniqueName: \"kubernetes.io/projected/636315c8-c8fd-44b4-9ae5-5052b1b0af27-kube-api-access-xpccw\") pod \"payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs\" (UID: \"636315c8-c8fd-44b4-9ae5-5052b1b0af27\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"
Nov 03 01:44:25.901559 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:25.901532    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636315c8-c8fd-44b4-9ae5-5052b1b0af27-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs\" (UID: \"636315c8-c8fd-44b4-9ae5-5052b1b0af27\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"
Nov 03 01:44:26.003174 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:26.003098    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpccw\" (UniqueName: \"kubernetes.io/projected/636315c8-c8fd-44b4-9ae5-5052b1b0af27-kube-api-access-xpccw\") pod \"payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs\" (UID: \"636315c8-c8fd-44b4-9ae5-5052b1b0af27\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"
Nov 03 01:44:26.003312 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:26.003219    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636315c8-c8fd-44b4-9ae5-5052b1b0af27-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs\" (UID: \"636315c8-c8fd-44b4-9ae5-5052b1b0af27\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"
Nov 03 01:44:26.003919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:26.003900    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636315c8-c8fd-44b4-9ae5-5052b1b0af27-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs\" (UID: \"636315c8-c8fd-44b4-9ae5-5052b1b0af27\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"
Nov 03 01:44:26.060595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:26.060567    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpccw\" (UniqueName: \"kubernetes.io/projected/636315c8-c8fd-44b4-9ae5-5052b1b0af27-kube-api-access-xpccw\") pod \"payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs\" (UID: \"636315c8-c8fd-44b4-9ae5-5052b1b0af27\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"
Nov 03 01:44:26.167424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:26.167391    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"
Nov 03 01:44:26.550616 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:26.550581    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"]
Nov 03 01:44:26.555801 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:44:26.555769    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636315c8_c8fd_44b4_9ae5_5052b1b0af27.slice/crio-265840442cc15b24de2edeebae0da7792fb49b5cdc95309891f95d18b1030cca WatchSource:0}: Error finding container 265840442cc15b24de2edeebae0da7792fb49b5cdc95309891f95d18b1030cca: Status 404 returned error can't find the container with id 265840442cc15b24de2edeebae0da7792fb49b5cdc95309891f95d18b1030cca
Nov 03 01:44:27.068453 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:27.068418    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"]
Nov 03 01:44:27.119994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:27.119961    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0bc4711-f87g7"]
Nov 03 01:44:27.129984 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:27.129949    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs" event={"ID":"636315c8-c8fd-44b4-9ae5-5052b1b0af27","Type":"ContainerStarted","Data":"c4d70704128f7c8e9932536f5917bfa14bdcd3a76575bf861f2cb306c7f0ee95"}
Nov 03 01:44:27.130254 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:27.130232    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs" event={"ID":"636315c8-c8fd-44b4-9ae5-5052b1b0af27","Type":"ContainerStarted","Data":"265840442cc15b24de2edeebae0da7792fb49b5cdc95309891f95d18b1030cca"}
Nov 03 01:44:27.131974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:27.131954    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"
Nov 03 01:44:27.147803 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:27.147770    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:44:27.241849 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:27.241801    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs" podStartSLOduration=2.241786792 podStartE2EDuration="2.241786792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:27.240017558 +0000 UTC m=+1546.665624050" watchObservedRunningTime="2026-11-03 01:44:27.241786792 +0000 UTC m=+1546.667393257"
Nov 03 01:44:28.693594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:28.693560    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69356caf-0473-482d-8254-592b6f445e0f" path="/var/lib/kubelet/pods/69356caf-0473-482d-8254-592b6f445e0f/volumes"
Nov 03 01:44:29.106422 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:29.103396    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"]
Nov 03 01:44:29.111752 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:29.111720    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fb19f13-rwf8c"]
Nov 03 01:44:30.695204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:30.695166    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1" path="/var/lib/kubelet/pods/0133c8a1-8a8e-4e46-9fca-bf7fcfa242f1/volumes"
Nov 03 01:44:31.046108 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:31.046032    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"]
Nov 03 01:44:31.092169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:31.092135    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26506e6f-nsx49"]
Nov 03 01:44:32.568502 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:32.568465    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx" podUID="59e1dcd4-ac74-4991-a93e-546e3f5e1231" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:44:32.694860 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:32.694824    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c556e115-0844-4a98-804c-e6a3283855ce" path="/var/lib/kubelet/pods/c556e115-0844-4a98-804c-e6a3283855ce/volumes"
Nov 03 01:44:35.246454 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:35.246419    5508 generic.go:356] "Generic (PLEG): container finished" podID="59e1dcd4-ac74-4991-a93e-546e3f5e1231" containerID="2d2ffa070f2daa4d1864aca1838bd37798fae8ba5daf676b2f8d9f4e98800682" exitCode=0
Nov 03 01:44:35.246856 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:35.246708    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx" event={"ID":"59e1dcd4-ac74-4991-a93e-546e3f5e1231","Type":"ContainerDied","Data":"2d2ffa070f2daa4d1864aca1838bd37798fae8ba5daf676b2f8d9f4e98800682"}
Nov 03 01:44:35.460382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:35.460334    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"
Nov 03 01:44:35.619114 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:35.619076    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59e1dcd4-ac74-4991-a93e-546e3f5e1231-kube-api-access-d2qx9\" (UniqueName: \"kubernetes.io/projected/59e1dcd4-ac74-4991-a93e-546e3f5e1231-kube-api-access-d2qx9\") pod \"59e1dcd4-ac74-4991-a93e-546e3f5e1231\" (UID: \"59e1dcd4-ac74-4991-a93e-546e3f5e1231\") "
Nov 03 01:44:35.619332 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:35.619214    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59e1dcd4-ac74-4991-a93e-546e3f5e1231-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e1dcd4-ac74-4991-a93e-546e3f5e1231-plugins-config-volume\") pod \"59e1dcd4-ac74-4991-a93e-546e3f5e1231\" (UID: \"59e1dcd4-ac74-4991-a93e-546e3f5e1231\") "
Nov 03 01:44:35.620245 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:35.620218    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e1dcd4-ac74-4991-a93e-546e3f5e1231-plugins-config-volume" pod "59e1dcd4-ac74-4991-a93e-546e3f5e1231" (UID: "59e1dcd4-ac74-4991-a93e-546e3f5e1231"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:44:35.622307 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:35.622279    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e1dcd4-ac74-4991-a93e-546e3f5e1231-kube-api-access-d2qx9" pod "59e1dcd4-ac74-4991-a93e-546e3f5e1231" (UID: "59e1dcd4-ac74-4991-a93e-546e3f5e1231"). InnerVolumeSpecName "kube-api-access-d2qx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:44:35.721358 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:35.721251    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2qx9\" (UniqueName: \"kubernetes.io/projected/59e1dcd4-ac74-4991-a93e-546e3f5e1231-kube-api-access-d2qx9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:44:35.721358 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:35.721299    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e1dcd4-ac74-4991-a93e-546e3f5e1231-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:44:36.262369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:36.262327    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"
Nov 03 01:44:36.262958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:36.262334    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx" event={"ID":"59e1dcd4-ac74-4991-a93e-546e3f5e1231","Type":"ContainerDied","Data":"3d5e782b524a66cf0cbf5a1d025850b939f46943b40c33c20826dff013d0cda6"}
Nov 03 01:44:36.262958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:36.262478    5508 scope.go:122] "RemoveContainer" containerID="2d2ffa070f2daa4d1864aca1838bd37798fae8ba5daf676b2f8d9f4e98800682"
Nov 03 01:44:36.425731 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:36.425678    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"]
Nov 03 01:44:36.472075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:36.472032    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-hzcqx"]
Nov 03 01:44:36.697942 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:36.697908    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59e1dcd4-ac74-4991-a93e-546e3f5e1231" path="/var/lib/kubelet/pods/59e1dcd4-ac74-4991-a93e-546e3f5e1231/volumes"
Nov 03 01:44:37.275539 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:37.275513    5508 generic.go:356] "Generic (PLEG): container finished" podID="916e646a-6298-47da-a869-a863a7c410fc" containerID="368a31001287facf53ada9a06e66ae576581db3f0d1627395416ad4f393644e6" exitCode=0
Nov 03 01:44:37.275928 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:37.275582    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" event={"ID":"916e646a-6298-47da-a869-a863a7c410fc","Type":"ContainerDied","Data":"368a31001287facf53ada9a06e66ae576581db3f0d1627395416ad4f393644e6"}
Nov 03 01:44:37.276036 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:37.276015    5508 scope.go:122] "RemoveContainer" containerID="368a31001287facf53ada9a06e66ae576581db3f0d1627395416ad4f393644e6"
Nov 03 01:44:37.277730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:37.277705    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:44:37.277812 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:37.277771    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:44:37.277850 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:37.277826    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:44:38.321372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:38.321017    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" event={"ID":"916e646a-6298-47da-a869-a863a7c410fc","Type":"ContainerStarted","Data":"7418960d236921a254f8e659a62ddd6b5a69a9c69f793205450fb9a3e31bb4da"}
Nov 03 01:44:38.983449 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:38.973739    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:44:38.983449 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:38.973863    5508 patch_prober.go:29] interesting pod/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 01:44:38.983449 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:38.973913    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" podUID="916e646a-6298-47da-a869-a863a7c410fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 01:44:39.196423 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:39.196394    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"
Nov 03 01:44:39.973903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:39.973844    5508 patch_prober.go:29] interesting pod/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 01:44:39.974474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:39.973928    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" podUID="916e646a-6298-47da-a869-a863a7c410fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 01:44:40.973443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:40.970800    5508 patch_prober.go:29] interesting pod/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 01:44:40.973443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:40.970873    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" podUID="916e646a-6298-47da-a869-a863a7c410fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 01:44:41.087372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:41.087004    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"]
Nov 03 01:44:41.096372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:41.096161    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9645045f-svvqt"]
Nov 03 01:44:41.915772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:41.915742    5508 scope.go:122] "RemoveContainer" containerID="c80d8f77649378fce0c8218c4e0b797290fcb1200ec5fd5fb26fbc381733dba4"
Nov 03 01:44:41.928014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:41.927990    5508 scope.go:122] "RemoveContainer" containerID="00c22be3cde21736ae6040edb87ca3444387ae5c475ec370ba1981a484eedc11"
Nov 03 01:44:41.940455 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:41.940428    5508 scope.go:122] "RemoveContainer" containerID="49280a814147b0a0239539bf17d488d3001d346751ee54ac47deb29ed94b0548"
Nov 03 01:44:41.980160 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:41.980120    5508 patch_prober.go:29] interesting pod/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 01:44:41.980425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:41.980393    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" podUID="916e646a-6298-47da-a869-a863a7c410fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 01:44:41.982460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:41.982441    5508 scope.go:122] "RemoveContainer" containerID="9041b079d5b2333ffb09e0fa173b762c63ed147d4f333345087988620863a577"
Nov 03 01:44:41.993731 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:41.993707    5508 scope.go:122] "RemoveContainer" containerID="0467a80b09d045b84d525338df7bc48ba08f1a21ccec30dcf9c03acee2841a50"
Nov 03 01:44:42.697533 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:42.697495    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9164b080-801e-41b3-bcca-5b7d42ca6204" path="/var/lib/kubelet/pods/9164b080-801e-41b3-bcca-5b7d42ca6204/volumes"
Nov 03 01:44:42.971433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:42.970789    5508 patch_prober.go:29] interesting pod/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 01:44:42.971433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:42.970873    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" podUID="916e646a-6298-47da-a869-a863a7c410fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 01:44:43.980374 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:44:43.977557    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:44:44.006424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:44.000698    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:44:44.006424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:44.002943    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:44:44.024245 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:44.022289    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:44:45.046846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.046807    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"]
Nov 03 01:44:45.092367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.092322    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e9ea64f-pwn4f"]
Nov 03 01:44:45.118760 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.118726    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"]
Nov 03 01:44:45.119427 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.119402    5508 cpu_manager.go:414] "containerMap: removing container" podUID="59e1dcd4-ac74-4991-a93e-546e3f5e1231" containerName="payload-processing"
Nov 03 01:44:45.119427 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.119421    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59e1dcd4-ac74-4991-a93e-546e3f5e1231" containerName="payload-processing"
Nov 03 01:44:45.119622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.119610    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="59e1dcd4-ac74-4991-a93e-546e3f5e1231" containerName="payload-processing"
Nov 03 01:44:45.125508 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.125487    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.182112 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.182074    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-certs\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.182405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.182213    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9tx9\" (UniqueName: \"kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-kube-api-access-v9tx9\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.182405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.182257    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-data\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.182405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.182343    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-socket\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.182616 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.182407    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-credential-socket\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.182616 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.182443    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-istio-token\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.182718 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.182616    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f930409c-7286-4a06-bb5f-93015858b97e-istiod-ca-cert\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.182775 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.182743    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f930409c-7286-4a06-bb5f-93015858b97e-istio-podinfo\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.182830 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.182810    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-envoy\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.222781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.222745    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-118857-openshift-default-dockercfg-thss5\""
Nov 03 01:44:45.242681 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.242646    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"]
Nov 03 01:44:45.285430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.285388    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f930409c-7286-4a06-bb5f-93015858b97e-istio-podinfo\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.285775 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.285521    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-envoy\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.285775 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.285562    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-certs\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.285775 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.285622    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9tx9\" (UniqueName: \"kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-kube-api-access-v9tx9\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.285775 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.285649    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-data\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.285775 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.285729    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-socket\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.286239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.285778    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-credential-socket\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.286239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.285810    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-istio-token\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.286239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.285974    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f930409c-7286-4a06-bb5f-93015858b97e-istiod-ca-cert\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.287096 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.287071    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f930409c-7286-4a06-bb5f-93015858b97e-istiod-ca-cert\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.287954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.287926    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-socket\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.288119 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.287980    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-certs\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.288224 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.288125    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-data\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.288404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.288209    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-credential-socket\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.290389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.290341    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f930409c-7286-4a06-bb5f-93015858b97e-istio-podinfo\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.290509 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.290477    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-envoy\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.346859 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.346788    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-istio-token\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.347314 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.347294    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9tx9\" (UniqueName: \"kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-kube-api-access-v9tx9\") pod \"e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") " pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.442687 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.441022    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:45.949223 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.949180    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"]
Nov 03 01:44:45.954221 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:44:45.954188    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf930409c_7286_4a06_bb5f_93015858b97e.slice/crio-12111cd2d7307a2c485b73749d3430049206da77d3d181164746b4c16d3e8b0f WatchSource:0}: Error finding container 12111cd2d7307a2c485b73749d3430049206da77d3d181164746b4c16d3e8b0f: Status 404 returned error can't find the container with id 12111cd2d7307a2c485b73749d3430049206da77d3d181164746b4c16d3e8b0f
Nov 03 01:44:45.956978 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.956943    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:44:45.957072 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.957016    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:44:45.957072 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:45.957050    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:44:46.549271 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:46.549224    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b" event={"ID":"f930409c-7286-4a06-bb5f-93015858b97e","Type":"ContainerStarted","Data":"482a99fc36dbee9778bb283835a122cf1dd20ef208186f85d2297de03c2df9de"}
Nov 03 01:44:46.549769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:46.549318    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b" event={"ID":"f930409c-7286-4a06-bb5f-93015858b97e","Type":"ContainerStarted","Data":"12111cd2d7307a2c485b73749d3430049206da77d3d181164746b4c16d3e8b0f"}
Nov 03 01:44:46.663869 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:46.663814    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b" podStartSLOduration=1.6637990839999999 podStartE2EDuration="1.663799084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:46.661996697 +0000 UTC m=+1566.087603179" watchObservedRunningTime="2026-11-03 01:44:46.663799084 +0000 UTC m=+1566.089405548"
Nov 03 01:44:46.694570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:46.694540    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="259fb926-d52a-42c5-8e29-ccd173550559" path="/var/lib/kubelet/pods/259fb926-d52a-42c5-8e29-ccd173550559/volumes"
Nov 03 01:44:47.441839 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:47.441794    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:47.448486 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:47.448457    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:47.558983 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:47.558921    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:47.560089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:47.560068    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:44:49.047016 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:49.046979    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"]
Nov 03 01:44:49.062469 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:49.062326    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5067c679-nzcfp"]
Nov 03 01:44:50.693535 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:50.693496    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe508df-0678-4535-9d11-506497d147cf" path="/var/lib/kubelet/pods/0fe508df-0678-4535-9d11-506497d147cf/volumes"
Nov 03 01:44:53.322380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:53.321444    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"]
Nov 03 01:44:53.633651 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:53.630917    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"]
Nov 03 01:44:53.633651 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:53.631025    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"
Nov 03 01:44:53.732604 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:53.732567    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-118857\""
Nov 03 01:44:53.732979 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:53.732919    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-118857-dockercfg-jhcc9\""
Nov 03 01:44:53.737643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:53.737608    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xh4k\" (UniqueName: \"kubernetes.io/projected/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-kube-api-access-7xh4k\") pod \"payload-processing-e2e-api-b-118857-858d9f85-x57bn\" (UID: \"03b5ab3a-3ae0-4313-9619-42fdc2ca9682\") " pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"
Nov 03 01:44:53.761417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:53.761381    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-plugins-config-volume\") pod \"payload-processing-e2e-api-b-118857-858d9f85-x57bn\" (UID: \"03b5ab3a-3ae0-4313-9619-42fdc2ca9682\") " pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"
Nov 03 01:44:53.870375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:53.867967    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xh4k\" (UniqueName: \"kubernetes.io/projected/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-kube-api-access-7xh4k\") pod \"payload-processing-e2e-api-b-118857-858d9f85-x57bn\" (UID: \"03b5ab3a-3ae0-4313-9619-42fdc2ca9682\") " pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"
Nov 03 01:44:53.870375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:53.868049    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-plugins-config-volume\") pod \"payload-processing-e2e-api-b-118857-858d9f85-x57bn\" (UID: \"03b5ab3a-3ae0-4313-9619-42fdc2ca9682\") " pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"
Nov 03 01:44:53.870375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:53.869019    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-plugins-config-volume\") pod \"payload-processing-e2e-api-b-118857-858d9f85-x57bn\" (UID: \"03b5ab3a-3ae0-4313-9619-42fdc2ca9682\") " pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"
Nov 03 01:44:53.957923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:53.957656    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xh4k\" (UniqueName: \"kubernetes.io/projected/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-kube-api-access-7xh4k\") pod \"payload-processing-e2e-api-b-118857-858d9f85-x57bn\" (UID: \"03b5ab3a-3ae0-4313-9619-42fdc2ca9682\") " pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"
Nov 03 01:44:54.049931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:54.049890    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"]
Nov 03 01:44:54.056773 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:54.056736    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06c2f5cb-lrqth"]
Nov 03 01:44:54.250807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:54.250702    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"
Nov 03 01:44:54.660129 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:54.660074    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"]
Nov 03 01:44:54.694004 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:44:54.693960    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b5ab3a_3ae0_4313_9619_42fdc2ca9682.slice/crio-743b8267a7b405076958c1e6c43cd5465be2431e2b05e0bf423012949064d2db WatchSource:0}: Error finding container 743b8267a7b405076958c1e6c43cd5465be2431e2b05e0bf423012949064d2db: Status 404 returned error can't find the container with id 743b8267a7b405076958c1e6c43cd5465be2431e2b05e0bf423012949064d2db
Nov 03 01:44:54.710867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:54.710840    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04370439-8d45-472c-8af2-c69eca905ea4" path="/var/lib/kubelet/pods/04370439-8d45-472c-8af2-c69eca905ea4/volumes"
Nov 03 01:44:55.735157 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:55.735122    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn" event={"ID":"03b5ab3a-3ae0-4313-9619-42fdc2ca9682","Type":"ContainerStarted","Data":"d3de9abd26031a7bfa91db2e061dfe9e005d40d229a216766699a9640c2c9065"}
Nov 03 01:44:55.735827 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:55.735801    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"
Nov 03 01:44:55.735888 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:55.735840    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn" event={"ID":"03b5ab3a-3ae0-4313-9619-42fdc2ca9682","Type":"ContainerStarted","Data":"743b8267a7b405076958c1e6c43cd5465be2431e2b05e0bf423012949064d2db"}
Nov 03 01:44:55.843620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:55.843562    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn" podStartSLOduration=2.84354417 podStartE2EDuration="2.84354417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:55.843444262 +0000 UTC m=+1575.269050727" watchObservedRunningTime="2026-11-03 01:44:55.84354417 +0000 UTC m=+1575.269150635"
Nov 03 01:44:57.047502 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:57.047464    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"]
Nov 03 01:44:57.106037 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:57.095154    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c92765ce-nxbg4"]
Nov 03 01:44:58.694708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:44:58.694663    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="825d4108-3d57-4c2c-8831-ceb66e8e4ad8" path="/var/lib/kubelet/pods/825d4108-3d57-4c2c-8831-ceb66e8e4ad8/volumes"
Nov 03 01:45:00.047045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.047008    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"]
Nov 03 01:45:00.092685 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.092647    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8812863-c2dns"]
Nov 03 01:45:00.164093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.164062    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"]
Nov 03 01:45:00.168761 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.168743    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"
Nov 03 01:45:00.170871 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.170844    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"]
Nov 03 01:45:00.176231 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.176206    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"]
Nov 03 01:45:00.176381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.176334    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"
Nov 03 01:45:00.181220 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.181200    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"]
Nov 03 01:45:00.181584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.181563    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"
Nov 03 01:45:00.186187 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.186167    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"
Nov 03 01:45:00.186539 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.186445    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"]
Nov 03 01:45:00.197447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.195855    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"]
Nov 03 01:45:00.197447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.196072    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"
Nov 03 01:45:00.201323 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.201293    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"]
Nov 03 01:45:00.201540 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.201521    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"
Nov 03 01:45:00.206858 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.206834    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"
Nov 03 01:45:00.208312 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.208289    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp\" (UID: \"dc9a7667-192a-40aa-b2cb-f0fc140aa55f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"
Nov 03 01:45:00.208431 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.208410    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px\" (UID: \"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"
Nov 03 01:45:00.208510 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.208494    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6d69\" (UniqueName: \"kubernetes.io/projected/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-kube-api-access-g6d69\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp\" (UID: \"dc9a7667-192a-40aa-b2cb-f0fc140aa55f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"
Nov 03 01:45:00.208564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.208535    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wc44\" (UniqueName: \"kubernetes.io/projected/4b49ab06-b764-450f-8049-f2d092c03414-kube-api-access-8wc44\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl\" (UID: \"4b49ab06-b764-450f-8049-f2d092c03414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"
Nov 03 01:45:00.208690 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.208675    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab2b38d-28f9-4676-85fb-4f315928c129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9\" (UID: \"5ab2b38d-28f9-4676-85fb-4f315928c129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"
Nov 03 01:45:00.208725 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.208704    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b49ab06-b764-450f-8049-f2d092c03414-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl\" (UID: \"4b49ab06-b764-450f-8049-f2d092c03414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"
Nov 03 01:45:00.208784 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.208769    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnr64\" (UniqueName: \"kubernetes.io/projected/5ab2b38d-28f9-4676-85fb-4f315928c129-kube-api-access-rnr64\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9\" (UID: \"5ab2b38d-28f9-4676-85fb-4f315928c129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"
Nov 03 01:45:00.208881 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.208865    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs76v\" (UniqueName: \"kubernetes.io/projected/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-kube-api-access-cs76v\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px\" (UID: \"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"
Nov 03 01:45:00.218158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.218128    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"]
Nov 03 01:45:00.218774 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.218754    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:45:00.220297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.220274    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:45:00.220996 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.220976    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:45:00.227166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.227141    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"
Nov 03 01:45:00.234563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.234506    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"]
Nov 03 01:45:00.245639 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.240325    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"]
Nov 03 01:45:00.260774 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.260726    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"]
Nov 03 01:45:00.303420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.302419    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"]
Nov 03 01:45:00.311949 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.311737    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"]
Nov 03 01:45:00.314055 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.314021    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f039b14-e4cc-44bf-a808-63a8c543e128-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h\" (UID: \"3f039b14-e4cc-44bf-a808-63a8c543e128\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"
Nov 03 01:45:00.314324 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.314299    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh\" (UID: \"4bde3b86-9dd2-482f-9598-7e42c44ef5ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"
Nov 03 01:45:00.314520 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.314489    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp\" (UID: \"dc9a7667-192a-40aa-b2cb-f0fc140aa55f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"
Nov 03 01:45:00.318953 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.318831    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px\" (UID: \"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"
Nov 03 01:45:00.385868 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.385840    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4td92\" (UniqueName: \"kubernetes.io/projected/b40d7637-4656-4d80-ba61-aaa61ba14ccf-kube-api-access-4td92\") pod \"maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz\" (UID: \"b40d7637-4656-4d80-ba61-aaa61ba14ccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"
Nov 03 01:45:00.386129 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.386100    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6d69\" (UniqueName: \"kubernetes.io/projected/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-kube-api-access-g6d69\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp\" (UID: \"dc9a7667-192a-40aa-b2cb-f0fc140aa55f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"
Nov 03 01:45:00.396614 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.385705    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp\" (UID: \"dc9a7667-192a-40aa-b2cb-f0fc140aa55f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"
Nov 03 01:45:00.405411 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.327337    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px\" (UID: \"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"
Nov 03 01:45:00.413385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.363505    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"]
Nov 03 01:45:00.512986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.396895    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wc44\" (UniqueName: \"kubernetes.io/projected/4b49ab06-b764-450f-8049-f2d092c03414-kube-api-access-8wc44\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl\" (UID: \"4b49ab06-b764-450f-8049-f2d092c03414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"
Nov 03 01:45:00.513543 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.513519    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmcph\" (UniqueName: \"kubernetes.io/projected/3f039b14-e4cc-44bf-a808-63a8c543e128-kube-api-access-zmcph\") pod \"maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h\" (UID: \"3f039b14-e4cc-44bf-a808-63a8c543e128\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"
Nov 03 01:45:00.513863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.513845    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab2b38d-28f9-4676-85fb-4f315928c129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9\" (UID: \"5ab2b38d-28f9-4676-85fb-4f315928c129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"
Nov 03 01:45:00.514750 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.514722    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b49ab06-b764-450f-8049-f2d092c03414-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl\" (UID: \"4b49ab06-b764-450f-8049-f2d092c03414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"
Nov 03 01:45:00.601496 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.519510    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40d7637-4656-4d80-ba61-aaa61ba14ccf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz\" (UID: \"b40d7637-4656-4d80-ba61-aaa61ba14ccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"
Nov 03 01:45:00.607550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.607509    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnr64\" (UniqueName: \"kubernetes.io/projected/5ab2b38d-28f9-4676-85fb-4f315928c129-kube-api-access-rnr64\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9\" (UID: \"5ab2b38d-28f9-4676-85fb-4f315928c129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"
Nov 03 01:45:00.607781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.607763    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvm86\" (UniqueName: \"kubernetes.io/projected/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-kube-api-access-zvm86\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh\" (UID: \"4bde3b86-9dd2-482f-9598-7e42c44ef5ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"
Nov 03 01:45:00.623404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.610600    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7p5j\" (UniqueName: \"kubernetes.io/projected/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-kube-api-access-z7p5j\") pod \"maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph\" (UID: \"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"
Nov 03 01:45:00.623801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.623767    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs76v\" (UniqueName: \"kubernetes.io/projected/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-kube-api-access-cs76v\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px\" (UID: \"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"
Nov 03 01:45:00.634407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.627414    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph\" (UID: \"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"
Nov 03 01:45:00.653692 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.514685    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"]
Nov 03 01:45:00.653990 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.653965    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"]
Nov 03 01:45:00.654147 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.519457    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab2b38d-28f9-4676-85fb-4f315928c129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9\" (UID: \"5ab2b38d-28f9-4676-85fb-4f315928c129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"
Nov 03 01:45:00.654343 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.519099    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b49ab06-b764-450f-8049-f2d092c03414-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl\" (UID: \"4b49ab06-b764-450f-8049-f2d092c03414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"
Nov 03 01:45:00.684376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.684325    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6d69\" (UniqueName: \"kubernetes.io/projected/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-kube-api-access-g6d69\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp\" (UID: \"dc9a7667-192a-40aa-b2cb-f0fc140aa55f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"
Nov 03 01:45:00.687184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.687118    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wc44\" (UniqueName: \"kubernetes.io/projected/4b49ab06-b764-450f-8049-f2d092c03414-kube-api-access-8wc44\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl\" (UID: \"4b49ab06-b764-450f-8049-f2d092c03414\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"
Nov 03 01:45:00.694239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.694206    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnr64\" (UniqueName: \"kubernetes.io/projected/5ab2b38d-28f9-4676-85fb-4f315928c129-kube-api-access-rnr64\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9\" (UID: \"5ab2b38d-28f9-4676-85fb-4f315928c129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"
Nov 03 01:45:00.705849 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.705815    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c0eb6d-2862-4fca-8d0f-a1cca074a892" path="/var/lib/kubelet/pods/63c0eb6d-2862-4fca-8d0f-a1cca074a892/volumes"
Nov 03 01:45:00.735390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.735323    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs76v\" (UniqueName: \"kubernetes.io/projected/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-kube-api-access-cs76v\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px\" (UID: \"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"
Nov 03 01:45:00.756335 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.756291    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f039b14-e4cc-44bf-a808-63a8c543e128-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h\" (UID: \"3f039b14-e4cc-44bf-a808-63a8c543e128\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"
Nov 03 01:45:00.756560 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.756396    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh\" (UID: \"4bde3b86-9dd2-482f-9598-7e42c44ef5ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"
Nov 03 01:45:00.756560 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.756500    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4td92\" (UniqueName: \"kubernetes.io/projected/b40d7637-4656-4d80-ba61-aaa61ba14ccf-kube-api-access-4td92\") pod \"maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz\" (UID: \"b40d7637-4656-4d80-ba61-aaa61ba14ccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"
Nov 03 01:45:00.756690 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.756581    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmcph\" (UniqueName: \"kubernetes.io/projected/3f039b14-e4cc-44bf-a808-63a8c543e128-kube-api-access-zmcph\") pod \"maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h\" (UID: \"3f039b14-e4cc-44bf-a808-63a8c543e128\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"
Nov 03 01:45:00.756690 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.756648    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40d7637-4656-4d80-ba61-aaa61ba14ccf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz\" (UID: \"b40d7637-4656-4d80-ba61-aaa61ba14ccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"
Nov 03 01:45:00.756796 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.756687    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvm86\" (UniqueName: \"kubernetes.io/projected/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-kube-api-access-zvm86\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh\" (UID: \"4bde3b86-9dd2-482f-9598-7e42c44ef5ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"
Nov 03 01:45:00.756796 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.756742    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7p5j\" (UniqueName: \"kubernetes.io/projected/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-kube-api-access-z7p5j\") pod \"maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph\" (UID: \"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"
Nov 03 01:45:00.756796 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.756792    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph\" (UID: \"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"
Nov 03 01:45:00.757775 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.757743    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph\" (UID: \"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"
Nov 03 01:45:00.758483 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.758458    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f039b14-e4cc-44bf-a808-63a8c543e128-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h\" (UID: \"3f039b14-e4cc-44bf-a808-63a8c543e128\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"
Nov 03 01:45:00.760622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.759098    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh\" (UID: \"4bde3b86-9dd2-482f-9598-7e42c44ef5ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"
Nov 03 01:45:00.761453 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.761430    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40d7637-4656-4d80-ba61-aaa61ba14ccf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz\" (UID: \"b40d7637-4656-4d80-ba61-aaa61ba14ccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"
Nov 03 01:45:00.787981 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.787948    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"
Nov 03 01:45:00.800880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.800846    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"
Nov 03 01:45:00.810691 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.810654    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"
Nov 03 01:45:00.820678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.819411    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4td92\" (UniqueName: \"kubernetes.io/projected/b40d7637-4656-4d80-ba61-aaa61ba14ccf-kube-api-access-4td92\") pod \"maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz\" (UID: \"b40d7637-4656-4d80-ba61-aaa61ba14ccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"
Nov 03 01:45:00.820678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.820048    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmcph\" (UniqueName: \"kubernetes.io/projected/3f039b14-e4cc-44bf-a808-63a8c543e128-kube-api-access-zmcph\") pod \"maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h\" (UID: \"3f039b14-e4cc-44bf-a808-63a8c543e128\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"
Nov 03 01:45:00.820678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.820609    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7p5j\" (UniqueName: \"kubernetes.io/projected/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-kube-api-access-z7p5j\") pod \"maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph\" (UID: \"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"
Nov 03 01:45:00.821637 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.821605    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvm86\" (UniqueName: \"kubernetes.io/projected/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-kube-api-access-zvm86\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh\" (UID: \"4bde3b86-9dd2-482f-9598-7e42c44ef5ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"
Nov 03 01:45:00.824715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.824686    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"
Nov 03 01:45:00.880702 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.880433    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"
Nov 03 01:45:00.885816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.885763    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"
Nov 03 01:45:00.962293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:00.962260    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"
Nov 03 01:45:01.026820 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.026603    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"
Nov 03 01:45:01.221549 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.221516    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"]
Nov 03 01:45:01.226887 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:45:01.226814    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab2b38d_28f9_4676_85fb_4f315928c129.slice/crio-64f7e4802956adf4d629e7696cd125365a021f0647c1cffac0fadc1081b3ae75 WatchSource:0}: Error finding container 64f7e4802956adf4d629e7696cd125365a021f0647c1cffac0fadc1081b3ae75: Status 404 returned error can't find the container with id 64f7e4802956adf4d629e7696cd125365a021f0647c1cffac0fadc1081b3ae75
Nov 03 01:45:01.554382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.551995    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"]
Nov 03 01:45:01.641371 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:45:01.635202    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9a7667_192a_40aa_b2cb_f0fc140aa55f.slice/crio-f62d6fa98a476cb8f5140db108d1b62d3b9c0ee1f9d746ddf49a7d3cb2b46554 WatchSource:0}: Error finding container f62d6fa98a476cb8f5140db108d1b62d3b9c0ee1f9d746ddf49a7d3cb2b46554: Status 404 returned error can't find the container with id f62d6fa98a476cb8f5140db108d1b62d3b9c0ee1f9d746ddf49a7d3cb2b46554
Nov 03 01:45:01.675596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.675500    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"]
Nov 03 01:45:01.719146 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.716931    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"]
Nov 03 01:45:01.721415 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:45:01.721385    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1e5080_28b5_409a_ac9d_63bd5bf7ea42.slice/crio-bf89395072cf7e6f3a1743790ffe619419a7c88d0168ba65c6c72a7e82981bbf WatchSource:0}: Error finding container bf89395072cf7e6f3a1743790ffe619419a7c88d0168ba65c6c72a7e82981bbf: Status 404 returned error can't find the container with id bf89395072cf7e6f3a1743790ffe619419a7c88d0168ba65c6c72a7e82981bbf
Nov 03 01:45:01.722500 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:45:01.722465    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f039b14_e4cc_44bf_a808_63a8c543e128.slice/crio-51534bafd0adf4c6cab894cfc868892a1ada1ef6068a5e876f1e6a47a67f1a98 WatchSource:0}: Error finding container 51534bafd0adf4c6cab894cfc868892a1ada1ef6068a5e876f1e6a47a67f1a98: Status 404 returned error can't find the container with id 51534bafd0adf4c6cab894cfc868892a1ada1ef6068a5e876f1e6a47a67f1a98
Nov 03 01:45:01.723858 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.723840    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"]
Nov 03 01:45:01.730735 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.730711    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"]
Nov 03 01:45:01.768262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.768235    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"]
Nov 03 01:45:01.775153 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:45:01.775115    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d39f5d_61a3_424c_85a2_0d26f2ce1ac3.slice/crio-3e0545c1ecd5612536151e16c29e1310391f6a83bad37a0681ed6a10d6646a2f WatchSource:0}: Error finding container 3e0545c1ecd5612536151e16c29e1310391f6a83bad37a0681ed6a10d6646a2f: Status 404 returned error can't find the container with id 3e0545c1ecd5612536151e16c29e1310391f6a83bad37a0681ed6a10d6646a2f
Nov 03 01:45:01.789790 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.789467    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh" event={"ID":"4bde3b86-9dd2-482f-9598-7e42c44ef5ff","Type":"ContainerStarted","Data":"2a9604811bc858d35e7d2e1147da7ba9c2d757fa958f3a943e7d3f5c54965fa0"}
Nov 03 01:45:01.791194 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.791168    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9" event={"ID":"5ab2b38d-28f9-4676-85fb-4f315928c129","Type":"ContainerStarted","Data":"720452448e29cb4b7d3042024732bad00dbe718604184d5632d1635063cdd408"}
Nov 03 01:45:01.791308 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.791211    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9" event={"ID":"5ab2b38d-28f9-4676-85fb-4f315928c129","Type":"ContainerStarted","Data":"64f7e4802956adf4d629e7696cd125365a021f0647c1cffac0fadc1081b3ae75"}
Nov 03 01:45:01.793889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.793839    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px" event={"ID":"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3","Type":"ContainerStarted","Data":"3e0545c1ecd5612536151e16c29e1310391f6a83bad37a0681ed6a10d6646a2f"}
Nov 03 01:45:01.795221 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.795202    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl" event={"ID":"4b49ab06-b764-450f-8049-f2d092c03414","Type":"ContainerStarted","Data":"d3e33129900b2082fda61b917f26b22b08a6ff40c69b3288b4dcf754b6407e27"}
Nov 03 01:45:01.798060 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.798034    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h" event={"ID":"3f039b14-e4cc-44bf-a808-63a8c543e128","Type":"ContainerStarted","Data":"51534bafd0adf4c6cab894cfc868892a1ada1ef6068a5e876f1e6a47a67f1a98"}
Nov 03 01:45:01.799324 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.799297    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp" event={"ID":"dc9a7667-192a-40aa-b2cb-f0fc140aa55f","Type":"ContainerStarted","Data":"f62d6fa98a476cb8f5140db108d1b62d3b9c0ee1f9d746ddf49a7d3cb2b46554"}
Nov 03 01:45:01.800659 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.800637    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph" event={"ID":"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42","Type":"ContainerStarted","Data":"bf89395072cf7e6f3a1743790ffe619419a7c88d0168ba65c6c72a7e82981bbf"}
Nov 03 01:45:01.874696 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.872260    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"]
Nov 03 01:45:01.905462 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:01.905252    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9" podStartSLOduration=1.905228071 podStartE2EDuration="1.905228071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:01.901878947 +0000 UTC m=+1581.327485424" watchObservedRunningTime="2026-11-03 01:45:01.905228071 +0000 UTC m=+1581.330834538"
Nov 03 01:45:02.781782 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:02.781546    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"]
Nov 03 01:45:02.812181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:02.812142    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px" event={"ID":"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3","Type":"ContainerStarted","Data":"9883b6b0ed051aa311dfe4f9bfba8e45617796e7540c364eb34da63532b6e374"}
Nov 03 01:45:02.813834 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:02.813805    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl" event={"ID":"4b49ab06-b764-450f-8049-f2d092c03414","Type":"ContainerStarted","Data":"763bcb3710f494cf7d47ca62dad4dd11909555f6999bde7f9da70cf51e358da2"}
Nov 03 01:45:02.815362 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:02.815326    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz" event={"ID":"b40d7637-4656-4d80-ba61-aaa61ba14ccf","Type":"ContainerStarted","Data":"2fbb7de9ed7c82479f19efa8e689a07a758aa708a509d091e459be029778a5c0"}
Nov 03 01:45:02.815458 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:02.815395    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz" event={"ID":"b40d7637-4656-4d80-ba61-aaa61ba14ccf","Type":"ContainerStarted","Data":"5719c332aae4ee4b1c798ce46641452ac8d946ab82667bfc79e1a42b6f3f8621"}
Nov 03 01:45:02.816969 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:02.816949    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h" event={"ID":"3f039b14-e4cc-44bf-a808-63a8c543e128","Type":"ContainerStarted","Data":"cb00080e251e011727212e07baa363e112587078a2685b951ac3780cb9851b74"}
Nov 03 01:45:02.818795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:02.818776    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp" event={"ID":"dc9a7667-192a-40aa-b2cb-f0fc140aa55f","Type":"ContainerStarted","Data":"8e8ea5ce2b43df581beb5ce20d5d89f852b729d0b18c8767c565e884902fa311"}
Nov 03 01:45:02.820757 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:02.820629    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph" event={"ID":"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42","Type":"ContainerStarted","Data":"e4d4965a82042a3e422e5021b64af8096c152154bbbe20b477a114ffc4db8ff8"}
Nov 03 01:45:02.822225 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:02.822200    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh" event={"ID":"4bde3b86-9dd2-482f-9598-7e42c44ef5ff","Type":"ContainerStarted","Data":"f4a4d851f1756a7073f2a08349e5cd87a28e7d8b2534d364f0560e1536cdba24"}
Nov 03 01:45:02.823592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:02.823567    5508 generic.go:356] "Generic (PLEG): container finished" podID="5ab2b38d-28f9-4676-85fb-4f315928c129" containerID="720452448e29cb4b7d3042024732bad00dbe718604184d5632d1635063cdd408" exitCode=60
Nov 03 01:45:02.823663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:02.823609    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9" event={"ID":"5ab2b38d-28f9-4676-85fb-4f315928c129","Type":"ContainerDied","Data":"720452448e29cb4b7d3042024732bad00dbe718604184d5632d1635063cdd408"}
Nov 03 01:45:02.823832 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:02.823819    5508 scope.go:122] "RemoveContainer" containerID="720452448e29cb4b7d3042024732bad00dbe718604184d5632d1635063cdd408"
Nov 03 01:45:02.874723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:02.874638    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"]
Nov 03 01:45:03.012470 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.012414    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp" podStartSLOduration=3.012400831 podStartE2EDuration="3.012400831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:03.003148983 +0000 UTC m=+1582.428755459" watchObservedRunningTime="2026-11-03 01:45:03.012400831 +0000 UTC m=+1582.438007295"
Nov 03 01:45:03.113563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.113463    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz" podStartSLOduration=3.113446568 podStartE2EDuration="3.113446568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:03.111115148 +0000 UTC m=+1582.536721631" watchObservedRunningTime="2026-11-03 01:45:03.113446568 +0000 UTC m=+1582.539053029"
Nov 03 01:45:03.590732 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.590688    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph" podStartSLOduration=3.590672793 podStartE2EDuration="3.590672793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:03.301618769 +0000 UTC m=+1582.727225236" watchObservedRunningTime="2026-11-03 01:45:03.590672793 +0000 UTC m=+1583.016279307"
Nov 03 01:45:03.596204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.596143    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl" podStartSLOduration=3.596125583 podStartE2EDuration="3.596125583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:03.589402888 +0000 UTC m=+1583.015009375" watchObservedRunningTime="2026-11-03 01:45:03.596125583 +0000 UTC m=+1583.021732046"
Nov 03 01:45:03.809196 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.809131    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px" podStartSLOduration=3.809114097 podStartE2EDuration="3.809114097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:03.802880871 +0000 UTC m=+1583.228487345" watchObservedRunningTime="2026-11-03 01:45:03.809114097 +0000 UTC m=+1583.234720561"
Nov 03 01:45:03.845986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.845893    5508 generic.go:356] "Generic (PLEG): container finished" podID="4b49ab06-b764-450f-8049-f2d092c03414" containerID="763bcb3710f494cf7d47ca62dad4dd11909555f6999bde7f9da70cf51e358da2" exitCode=60
Nov 03 01:45:03.846160 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.846016    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl" event={"ID":"4b49ab06-b764-450f-8049-f2d092c03414","Type":"ContainerDied","Data":"763bcb3710f494cf7d47ca62dad4dd11909555f6999bde7f9da70cf51e358da2"}
Nov 03 01:45:03.846431 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.846412    5508 scope.go:122] "RemoveContainer" containerID="763bcb3710f494cf7d47ca62dad4dd11909555f6999bde7f9da70cf51e358da2"
Nov 03 01:45:03.881310 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.881263    5508 generic.go:356] "Generic (PLEG): container finished" podID="b40d7637-4656-4d80-ba61-aaa61ba14ccf" containerID="2fbb7de9ed7c82479f19efa8e689a07a758aa708a509d091e459be029778a5c0" exitCode=60
Nov 03 01:45:03.881589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.881384    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz" event={"ID":"b40d7637-4656-4d80-ba61-aaa61ba14ccf","Type":"ContainerDied","Data":"2fbb7de9ed7c82479f19efa8e689a07a758aa708a509d091e459be029778a5c0"}
Nov 03 01:45:03.881922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.881903    5508 scope.go:122] "RemoveContainer" containerID="2fbb7de9ed7c82479f19efa8e689a07a758aa708a509d091e459be029778a5c0"
Nov 03 01:45:03.884283 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.884261    5508 generic.go:356] "Generic (PLEG): container finished" podID="3f039b14-e4cc-44bf-a808-63a8c543e128" containerID="cb00080e251e011727212e07baa363e112587078a2685b951ac3780cb9851b74" exitCode=60
Nov 03 01:45:03.884416 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.884388    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h" event={"ID":"3f039b14-e4cc-44bf-a808-63a8c543e128","Type":"ContainerDied","Data":"cb00080e251e011727212e07baa363e112587078a2685b951ac3780cb9851b74"}
Nov 03 01:45:03.884720 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.884703    5508 scope.go:122] "RemoveContainer" containerID="cb00080e251e011727212e07baa363e112587078a2685b951ac3780cb9851b74"
Nov 03 01:45:03.886823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.886798    5508 generic.go:356] "Generic (PLEG): container finished" podID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f" containerID="8e8ea5ce2b43df581beb5ce20d5d89f852b729d0b18c8767c565e884902fa311" exitCode=60
Nov 03 01:45:03.886910 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.886881    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp" event={"ID":"dc9a7667-192a-40aa-b2cb-f0fc140aa55f","Type":"ContainerDied","Data":"8e8ea5ce2b43df581beb5ce20d5d89f852b729d0b18c8767c565e884902fa311"}
Nov 03 01:45:03.887169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.887153    5508 scope.go:122] "RemoveContainer" containerID="8e8ea5ce2b43df581beb5ce20d5d89f852b729d0b18c8767c565e884902fa311"
Nov 03 01:45:03.894601 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.894579    5508 generic.go:356] "Generic (PLEG): container finished" podID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" containerID="e4d4965a82042a3e422e5021b64af8096c152154bbbe20b477a114ffc4db8ff8" exitCode=60
Nov 03 01:45:03.894695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.894671    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph" event={"ID":"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42","Type":"ContainerDied","Data":"e4d4965a82042a3e422e5021b64af8096c152154bbbe20b477a114ffc4db8ff8"}
Nov 03 01:45:03.896619 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.896597    5508 generic.go:356] "Generic (PLEG): container finished" podID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff" containerID="f4a4d851f1756a7073f2a08349e5cd87a28e7d8b2534d364f0560e1536cdba24" exitCode=60
Nov 03 01:45:03.896684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.896674    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh" event={"ID":"4bde3b86-9dd2-482f-9598-7e42c44ef5ff","Type":"ContainerDied","Data":"f4a4d851f1756a7073f2a08349e5cd87a28e7d8b2534d364f0560e1536cdba24"}
Nov 03 01:45:03.897085 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.897069    5508 scope.go:122] "RemoveContainer" containerID="f4a4d851f1756a7073f2a08349e5cd87a28e7d8b2534d364f0560e1536cdba24"
Nov 03 01:45:03.900136 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.900112    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9" event={"ID":"5ab2b38d-28f9-4676-85fb-4f315928c129","Type":"ContainerStarted","Data":"7a3953ebbc5bde67043b31c28a5d1c20370ed4e1c2fbb246df483b5731105207"}
Nov 03 01:45:03.903457 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.903389    5508 scope.go:122] "RemoveContainer" containerID="e4d4965a82042a3e422e5021b64af8096c152154bbbe20b477a114ffc4db8ff8"
Nov 03 01:45:03.906993 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.905987    5508 generic.go:356] "Generic (PLEG): container finished" podID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" containerID="9883b6b0ed051aa311dfe4f9bfba8e45617796e7540c364eb34da63532b6e374" exitCode=60
Nov 03 01:45:03.906993 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.906034    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px" event={"ID":"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3","Type":"ContainerDied","Data":"9883b6b0ed051aa311dfe4f9bfba8e45617796e7540c364eb34da63532b6e374"}
Nov 03 01:45:03.906993 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.906342    5508 scope.go:122] "RemoveContainer" containerID="9883b6b0ed051aa311dfe4f9bfba8e45617796e7540c364eb34da63532b6e374"
Nov 03 01:45:03.941649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:03.941596    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh" podStartSLOduration=3.941575697 podStartE2EDuration="3.941575697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:03.940467483 +0000 UTC m=+1583.366073966" watchObservedRunningTime="2026-11-03 01:45:03.941575697 +0000 UTC m=+1583.367182162"
Nov 03 01:45:04.056373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:04.053921    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h" podStartSLOduration=4.053901441 podStartE2EDuration="4.053901441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:04.051802441 +0000 UTC m=+1583.477408930" watchObservedRunningTime="2026-11-03 01:45:04.053901441 +0000 UTC m=+1583.479507908"
Nov 03 01:45:04.918615 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:04.918584    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h" event={"ID":"3f039b14-e4cc-44bf-a808-63a8c543e128","Type":"ContainerStarted","Data":"42b5a067369449b250dbf38ae7b849deb0e1dbd19d0ac21b91aea0b8a0657c4d"}
Nov 03 01:45:04.924372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:04.924335    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp" event={"ID":"dc9a7667-192a-40aa-b2cb-f0fc140aa55f","Type":"ContainerStarted","Data":"3febde81dfcd3b52c707d80e6fd3e0824e897cf571339a6767d96339ed960d11"}
Nov 03 01:45:04.926734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:04.926704    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh" event={"ID":"4bde3b86-9dd2-482f-9598-7e42c44ef5ff","Type":"ContainerStarted","Data":"bccec52f3fe3ab409470a4767d42bcf0b06d60c52a488b1d0e6a7c62b2cfec52"}
Nov 03 01:45:04.928544 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:04.928523    5508 generic.go:356] "Generic (PLEG): container finished" podID="5ab2b38d-28f9-4676-85fb-4f315928c129" containerID="7a3953ebbc5bde67043b31c28a5d1c20370ed4e1c2fbb246df483b5731105207" exitCode=60
Nov 03 01:45:04.928693 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:04.928596    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9" event={"ID":"5ab2b38d-28f9-4676-85fb-4f315928c129","Type":"ContainerDied","Data":"7a3953ebbc5bde67043b31c28a5d1c20370ed4e1c2fbb246df483b5731105207"}
Nov 03 01:45:04.928781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:04.928722    5508 scope.go:122] "RemoveContainer" containerID="720452448e29cb4b7d3042024732bad00dbe718604184d5632d1635063cdd408"
Nov 03 01:45:04.928833 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:04.928787    5508 scope.go:122] "RemoveContainer" containerID="7a3953ebbc5bde67043b31c28a5d1c20370ed4e1c2fbb246df483b5731105207"
Nov 03 01:45:04.929130 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:04.929101    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9_odh-ai-gateway-infra(5ab2b38d-28f9-4676-85fb-4f315928c129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9" podUID="5ab2b38d-28f9-4676-85fb-4f315928c129"
Nov 03 01:45:04.932231 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:04.932210    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px" event={"ID":"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3","Type":"ContainerStarted","Data":"89abef8144446b6c401ff48beb6346e45fc2a203cb158845a856b5524e13a0be"}
Nov 03 01:45:04.934385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:04.934321    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl" event={"ID":"4b49ab06-b764-450f-8049-f2d092c03414","Type":"ContainerStarted","Data":"2636b46325a428743cc99166538a4aa7f007da5c2a868a99a616b1e0d374746b"}
Nov 03 01:45:04.936382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:04.936328    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz" event={"ID":"b40d7637-4656-4d80-ba61-aaa61ba14ccf","Type":"ContainerStarted","Data":"afcfb9241a4bd5675dcd26ca89221dc3c5a698cde24c589fd5a146a2a031ff2c"}
Nov 03 01:45:05.962374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.954977    5508 scope.go:122] "RemoveContainer" containerID="89abef8144446b6c401ff48beb6346e45fc2a203cb158845a856b5524e13a0be"
Nov 03 01:45:05.962374 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:05.955479    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px_odh-ai-gateway-infra(f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px" podUID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3"
Nov 03 01:45:05.962374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.954415    5508 generic.go:356] "Generic (PLEG): container finished" podID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" containerID="89abef8144446b6c401ff48beb6346e45fc2a203cb158845a856b5524e13a0be" exitCode=60
Nov 03 01:45:05.962374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.955800    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px" event={"ID":"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3","Type":"ContainerDied","Data":"89abef8144446b6c401ff48beb6346e45fc2a203cb158845a856b5524e13a0be"}
Nov 03 01:45:05.962374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.955892    5508 scope.go:122] "RemoveContainer" containerID="9883b6b0ed051aa311dfe4f9bfba8e45617796e7540c364eb34da63532b6e374"
Nov 03 01:45:05.986272 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.986231    5508 generic.go:356] "Generic (PLEG): container finished" podID="4b49ab06-b764-450f-8049-f2d092c03414" containerID="2636b46325a428743cc99166538a4aa7f007da5c2a868a99a616b1e0d374746b" exitCode=60
Nov 03 01:45:05.989858 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.989663    5508 scope.go:122] "RemoveContainer" containerID="2636b46325a428743cc99166538a4aa7f007da5c2a868a99a616b1e0d374746b"
Nov 03 01:45:05.989994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.989726    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl" event={"ID":"4b49ab06-b764-450f-8049-f2d092c03414","Type":"ContainerDied","Data":"2636b46325a428743cc99166538a4aa7f007da5c2a868a99a616b1e0d374746b"}
Nov 03 01:45:05.990895 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:05.990319    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl_odh-ai-gateway-infra(4b49ab06-b764-450f-8049-f2d092c03414)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl" podUID="4b49ab06-b764-450f-8049-f2d092c03414"
Nov 03 01:45:05.992906 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.992882    5508 generic.go:356] "Generic (PLEG): container finished" podID="b40d7637-4656-4d80-ba61-aaa61ba14ccf" containerID="afcfb9241a4bd5675dcd26ca89221dc3c5a698cde24c589fd5a146a2a031ff2c" exitCode=60
Nov 03 01:45:05.993031 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.992971    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz" event={"ID":"b40d7637-4656-4d80-ba61-aaa61ba14ccf","Type":"ContainerDied","Data":"afcfb9241a4bd5675dcd26ca89221dc3c5a698cde24c589fd5a146a2a031ff2c"}
Nov 03 01:45:05.993432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.993411    5508 scope.go:122] "RemoveContainer" containerID="afcfb9241a4bd5675dcd26ca89221dc3c5a698cde24c589fd5a146a2a031ff2c"
Nov 03 01:45:05.993973 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:05.993835    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz_odh-ai-gateway-infra(b40d7637-4656-4d80-ba61-aaa61ba14ccf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz" podUID="b40d7637-4656-4d80-ba61-aaa61ba14ccf"
Nov 03 01:45:05.994145 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.994088    5508 scope.go:122] "RemoveContainer" containerID="763bcb3710f494cf7d47ca62dad4dd11909555f6999bde7f9da70cf51e358da2"
Nov 03 01:45:05.996146 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.996123    5508 generic.go:356] "Generic (PLEG): container finished" podID="3f039b14-e4cc-44bf-a808-63a8c543e128" containerID="42b5a067369449b250dbf38ae7b849deb0e1dbd19d0ac21b91aea0b8a0657c4d" exitCode=60
Nov 03 01:45:05.996241 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.996202    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h" event={"ID":"3f039b14-e4cc-44bf-a808-63a8c543e128","Type":"ContainerDied","Data":"42b5a067369449b250dbf38ae7b849deb0e1dbd19d0ac21b91aea0b8a0657c4d"}
Nov 03 01:45:05.996714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.996698    5508 scope.go:122] "RemoveContainer" containerID="42b5a067369449b250dbf38ae7b849deb0e1dbd19d0ac21b91aea0b8a0657c4d"
Nov 03 01:45:05.997185 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:05.997161    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h_odh-ai-gateway-infra(3f039b14-e4cc-44bf-a808-63a8c543e128)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h" podUID="3f039b14-e4cc-44bf-a808-63a8c543e128"
Nov 03 01:45:05.998513 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.998479    5508 generic.go:356] "Generic (PLEG): container finished" podID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f" containerID="3febde81dfcd3b52c707d80e6fd3e0824e897cf571339a6767d96339ed960d11" exitCode=60
Nov 03 01:45:05.998622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.998561    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp" event={"ID":"dc9a7667-192a-40aa-b2cb-f0fc140aa55f","Type":"ContainerDied","Data":"3febde81dfcd3b52c707d80e6fd3e0824e897cf571339a6767d96339ed960d11"}
Nov 03 01:45:05.998833 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:05.998819    5508 scope.go:122] "RemoveContainer" containerID="3febde81dfcd3b52c707d80e6fd3e0824e897cf571339a6767d96339ed960d11"
Nov 03 01:45:05.999181 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:05.999157    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp_odh-ai-gateway-infra(dc9a7667-192a-40aa-b2cb-f0fc140aa55f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp" podUID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f"
Nov 03 01:45:06.000833 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:06.000808    5508 generic.go:356] "Generic (PLEG): container finished" podID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" containerID="54fb6fc70b14106b950e6e71ec655d690d019d9c1d94cf5d4083fdb9b6a2cb74" exitCode=60
Nov 03 01:45:06.000926 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:06.000900    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph" event={"ID":"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42","Type":"ContainerDied","Data":"54fb6fc70b14106b950e6e71ec655d690d019d9c1d94cf5d4083fdb9b6a2cb74"}
Nov 03 01:45:06.001168 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:06.001152    5508 scope.go:122] "RemoveContainer" containerID="54fb6fc70b14106b950e6e71ec655d690d019d9c1d94cf5d4083fdb9b6a2cb74"
Nov 03 01:45:06.001542 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:06.001521    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph_odh-ai-gateway-infra(eb1e5080-28b5-409a-ac9d-63bd5bf7ea42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph" podUID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42"
Nov 03 01:45:06.004147 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:06.004126    5508 generic.go:356] "Generic (PLEG): container finished" podID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff" containerID="bccec52f3fe3ab409470a4767d42bcf0b06d60c52a488b1d0e6a7c62b2cfec52" exitCode=60
Nov 03 01:45:06.004247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:06.004208    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh" event={"ID":"4bde3b86-9dd2-482f-9598-7e42c44ef5ff","Type":"ContainerDied","Data":"bccec52f3fe3ab409470a4767d42bcf0b06d60c52a488b1d0e6a7c62b2cfec52"}
Nov 03 01:45:06.004501 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:06.004483    5508 scope.go:122] "RemoveContainer" containerID="bccec52f3fe3ab409470a4767d42bcf0b06d60c52a488b1d0e6a7c62b2cfec52"
Nov 03 01:45:06.004819 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:06.004789    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh_odh-ai-gateway-infra(4bde3b86-9dd2-482f-9598-7e42c44ef5ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh" podUID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff"
Nov 03 01:45:06.006567 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:06.006550    5508 scope.go:122] "RemoveContainer" containerID="7a3953ebbc5bde67043b31c28a5d1c20370ed4e1c2fbb246df483b5731105207"
Nov 03 01:45:06.006811 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:06.006784    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9_odh-ai-gateway-infra(5ab2b38d-28f9-4676-85fb-4f315928c129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9" podUID="5ab2b38d-28f9-4676-85fb-4f315928c129"
Nov 03 01:45:06.010658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:06.010625    5508 scope.go:122] "RemoveContainer" containerID="2fbb7de9ed7c82479f19efa8e689a07a758aa708a509d091e459be029778a5c0"
Nov 03 01:45:06.025250 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:06.025219    5508 scope.go:122] "RemoveContainer" containerID="cb00080e251e011727212e07baa363e112587078a2685b951ac3780cb9851b74"
Nov 03 01:45:06.042654 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:06.042629    5508 scope.go:122] "RemoveContainer" containerID="8e8ea5ce2b43df581beb5ce20d5d89f852b729d0b18c8767c565e884902fa311"
Nov 03 01:45:06.093623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:06.093600    5508 scope.go:122] "RemoveContainer" containerID="e4d4965a82042a3e422e5021b64af8096c152154bbbe20b477a114ffc4db8ff8"
Nov 03 01:45:06.109244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:06.109214    5508 scope.go:122] "RemoveContainer" containerID="f4a4d851f1756a7073f2a08349e5cd87a28e7d8b2534d364f0560e1536cdba24"
Nov 03 01:45:07.017335 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:07.017307    5508 scope.go:122] "RemoveContainer" containerID="42b5a067369449b250dbf38ae7b849deb0e1dbd19d0ac21b91aea0b8a0657c4d"
Nov 03 01:45:07.017922 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:07.017805    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h_odh-ai-gateway-infra(3f039b14-e4cc-44bf-a808-63a8c543e128)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h" podUID="3f039b14-e4cc-44bf-a808-63a8c543e128"
Nov 03 01:45:07.019205 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:07.019180    5508 scope.go:122] "RemoveContainer" containerID="3febde81dfcd3b52c707d80e6fd3e0824e897cf571339a6767d96339ed960d11"
Nov 03 01:45:07.019601 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:07.019580    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp_odh-ai-gateway-infra(dc9a7667-192a-40aa-b2cb-f0fc140aa55f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp" podUID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f"
Nov 03 01:45:07.020880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:07.020862    5508 scope.go:122] "RemoveContainer" containerID="54fb6fc70b14106b950e6e71ec655d690d019d9c1d94cf5d4083fdb9b6a2cb74"
Nov 03 01:45:07.021194 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:07.021170    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph_odh-ai-gateway-infra(eb1e5080-28b5-409a-ac9d-63bd5bf7ea42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph" podUID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42"
Nov 03 01:45:07.022726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:07.022708    5508 scope.go:122] "RemoveContainer" containerID="bccec52f3fe3ab409470a4767d42bcf0b06d60c52a488b1d0e6a7c62b2cfec52"
Nov 03 01:45:07.023013 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:07.022994    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh_odh-ai-gateway-infra(4bde3b86-9dd2-482f-9598-7e42c44ef5ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh" podUID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff"
Nov 03 01:45:07.024360 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:07.024330    5508 scope.go:122] "RemoveContainer" containerID="89abef8144446b6c401ff48beb6346e45fc2a203cb158845a856b5524e13a0be"
Nov 03 01:45:07.024644 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:07.024622    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px_odh-ai-gateway-infra(f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px" podUID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3"
Nov 03 01:45:07.025909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:07.025893    5508 scope.go:122] "RemoveContainer" containerID="2636b46325a428743cc99166538a4aa7f007da5c2a868a99a616b1e0d374746b"
Nov 03 01:45:07.026160 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:07.026138    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl_odh-ai-gateway-infra(4b49ab06-b764-450f-8049-f2d092c03414)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl" podUID="4b49ab06-b764-450f-8049-f2d092c03414"
Nov 03 01:45:07.027460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:07.027444    5508 scope.go:122] "RemoveContainer" containerID="afcfb9241a4bd5675dcd26ca89221dc3c5a698cde24c589fd5a146a2a031ff2c"
Nov 03 01:45:07.027688 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:07.027667    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz_odh-ai-gateway-infra(b40d7637-4656-4d80-ba61-aaa61ba14ccf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz" podUID="b40d7637-4656-4d80-ba61-aaa61ba14ccf"
Nov 03 01:45:07.779915 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:07.779881    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"
Nov 03 01:45:07.785475 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:07.785408    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p" podUID="99a5b548-b10a-42f7-9194-b185d7408231" containerName="payload-processing" containerID="cri-o://57bc060e184be6c18c3b1f42ef014981f9838d576344d121f0162d14c9632cb4" gracePeriod=25
Nov 03 01:45:07.879444 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:07.879403    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r" podUID="69203466-b06c-468f-850e-eec3a5cb0dfb" containerName="payload-pre-processing" containerID="cri-o://5555481c265536bcc012730a6f83ea6a9db5f4d223c2bf2412bb82a6ed216527" gracePeriod=25
Nov 03 01:45:08.044076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.043977    5508 generic.go:356] "Generic (PLEG): container finished" podID="99a5b548-b10a-42f7-9194-b185d7408231" containerID="57bc060e184be6c18c3b1f42ef014981f9838d576344d121f0162d14c9632cb4" exitCode=0
Nov 03 01:45:08.044756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.044154    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p" event={"ID":"99a5b548-b10a-42f7-9194-b185d7408231","Type":"ContainerDied","Data":"57bc060e184be6c18c3b1f42ef014981f9838d576344d121f0162d14c9632cb4"}
Nov 03 01:45:08.046079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.046054    5508 generic.go:356] "Generic (PLEG): container finished" podID="69203466-b06c-468f-850e-eec3a5cb0dfb" containerID="5555481c265536bcc012730a6f83ea6a9db5f4d223c2bf2412bb82a6ed216527" exitCode=0
Nov 03 01:45:08.046305 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.046287    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r" event={"ID":"69203466-b06c-468f-850e-eec3a5cb0dfb","Type":"ContainerDied","Data":"5555481c265536bcc012730a6f83ea6a9db5f4d223c2bf2412bb82a6ed216527"}
Nov 03 01:45:08.274745 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.274715    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"
Nov 03 01:45:08.364280 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.364244    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"
Nov 03 01:45:08.374503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.374462    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a5b548-b10a-42f7-9194-b185d7408231-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a5b548-b10a-42f7-9194-b185d7408231-plugins-config-volume\") pod \"99a5b548-b10a-42f7-9194-b185d7408231\" (UID: \"99a5b548-b10a-42f7-9194-b185d7408231\") "
Nov 03 01:45:08.374811 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.374792    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a5b548-b10a-42f7-9194-b185d7408231-kube-api-access-z6k64\" (UniqueName: \"kubernetes.io/projected/99a5b548-b10a-42f7-9194-b185d7408231-kube-api-access-z6k64\") pod \"99a5b548-b10a-42f7-9194-b185d7408231\" (UID: \"99a5b548-b10a-42f7-9194-b185d7408231\") "
Nov 03 01:45:08.374899 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.374832    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a5b548-b10a-42f7-9194-b185d7408231-plugins-config-volume" pod "99a5b548-b10a-42f7-9194-b185d7408231" (UID: "99a5b548-b10a-42f7-9194-b185d7408231"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:08.375564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.375530    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a5b548-b10a-42f7-9194-b185d7408231-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:08.377251 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.377217    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a5b548-b10a-42f7-9194-b185d7408231-kube-api-access-z6k64" pod "99a5b548-b10a-42f7-9194-b185d7408231" (UID: "99a5b548-b10a-42f7-9194-b185d7408231"). InnerVolumeSpecName "kube-api-access-z6k64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:08.478473 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.476775    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69203466-b06c-468f-850e-eec3a5cb0dfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69203466-b06c-468f-850e-eec3a5cb0dfb-plugins-config-volume\") pod \"69203466-b06c-468f-850e-eec3a5cb0dfb\" (UID: \"69203466-b06c-468f-850e-eec3a5cb0dfb\") "
Nov 03 01:45:08.478473 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.476859    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69203466-b06c-468f-850e-eec3a5cb0dfb-kube-api-access-85txs\" (UniqueName: \"kubernetes.io/projected/69203466-b06c-468f-850e-eec3a5cb0dfb-kube-api-access-85txs\") pod \"69203466-b06c-468f-850e-eec3a5cb0dfb\" (UID: \"69203466-b06c-468f-850e-eec3a5cb0dfb\") "
Nov 03 01:45:08.478473 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.478112    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6k64\" (UniqueName: \"kubernetes.io/projected/99a5b548-b10a-42f7-9194-b185d7408231-kube-api-access-z6k64\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:08.478924 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.478887    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69203466-b06c-468f-850e-eec3a5cb0dfb-plugins-config-volume" pod "69203466-b06c-468f-850e-eec3a5cb0dfb" (UID: "69203466-b06c-468f-850e-eec3a5cb0dfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:08.487409 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.487332    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69203466-b06c-468f-850e-eec3a5cb0dfb-kube-api-access-85txs" pod "69203466-b06c-468f-850e-eec3a5cb0dfb" (UID: "69203466-b06c-468f-850e-eec3a5cb0dfb"). InnerVolumeSpecName "kube-api-access-85txs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:08.580006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.579915    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69203466-b06c-468f-850e-eec3a5cb0dfb-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:08.580006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:08.579961    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85txs\" (UniqueName: \"kubernetes.io/projected/69203466-b06c-468f-850e-eec3a5cb0dfb-kube-api-access-85txs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:09.057181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:09.057151    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"
Nov 03 01:45:09.057181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:09.057160    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p" event={"ID":"99a5b548-b10a-42f7-9194-b185d7408231","Type":"ContainerDied","Data":"88ddd2b774666a45cf6aa4eabc3ff47e561963ed3913ea1f9b7fc2f659adaa61"}
Nov 03 01:45:09.057780 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:09.057253    5508 scope.go:122] "RemoveContainer" containerID="57bc060e184be6c18c3b1f42ef014981f9838d576344d121f0162d14c9632cb4"
Nov 03 01:45:09.059261 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:09.059231    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r" event={"ID":"69203466-b06c-468f-850e-eec3a5cb0dfb","Type":"ContainerDied","Data":"4698dba44592c6007e4f5d9cbb58b67e792b19748d39176341f91417eb67732f"}
Nov 03 01:45:09.059427 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:09.059271    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"
Nov 03 01:45:09.071012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:09.070983    5508 scope.go:122] "RemoveContainer" containerID="5555481c265536bcc012730a6f83ea6a9db5f4d223c2bf2412bb82a6ed216527"
Nov 03 01:45:09.214529 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:09.214479    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"]
Nov 03 01:45:09.221737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:09.221709    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-51bcf3-78c98f9cc9-xzs8p"]
Nov 03 01:45:09.323594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:09.323503    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"]
Nov 03 01:45:09.328127 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:09.328085    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-51bcf3-5cb56d44c9-vpf8r"]
Nov 03 01:45:10.053369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:10.053310    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"]
Nov 03 01:45:10.059823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:10.059791    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8740514f-mnp9q"]
Nov 03 01:45:10.697405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:10.697330    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69203466-b06c-468f-850e-eec3a5cb0dfb" path="/var/lib/kubelet/pods/69203466-b06c-468f-850e-eec3a5cb0dfb/volumes"
Nov 03 01:45:10.697934 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:10.697912    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a5b548-b10a-42f7-9194-b185d7408231" path="/var/lib/kubelet/pods/99a5b548-b10a-42f7-9194-b185d7408231/volumes"
Nov 03 01:45:10.698439 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:10.698420    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df935e9d-c812-4ac5-b899-c575f946827c" path="/var/lib/kubelet/pods/df935e9d-c812-4ac5-b899-c575f946827c/volumes"
Nov 03 01:45:12.246115 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.246078    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"]
Nov 03 01:45:12.246907 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.246870    5508 cpu_manager.go:414] "containerMap: removing container" podUID="69203466-b06c-468f-850e-eec3a5cb0dfb" containerName="payload-pre-processing"
Nov 03 01:45:12.246907 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.246888    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69203466-b06c-468f-850e-eec3a5cb0dfb" containerName="payload-pre-processing"
Nov 03 01:45:12.247051 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.246915    5508 cpu_manager.go:414] "containerMap: removing container" podUID="99a5b548-b10a-42f7-9194-b185d7408231" containerName="payload-processing"
Nov 03 01:45:12.247051 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.246921    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a5b548-b10a-42f7-9194-b185d7408231" containerName="payload-processing"
Nov 03 01:45:12.247051 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.247030    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="69203466-b06c-468f-850e-eec3a5cb0dfb" containerName="payload-pre-processing"
Nov 03 01:45:12.247051 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.247043    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a5b548-b10a-42f7-9194-b185d7408231" containerName="payload-processing"
Nov 03 01:45:12.250755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.250732    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.339495 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.339456    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.341923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.339527    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmdp5\" (UniqueName: \"kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-kube-api-access-hmdp5\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.341923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.339715    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.341923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.339757    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.341923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.339786    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.341923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.339877    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.341923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.339939    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.341923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.339958    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.341923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.339997    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.442128 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.442074    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.442323 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.442245    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.442323 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.442293    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.442487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.442363    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.442487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.442414    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.442487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.442463    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmdp5\" (UniqueName: \"kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-kube-api-access-hmdp5\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.442657 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.442611    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.442708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.442667    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.442708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.442699    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-data\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.442818 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.442712    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.443802 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.443659    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-credential-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.444148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.444101    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-socket\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.444376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.444271    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istiod-ca-cert\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.444781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.444732    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-certs\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.451714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.451656    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-envoy\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.453631 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.453551    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"]
Nov 03 01:45:12.455291 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.455251    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-podinfo\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.500645 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.500617    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-token\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.500770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.500678    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmdp5\" (UniqueName: \"kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-kube-api-access-hmdp5\") pod \"e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") " pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:12.567474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:12.567395    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:13.007774 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.007737    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"]
Nov 03 01:45:13.019964 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.019925    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:45:13.020061 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.019991    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:45:13.020061 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.020018    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:45:13.106919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.106884    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g" event={"ID":"b14432b6-e3db-4a33-aaa8-1dd9c3e96305","Type":"ContainerStarted","Data":"eb71cfa854eff391feb46bca6128de54aa7c5015491e3dec5fe954cfcc1f1d92"}
Nov 03 01:45:13.107116 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.106949    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g" event={"ID":"b14432b6-e3db-4a33-aaa8-1dd9c3e96305","Type":"ContainerStarted","Data":"edbae903cd44076b2c5899349ec4a8337c27ed325e4ab412bb53f61ea0dd2d50"}
Nov 03 01:45:13.108754 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.108731    5508 generic.go:356] "Generic (PLEG): container finished" podID="c55a9f34-b745-4306-83d9-5460d87832e5" containerID="40be624633397e27b1ce74e2571f33917b777b75e8b286edcfa494655021de62" exitCode=0
Nov 03 01:45:13.108860 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.108781    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" event={"ID":"c55a9f34-b745-4306-83d9-5460d87832e5","Type":"ContainerDied","Data":"40be624633397e27b1ce74e2571f33917b777b75e8b286edcfa494655021de62"}
Nov 03 01:45:13.109205 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.109189    5508 scope.go:122] "RemoveContainer" containerID="40be624633397e27b1ce74e2571f33917b777b75e8b286edcfa494655021de62"
Nov 03 01:45:13.110904 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.110873    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:45:13.110998 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.110951    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:45:13.110998 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.110981    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:45:13.222994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.222936    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g" podStartSLOduration=1.222918678 podStartE2EDuration="1.222918678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:13.220035253 +0000 UTC m=+1592.645641728" watchObservedRunningTime="2026-11-03 01:45:13.222918678 +0000 UTC m=+1592.648525141"
Nov 03 01:45:13.568075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.568034    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:13.828668 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:13.828578    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:45:13.900419 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:13.898989    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:45:14.119308 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:14.119217    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" event={"ID":"c55a9f34-b745-4306-83d9-5460d87832e5","Type":"ContainerStarted","Data":"6fc0ee606e99d3142d49459afdc65404baa6682d46e026e0818ddc66a3505d5c"}
Nov 03 01:45:14.594373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:14.591506    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 01:45:14.594373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:14.591578    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g" podUID="b14432b6-e3db-4a33-aaa8-1dd9c3e96305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 01:45:14.740807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:14.740774    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"]
Nov 03 01:45:15.093386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:15.089271    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:45:15.112663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:15.097487    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 01:45:15.112663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:15.097557    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 01:45:15.684380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:15.665096    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 01:45:15.684380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:15.665180    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g" podUID="b14432b6-e3db-4a33-aaa8-1dd9c3e96305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 01:45:15.966738 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:15.966655    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"
Nov 03 01:45:16.078448 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.073052    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f039b14-e4cc-44bf-a808-63a8c543e128-kube-api-access-zmcph\" (UniqueName: \"kubernetes.io/projected/3f039b14-e4cc-44bf-a808-63a8c543e128-kube-api-access-zmcph\") pod \"3f039b14-e4cc-44bf-a808-63a8c543e128\" (UID: \"3f039b14-e4cc-44bf-a808-63a8c543e128\") "
Nov 03 01:45:16.078448 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.073333    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f039b14-e4cc-44bf-a808-63a8c543e128-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f039b14-e4cc-44bf-a808-63a8c543e128-maas-api-service-ca\") pod \"3f039b14-e4cc-44bf-a808-63a8c543e128\" (UID: \"3f039b14-e4cc-44bf-a808-63a8c543e128\") "
Nov 03 01:45:16.091640 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.091597    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f039b14-e4cc-44bf-a808-63a8c543e128-maas-api-service-ca" pod "3f039b14-e4cc-44bf-a808-63a8c543e128" (UID: "3f039b14-e4cc-44bf-a808-63a8c543e128"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:16.102774 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.091605    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 01:45:16.102975 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.092144    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f039b14-e4cc-44bf-a808-63a8c543e128-kube-api-access-zmcph" pod "3f039b14-e4cc-44bf-a808-63a8c543e128" (UID: "3f039b14-e4cc-44bf-a808-63a8c543e128"). InnerVolumeSpecName "kube-api-access-zmcph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:16.103134 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.103101    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 01:45:16.205587 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.204557    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmcph\" (UniqueName: \"kubernetes.io/projected/3f039b14-e4cc-44bf-a808-63a8c543e128-kube-api-access-zmcph\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:16.205587 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.204598    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f039b14-e4cc-44bf-a808-63a8c543e128-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:16.211945 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.211749    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"
Nov 03 01:45:16.211945 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.211754    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h" event={"ID":"3f039b14-e4cc-44bf-a808-63a8c543e128","Type":"ContainerDied","Data":"51534bafd0adf4c6cab894cfc868892a1ada1ef6068a5e876f1e6a47a67f1a98"}
Nov 03 01:45:16.211945 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.211856    5508 scope.go:122] "RemoveContainer" containerID="42b5a067369449b250dbf38ae7b849deb0e1dbd19d0ac21b91aea0b8a0657c4d"
Nov 03 01:45:16.413666 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.413620    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"]
Nov 03 01:45:16.420365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.419162    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-2b9f22c1-29894505-vg28h"]
Nov 03 01:45:16.571875 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.571776    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 01:45:16.572157 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.572124    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g" podUID="b14432b6-e3db-4a33-aaa8-1dd9c3e96305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 01:45:16.795196 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.795157    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f039b14-e4cc-44bf-a808-63a8c543e128" path="/var/lib/kubelet/pods/3f039b14-e4cc-44bf-a808-63a8c543e128/volumes"
Nov 03 01:45:16.796413 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.796382    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"]
Nov 03 01:45:16.811998 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:16.811185    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2b9f22c1-kkt4t"]
Nov 03 01:45:17.089373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:17.078985    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 01:45:17.089373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:17.079044    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 01:45:17.569369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:17.568718    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 01:45:17.569369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:17.568784    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g" podUID="b14432b6-e3db-4a33-aaa8-1dd9c3e96305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 01:45:17.686455 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:17.686417    5508 scope.go:122] "RemoveContainer" containerID="54fb6fc70b14106b950e6e71ec655d690d019d9c1d94cf5d4083fdb9b6a2cb74"
Nov 03 01:45:18.104374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:18.095305    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 01:45:18.104374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:18.095384    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 01:45:18.236786 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:18.236751    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph" event={"ID":"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42","Type":"ContainerStarted","Data":"eb999efcba5a91073f2a87c3198b00a2a6f3c94f49e7922c23e57a43c0c38367"}
Nov 03 01:45:18.588375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:18.575907    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 01:45:18.588375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:18.575990    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g" podUID="b14432b6-e3db-4a33-aaa8-1dd9c3e96305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 01:45:18.694470 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:18.692039    5508 scope.go:122] "RemoveContainer" containerID="2636b46325a428743cc99166538a4aa7f007da5c2a868a99a616b1e0d374746b"
Nov 03 01:45:18.694470 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:18.692880    5508 scope.go:122] "RemoveContainer" containerID="7a3953ebbc5bde67043b31c28a5d1c20370ed4e1c2fbb246df483b5731105207"
Nov 03 01:45:18.694470 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:18.693742    5508 scope.go:122] "RemoveContainer" containerID="afcfb9241a4bd5675dcd26ca89221dc3c5a698cde24c589fd5a146a2a031ff2c"
Nov 03 01:45:18.705990 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:18.694647    5508 scope.go:122] "RemoveContainer" containerID="bccec52f3fe3ab409470a4767d42bcf0b06d60c52a488b1d0e6a7c62b2cfec52"
Nov 03 01:45:18.716467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:18.716446    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945531b5-dd6e-473b-a0c3-f54b71cd150a" path="/var/lib/kubelet/pods/945531b5-dd6e-473b-a0c3-f54b71cd150a/volumes"
Nov 03 01:45:19.107168 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:19.103572    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 01:45:19.107168 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:19.103631    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 01:45:19.316721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:19.316682    5508 generic.go:356] "Generic (PLEG): container finished" podID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" containerID="eb999efcba5a91073f2a87c3198b00a2a6f3c94f49e7922c23e57a43c0c38367" exitCode=60
Nov 03 01:45:19.316937 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:19.316915    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph" event={"ID":"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42","Type":"ContainerDied","Data":"eb999efcba5a91073f2a87c3198b00a2a6f3c94f49e7922c23e57a43c0c38367"}
Nov 03 01:45:19.317032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:19.316976    5508 scope.go:122] "RemoveContainer" containerID="54fb6fc70b14106b950e6e71ec655d690d019d9c1d94cf5d4083fdb9b6a2cb74"
Nov 03 01:45:19.317693 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:19.317595    5508 scope.go:122] "RemoveContainer" containerID="eb999efcba5a91073f2a87c3198b00a2a6f3c94f49e7922c23e57a43c0c38367"
Nov 03 01:45:19.318122 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:19.318076    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph_odh-ai-gateway-infra(eb1e5080-28b5-409a-ac9d-63bd5bf7ea42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph" podUID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42"
Nov 03 01:45:19.324319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:19.324280    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl" event={"ID":"4b49ab06-b764-450f-8049-f2d092c03414","Type":"ContainerStarted","Data":"89e33e5199359c52baadcf79509dcb3f34d507e68bb9c157b09aa5946cee218f"}
Nov 03 01:45:19.420649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:19.419159    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"]
Nov 03 01:45:19.568755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:19.567817    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 03 01:45:19.568755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:19.567879    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g" podUID="b14432b6-e3db-4a33-aaa8-1dd9c3e96305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 03 01:45:19.690459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:19.690227    5508 scope.go:122] "RemoveContainer" containerID="3febde81dfcd3b52c707d80e6fd3e0824e897cf571339a6767d96339ed960d11"
Nov 03 01:45:19.693409 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:19.692672    5508 scope.go:122] "RemoveContainer" containerID="89abef8144446b6c401ff48beb6346e45fc2a203cb158845a856b5524e13a0be"
Nov 03 01:45:20.079861 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.079832    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 01:45:20.080036 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.079907    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 01:45:20.336004 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.335967    5508 generic.go:356] "Generic (PLEG): container finished" podID="b40d7637-4656-4d80-ba61-aaa61ba14ccf" containerID="732153e7c323828b637b7555b73373d55056311d5039e33517ec4978819d24cd" exitCode=60
Nov 03 01:45:20.336534 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.336035    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz" event={"ID":"b40d7637-4656-4d80-ba61-aaa61ba14ccf","Type":"ContainerDied","Data":"732153e7c323828b637b7555b73373d55056311d5039e33517ec4978819d24cd"}
Nov 03 01:45:20.336534 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.336091    5508 scope.go:122] "RemoveContainer" containerID="afcfb9241a4bd5675dcd26ca89221dc3c5a698cde24c589fd5a146a2a031ff2c"
Nov 03 01:45:20.336634 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.336592    5508 scope.go:122] "RemoveContainer" containerID="732153e7c323828b637b7555b73373d55056311d5039e33517ec4978819d24cd"
Nov 03 01:45:20.337102 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:20.337002    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz_odh-ai-gateway-infra(b40d7637-4656-4d80-ba61-aaa61ba14ccf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz" podUID="b40d7637-4656-4d80-ba61-aaa61ba14ccf"
Nov 03 01:45:20.338467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.338440    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp" event={"ID":"dc9a7667-192a-40aa-b2cb-f0fc140aa55f","Type":"ContainerStarted","Data":"05b2121802009bce193d940a1c9bd725d5957861b598ea5248c6ca6ddb73ac5f"}
Nov 03 01:45:20.342244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.342211    5508 generic.go:356] "Generic (PLEG): container finished" podID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff" containerID="a367c2b1331a6a86b48642597267694d0d37a743e95853b675450c3a0f660195" exitCode=60
Nov 03 01:45:20.342410 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.342288    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh" event={"ID":"4bde3b86-9dd2-482f-9598-7e42c44ef5ff","Type":"ContainerDied","Data":"a367c2b1331a6a86b48642597267694d0d37a743e95853b675450c3a0f660195"}
Nov 03 01:45:20.342791 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.342770    5508 scope.go:122] "RemoveContainer" containerID="a367c2b1331a6a86b48642597267694d0d37a743e95853b675450c3a0f660195"
Nov 03 01:45:20.346220 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:20.343145    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh_odh-ai-gateway-infra(4bde3b86-9dd2-482f-9598-7e42c44ef5ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh" podUID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff"
Nov 03 01:45:20.347479 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.347452    5508 generic.go:356] "Generic (PLEG): container finished" podID="5ab2b38d-28f9-4676-85fb-4f315928c129" containerID="f2076f39b5ccef2f2c970117d1fd61080effb25c81ce5f752b7cdcb87a258af4" exitCode=60
Nov 03 01:45:20.347599 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.347582    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9" event={"ID":"5ab2b38d-28f9-4676-85fb-4f315928c129","Type":"ContainerDied","Data":"f2076f39b5ccef2f2c970117d1fd61080effb25c81ce5f752b7cdcb87a258af4"}
Nov 03 01:45:20.348009 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.347992    5508 scope.go:122] "RemoveContainer" containerID="f2076f39b5ccef2f2c970117d1fd61080effb25c81ce5f752b7cdcb87a258af4"
Nov 03 01:45:20.348394 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:20.348329    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9_odh-ai-gateway-infra(5ab2b38d-28f9-4676-85fb-4f315928c129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9" podUID="5ab2b38d-28f9-4676-85fb-4f315928c129"
Nov 03 01:45:20.350246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.350206    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px" event={"ID":"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3","Type":"ContainerStarted","Data":"503ef7bd943c5f03e4203920d07eac86cd8c1c6614987f74f86b54cbb2ee7aa5"}
Nov 03 01:45:20.354673 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.354007    5508 generic.go:356] "Generic (PLEG): container finished" podID="4b49ab06-b764-450f-8049-f2d092c03414" containerID="89e33e5199359c52baadcf79509dcb3f34d507e68bb9c157b09aa5946cee218f" exitCode=60
Nov 03 01:45:20.354673 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.354150    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl" event={"ID":"4b49ab06-b764-450f-8049-f2d092c03414","Type":"ContainerDied","Data":"89e33e5199359c52baadcf79509dcb3f34d507e68bb9c157b09aa5946cee218f"}
Nov 03 01:45:20.354851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.354750    5508 scope.go:122] "RemoveContainer" containerID="89e33e5199359c52baadcf79509dcb3f34d507e68bb9c157b09aa5946cee218f"
Nov 03 01:45:20.355318 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:20.355195    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl_odh-ai-gateway-infra(4b49ab06-b764-450f-8049-f2d092c03414)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl" podUID="4b49ab06-b764-450f-8049-f2d092c03414"
Nov 03 01:45:20.357811 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.357788    5508 scope.go:122] "RemoveContainer" containerID="bccec52f3fe3ab409470a4767d42bcf0b06d60c52a488b1d0e6a7c62b2cfec52"
Nov 03 01:45:20.379416 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.379392    5508 scope.go:122] "RemoveContainer" containerID="7a3953ebbc5bde67043b31c28a5d1c20370ed4e1c2fbb246df483b5731105207"
Nov 03 01:45:20.397724 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.397697    5508 scope.go:122] "RemoveContainer" containerID="2636b46325a428743cc99166538a4aa7f007da5c2a868a99a616b1e0d374746b"
Nov 03 01:45:20.521845 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.521818    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"
Nov 03 01:45:20.555332 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.555287    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"]
Nov 03 01:45:20.573457 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.573423    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:20.573849 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.573832    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:20.574856 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.574832    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:45:20.590800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.590766    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-maas-api-service-ca\") pod \"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42\" (UID: \"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42\") "
Nov 03 01:45:20.591154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.591134    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-kube-api-access-z7p5j\" (UniqueName: \"kubernetes.io/projected/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-kube-api-access-z7p5j\") pod \"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42\" (UID: \"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42\") "
Nov 03 01:45:20.591496 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.591472    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-maas-api-service-ca" pod "eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" (UID: "eb1e5080-28b5-409a-ac9d-63bd5bf7ea42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:20.591781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.591767    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:20.593260 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.593234    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-kube-api-access-z7p5j" pod "eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" (UID: "eb1e5080-28b5-409a-ac9d-63bd5bf7ea42"). InnerVolumeSpecName "kube-api-access-z7p5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:20.693891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:20.693850    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7p5j\" (UniqueName: \"kubernetes.io/projected/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42-kube-api-access-z7p5j\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:20.875971 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:20.875925    5508 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d39f5d_61a3_424c_85a2_0d26f2ce1ac3.slice/crio-503ef7bd943c5f03e4203920d07eac86cd8c1c6614987f74f86b54cbb2ee7aa5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d39f5d_61a3_424c_85a2_0d26f2ce1ac3.slice/crio-conmon-503ef7bd943c5f03e4203920d07eac86cd8c1c6614987f74f86b54cbb2ee7aa5.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:45:21.052701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.052608    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"]
Nov 03 01:45:21.057059 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.057028    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb86700-8p4d2"]
Nov 03 01:45:21.084226 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.084192    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:45:21.084634 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.084605    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:45:21.085608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.085586    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:45:21.371925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.371833    5508 generic.go:356] "Generic (PLEG): container finished" podID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" containerID="503ef7bd943c5f03e4203920d07eac86cd8c1c6614987f74f86b54cbb2ee7aa5" exitCode=60
Nov 03 01:45:21.371925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.371903    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px" event={"ID":"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3","Type":"ContainerDied","Data":"503ef7bd943c5f03e4203920d07eac86cd8c1c6614987f74f86b54cbb2ee7aa5"}
Nov 03 01:45:21.372472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.371983    5508 scope.go:122] "RemoveContainer" containerID="89abef8144446b6c401ff48beb6346e45fc2a203cb158845a856b5524e13a0be"
Nov 03 01:45:21.373007 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.372644    5508 scope.go:122] "RemoveContainer" containerID="503ef7bd943c5f03e4203920d07eac86cd8c1c6614987f74f86b54cbb2ee7aa5"
Nov 03 01:45:21.373261 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:21.373228    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px_odh-ai-gateway-infra(f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px" podUID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3"
Nov 03 01:45:21.388473 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.388443    5508 generic.go:356] "Generic (PLEG): container finished" podID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f" containerID="05b2121802009bce193d940a1c9bd725d5957861b598ea5248c6ca6ddb73ac5f" exitCode=60
Nov 03 01:45:21.388869 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.388713    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp" event={"ID":"dc9a7667-192a-40aa-b2cb-f0fc140aa55f","Type":"ContainerDied","Data":"05b2121802009bce193d940a1c9bd725d5957861b598ea5248c6ca6ddb73ac5f"}
Nov 03 01:45:21.391271 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.391246    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph" event={"ID":"eb1e5080-28b5-409a-ac9d-63bd5bf7ea42","Type":"ContainerDied","Data":"bf89395072cf7e6f3a1743790ffe619419a7c88d0168ba65c6c72a7e82981bbf"}
Nov 03 01:45:21.391524 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.391509    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"
Nov 03 01:45:21.392402 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.392387    5508 scope.go:122] "RemoveContainer" containerID="05b2121802009bce193d940a1c9bd725d5957861b598ea5248c6ca6ddb73ac5f"
Nov 03 01:45:21.392920 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:21.392898    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp_odh-ai-gateway-infra(dc9a7667-192a-40aa-b2cb-f0fc140aa55f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp" podUID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f"
Nov 03 01:45:21.408311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.408289    5508 scope.go:122] "RemoveContainer" containerID="3febde81dfcd3b52c707d80e6fd3e0824e897cf571339a6767d96339ed960d11"
Nov 03 01:45:21.428589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.428532    5508 scope.go:122] "RemoveContainer" containerID="eb999efcba5a91073f2a87c3198b00a2a6f3c94f49e7922c23e57a43c0c38367"
Nov 03 01:45:21.483447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.483415    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"]
Nov 03 01:45:21.579768 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.576737    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"]
Nov 03 01:45:21.694006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.693972    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"
Nov 03 01:45:21.698657 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.698631    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"
Nov 03 01:45:21.799372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.796723    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"]
Nov 03 01:45:21.806372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.805432    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f4e704-29894505-sn5ph"]
Nov 03 01:45:21.813371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.812918    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40d7637-4656-4d80-ba61-aaa61ba14ccf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40d7637-4656-4d80-ba61-aaa61ba14ccf-maas-api-service-ca\") pod \"b40d7637-4656-4d80-ba61-aaa61ba14ccf\" (UID: \"b40d7637-4656-4d80-ba61-aaa61ba14ccf\") "
Nov 03 01:45:21.813371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.813088    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b49ab06-b764-450f-8049-f2d092c03414-kube-api-access-8wc44\" (UniqueName: \"kubernetes.io/projected/4b49ab06-b764-450f-8049-f2d092c03414-kube-api-access-8wc44\") pod \"4b49ab06-b764-450f-8049-f2d092c03414\" (UID: \"4b49ab06-b764-450f-8049-f2d092c03414\") "
Nov 03 01:45:21.813371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.813336    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b49ab06-b764-450f-8049-f2d092c03414-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b49ab06-b764-450f-8049-f2d092c03414-maas-api-service-ca\") pod \"4b49ab06-b764-450f-8049-f2d092c03414\" (UID: \"4b49ab06-b764-450f-8049-f2d092c03414\") "
Nov 03 01:45:21.813725 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.813386    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40d7637-4656-4d80-ba61-aaa61ba14ccf-kube-api-access-4td92\" (UniqueName: \"kubernetes.io/projected/b40d7637-4656-4d80-ba61-aaa61ba14ccf-kube-api-access-4td92\") pod \"b40d7637-4656-4d80-ba61-aaa61ba14ccf\" (UID: \"b40d7637-4656-4d80-ba61-aaa61ba14ccf\") "
Nov 03 01:45:21.821372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.815817    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b49ab06-b764-450f-8049-f2d092c03414-maas-api-service-ca" pod "4b49ab06-b764-450f-8049-f2d092c03414" (UID: "4b49ab06-b764-450f-8049-f2d092c03414"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:21.821372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.815953    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40d7637-4656-4d80-ba61-aaa61ba14ccf-maas-api-service-ca" pod "b40d7637-4656-4d80-ba61-aaa61ba14ccf" (UID: "b40d7637-4656-4d80-ba61-aaa61ba14ccf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:21.823745 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.823652    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40d7637-4656-4d80-ba61-aaa61ba14ccf-kube-api-access-4td92" pod "b40d7637-4656-4d80-ba61-aaa61ba14ccf" (UID: "b40d7637-4656-4d80-ba61-aaa61ba14ccf"). InnerVolumeSpecName "kube-api-access-4td92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:21.847864 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.844847    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b49ab06-b764-450f-8049-f2d092c03414-kube-api-access-8wc44" pod "4b49ab06-b764-450f-8049-f2d092c03414" (UID: "4b49ab06-b764-450f-8049-f2d092c03414"). InnerVolumeSpecName "kube-api-access-8wc44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:21.925020 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.924748    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wc44\" (UniqueName: \"kubernetes.io/projected/4b49ab06-b764-450f-8049-f2d092c03414-kube-api-access-8wc44\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:21.925020 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.924788    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b49ab06-b764-450f-8049-f2d092c03414-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:21.925020 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.924803    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4td92\" (UniqueName: \"kubernetes.io/projected/b40d7637-4656-4d80-ba61-aaa61ba14ccf-kube-api-access-4td92\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:21.925020 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.924817    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40d7637-4656-4d80-ba61-aaa61ba14ccf-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:21.931394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:21.928161    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"]
Nov 03 01:45:22.005623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.005581    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"]
Nov 03 01:45:22.115633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.115600    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"]
Nov 03 01:45:22.142624 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.142588    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"
Nov 03 01:45:22.229110 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.229079    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab2b38d-28f9-4676-85fb-4f315928c129-kube-api-access-rnr64\" (UniqueName: \"kubernetes.io/projected/5ab2b38d-28f9-4676-85fb-4f315928c129-kube-api-access-rnr64\") pod \"5ab2b38d-28f9-4676-85fb-4f315928c129\" (UID: \"5ab2b38d-28f9-4676-85fb-4f315928c129\") "
Nov 03 01:45:22.229317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.229296    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab2b38d-28f9-4676-85fb-4f315928c129-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab2b38d-28f9-4676-85fb-4f315928c129-maas-api-service-ca\") pod \"5ab2b38d-28f9-4676-85fb-4f315928c129\" (UID: \"5ab2b38d-28f9-4676-85fb-4f315928c129\") "
Nov 03 01:45:22.229659 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.229635    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab2b38d-28f9-4676-85fb-4f315928c129-maas-api-service-ca" pod "5ab2b38d-28f9-4676-85fb-4f315928c129" (UID: "5ab2b38d-28f9-4676-85fb-4f315928c129"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:22.230034 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.230014    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab2b38d-28f9-4676-85fb-4f315928c129-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:22.231793 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.231772    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab2b38d-28f9-4676-85fb-4f315928c129-kube-api-access-rnr64" pod "5ab2b38d-28f9-4676-85fb-4f315928c129" (UID: "5ab2b38d-28f9-4676-85fb-4f315928c129"). InnerVolumeSpecName "kube-api-access-rnr64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:22.312684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.312658    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"
Nov 03 01:45:22.331947 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.331909    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnr64\" (UniqueName: \"kubernetes.io/projected/5ab2b38d-28f9-4676-85fb-4f315928c129-kube-api-access-rnr64\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:22.411362 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.411317    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"
Nov 03 01:45:22.411362 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.411325    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh" event={"ID":"4bde3b86-9dd2-482f-9598-7e42c44ef5ff","Type":"ContainerDied","Data":"2a9604811bc858d35e7d2e1147da7ba9c2d757fa958f3a943e7d3f5c54965fa0"}
Nov 03 01:45:22.411903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.411427    5508 scope.go:122] "RemoveContainer" containerID="a367c2b1331a6a86b48642597267694d0d37a743e95853b675450c3a0f660195"
Nov 03 01:45:22.413129 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.413091    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9" event={"ID":"5ab2b38d-28f9-4676-85fb-4f315928c129","Type":"ContainerDied","Data":"64f7e4802956adf4d629e7696cd125365a021f0647c1cffac0fadc1081b3ae75"}
Nov 03 01:45:22.413237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.413193    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"
Nov 03 01:45:22.419244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.419207    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl" event={"ID":"4b49ab06-b764-450f-8049-f2d092c03414","Type":"ContainerDied","Data":"d3e33129900b2082fda61b917f26b22b08a6ff40c69b3288b4dcf754b6407e27"}
Nov 03 01:45:22.419404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.419384    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"
Nov 03 01:45:22.434824 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.434755    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-kube-api-access-zvm86\" (UniqueName: \"kubernetes.io/projected/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-kube-api-access-zvm86\") pod \"4bde3b86-9dd2-482f-9598-7e42c44ef5ff\" (UID: \"4bde3b86-9dd2-482f-9598-7e42c44ef5ff\") "
Nov 03 01:45:22.434969 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.434914    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-maas-api-service-ca\") pod \"4bde3b86-9dd2-482f-9598-7e42c44ef5ff\" (UID: \"4bde3b86-9dd2-482f-9598-7e42c44ef5ff\") "
Nov 03 01:45:22.435281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.435258    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-maas-api-service-ca" pod "4bde3b86-9dd2-482f-9598-7e42c44ef5ff" (UID: "4bde3b86-9dd2-482f-9598-7e42c44ef5ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:22.435634 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.435612    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:22.435720 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.435698    5508 scope.go:122] "RemoveContainer" containerID="f2076f39b5ccef2f2c970117d1fd61080effb25c81ce5f752b7cdcb87a258af4"
Nov 03 01:45:22.437498 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.437104    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-kube-api-access-zvm86" pod "4bde3b86-9dd2-482f-9598-7e42c44ef5ff" (UID: "4bde3b86-9dd2-482f-9598-7e42c44ef5ff"). InnerVolumeSpecName "kube-api-access-zvm86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:22.437498 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.437478    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz" event={"ID":"b40d7637-4656-4d80-ba61-aaa61ba14ccf","Type":"ContainerDied","Data":"5719c332aae4ee4b1c798ce46641452ac8d946ab82667bfc79e1a42b6f3f8621"}
Nov 03 01:45:22.437739 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.437635    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"
Nov 03 01:45:22.451664 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.451530    5508 scope.go:122] "RemoveContainer" containerID="89e33e5199359c52baadcf79509dcb3f34d507e68bb9c157b09aa5946cee218f"
Nov 03 01:45:22.466299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.466153    5508 scope.go:122] "RemoveContainer" containerID="732153e7c323828b637b7555b73373d55056311d5039e33517ec4978819d24cd"
Nov 03 01:45:22.538574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.538487    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvm86\" (UniqueName: \"kubernetes.io/projected/4bde3b86-9dd2-482f-9598-7e42c44ef5ff-kube-api-access-zvm86\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:22.590467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.590433    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"
Nov 03 01:45:22.651418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.651380    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"
Nov 03 01:45:22.695851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.695774    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af6d4708-bf8d-4c36-955d-0f63b572b5af" path="/var/lib/kubelet/pods/af6d4708-bf8d-4c36-955d-0f63b572b5af/volumes"
Nov 03 01:45:22.696213 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.696200    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" path="/var/lib/kubelet/pods/eb1e5080-28b5-409a-ac9d-63bd5bf7ea42/volumes"
Nov 03 01:45:22.740901 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.740861    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-maas-api-service-ca\") pod \"dc9a7667-192a-40aa-b2cb-f0fc140aa55f\" (UID: \"dc9a7667-192a-40aa-b2cb-f0fc140aa55f\") "
Nov 03 01:45:22.741094 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.740924    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-maas-api-service-ca\") pod \"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3\" (UID: \"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3\") "
Nov 03 01:45:22.741094 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.740998    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-kube-api-access-cs76v\" (UniqueName: \"kubernetes.io/projected/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-kube-api-access-cs76v\") pod \"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3\" (UID: \"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3\") "
Nov 03 01:45:22.741213 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.741199    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-kube-api-access-g6d69\" (UniqueName: \"kubernetes.io/projected/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-kube-api-access-g6d69\") pod \"dc9a7667-192a-40aa-b2cb-f0fc140aa55f\" (UID: \"dc9a7667-192a-40aa-b2cb-f0fc140aa55f\") "
Nov 03 01:45:22.741313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.741268    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-maas-api-service-ca" pod "dc9a7667-192a-40aa-b2cb-f0fc140aa55f" (UID: "dc9a7667-192a-40aa-b2cb-f0fc140aa55f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:22.741380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.741270    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-maas-api-service-ca" pod "f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" (UID: "f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:22.742069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.742045    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:22.742219 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.742085    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:22.743546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.743523    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-kube-api-access-cs76v" pod "f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" (UID: "f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3"). InnerVolumeSpecName "kube-api-access-cs76v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:22.743637 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.743594    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-kube-api-access-g6d69" pod "dc9a7667-192a-40aa-b2cb-f0fc140aa55f" (UID: "dc9a7667-192a-40aa-b2cb-f0fc140aa55f"). InnerVolumeSpecName "kube-api-access-g6d69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:22.791277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.791242    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"]
Nov 03 01:45:22.797178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.797140    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-012d31-29894505-v9wzz"]
Nov 03 01:45:22.843190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.843154    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6d69\" (UniqueName: \"kubernetes.io/projected/dc9a7667-192a-40aa-b2cb-f0fc140aa55f-kube-api-access-g6d69\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:22.843190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.843185    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs76v\" (UniqueName: \"kubernetes.io/projected/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3-kube-api-access-cs76v\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:22.900137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.900098    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"]
Nov 03 01:45:22.905722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:22.905688    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894505-kfcxl"]
Nov 03 01:45:23.007023 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.006916    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"]
Nov 03 01:45:23.011196 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.011158    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894505-pd2q9"]
Nov 03 01:45:23.337890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.337787    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"]
Nov 03 01:45:23.343125 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.343100    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894505-8ctjh"]
Nov 03 01:45:23.478584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.478556    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"
Nov 03 01:45:23.478584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.478564    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px" event={"ID":"f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3","Type":"ContainerDied","Data":"3e0545c1ecd5612536151e16c29e1310391f6a83bad37a0681ed6a10d6646a2f"}
Nov 03 01:45:23.479124 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.478631    5508 scope.go:122] "RemoveContainer" containerID="503ef7bd943c5f03e4203920d07eac86cd8c1c6614987f74f86b54cbb2ee7aa5"
Nov 03 01:45:23.486410 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.484975    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"
Nov 03 01:45:23.486410 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.484973    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp" event={"ID":"dc9a7667-192a-40aa-b2cb-f0fc140aa55f","Type":"ContainerDied","Data":"f62d6fa98a476cb8f5140db108d1b62d3b9c0ee1f9d746ddf49a7d3cb2b46554"}
Nov 03 01:45:23.504693 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.500031    5508 scope.go:122] "RemoveContainer" containerID="05b2121802009bce193d940a1c9bd725d5957861b598ea5248c6ca6ddb73ac5f"
Nov 03 01:45:23.635178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.635095    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"]
Nov 03 01:45:23.641047 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.641014    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894505-wj7qp"]
Nov 03 01:45:23.743848 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.743790    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"]
Nov 03 01:45:23.749904 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:23.749866    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894505-ms5px"]
Nov 03 01:45:24.693611 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:24.693580    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b49ab06-b764-450f-8049-f2d092c03414" path="/var/lib/kubelet/pods/4b49ab06-b764-450f-8049-f2d092c03414/volumes"
Nov 03 01:45:24.694114 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:24.694096    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff" path="/var/lib/kubelet/pods/4bde3b86-9dd2-482f-9598-7e42c44ef5ff/volumes"
Nov 03 01:45:24.694581 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:24.694566    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab2b38d-28f9-4676-85fb-4f315928c129" path="/var/lib/kubelet/pods/5ab2b38d-28f9-4676-85fb-4f315928c129/volumes"
Nov 03 01:45:24.695412 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:24.695390    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b40d7637-4656-4d80-ba61-aaa61ba14ccf" path="/var/lib/kubelet/pods/b40d7637-4656-4d80-ba61-aaa61ba14ccf/volumes"
Nov 03 01:45:24.695840 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:24.695826    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f" path="/var/lib/kubelet/pods/dc9a7667-192a-40aa-b2cb-f0fc140aa55f/volumes"
Nov 03 01:45:24.696232 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:24.696219    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" path="/var/lib/kubelet/pods/f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3/volumes"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.343595    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"]
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344607    5508 cpu_manager.go:414] "containerMap: removing container" podUID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344627    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344641    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5ab2b38d-28f9-4676-85fb-4f315928c129" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344646    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab2b38d-28f9-4676-85fb-4f315928c129" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344653    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344658    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344665    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344669    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344681    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3f039b14-e4cc-44bf-a808-63a8c543e128" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344703    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f039b14-e4cc-44bf-a808-63a8c543e128" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344712    5508 cpu_manager.go:414] "containerMap: removing container" podUID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344717    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344724    5508 cpu_manager.go:414] "containerMap: removing container" podUID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344729    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344741    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b40d7637-4656-4d80-ba61-aaa61ba14ccf" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344749    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40d7637-4656-4d80-ba61-aaa61ba14ccf" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344763    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4b49ab06-b764-450f-8049-f2d092c03414" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344771    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b49ab06-b764-450f-8049-f2d092c03414" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344787    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344795    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344801    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344805    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344812    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5ab2b38d-28f9-4676-85fb-4f315928c129" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344816    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab2b38d-28f9-4676-85fb-4f315928c129" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344824    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4b49ab06-b764-450f-8049-f2d092c03414" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344828    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b49ab06-b764-450f-8049-f2d092c03414" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344833    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5ab2b38d-28f9-4676-85fb-4f315928c129" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344840    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab2b38d-28f9-4676-85fb-4f315928c129" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344858    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4b49ab06-b764-450f-8049-f2d092c03414" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344863    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b49ab06-b764-450f-8049-f2d092c03414" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344879    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b40d7637-4656-4d80-ba61-aaa61ba14ccf" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344884    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40d7637-4656-4d80-ba61-aaa61ba14ccf" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344891    5508 cpu_manager.go:414] "containerMap: removing container" podUID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344895    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344907    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3f039b14-e4cc-44bf-a808-63a8c543e128" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344912    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f039b14-e4cc-44bf-a808-63a8c543e128" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344935    5508 cpu_manager.go:414] "containerMap: removing container" podUID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.344942    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345065    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345075    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345086    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40d7637-4656-4d80-ba61-aaa61ba14ccf" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345095    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b49ab06-b764-450f-8049-f2d092c03414" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345103    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345113    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345121    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab2b38d-28f9-4676-85fb-4f315928c129" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345129    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab2b38d-28f9-4676-85fb-4f315928c129" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345140    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b49ab06-b764-450f-8049-f2d092c03414" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345149    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40d7637-4656-4d80-ba61-aaa61ba14ccf" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345159    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40d7637-4656-4d80-ba61-aaa61ba14ccf" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345167    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345176    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345184    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f039b14-e4cc-44bf-a808-63a8c543e128" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345193    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345201    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345208    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345221    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345231    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff" containerName="cleanup"
Nov 03 01:45:33.347372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.345241    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f039b14-e4cc-44bf-a808-63a8c543e128" containerName="cleanup"
Nov 03 01:45:33.350686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.350122    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"]
Nov 03 01:45:33.350686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.350170    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"]
Nov 03 01:45:33.361186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.351186    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff" containerName="cleanup"
Nov 03 01:45:33.361186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.351210    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bde3b86-9dd2-482f-9598-7e42c44ef5ff" containerName="cleanup"
Nov 03 01:45:33.361186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.351237    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b40d7637-4656-4d80-ba61-aaa61ba14ccf" containerName="cleanup"
Nov 03 01:45:33.361186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.351246    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40d7637-4656-4d80-ba61-aaa61ba14ccf" containerName="cleanup"
Nov 03 01:45:33.361186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.351271    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" containerName="cleanup"
Nov 03 01:45:33.361186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.351279    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d39f5d-61a3-424c-85a2-0d26f2ce1ac3" containerName="cleanup"
Nov 03 01:45:33.361186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.351336    5508 cpu_manager.go:414] "containerMap: removing container" podUID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" containerName="cleanup"
Nov 03 01:45:33.361186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.351358    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1e5080-28b5-409a-ac9d-63bd5bf7ea42" containerName="cleanup"
Nov 03 01:45:33.361186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.351587    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc9a7667-192a-40aa-b2cb-f0fc140aa55f" containerName="cleanup"
Nov 03 01:45:33.361186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.351600    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b49ab06-b764-450f-8049-f2d092c03414" containerName="cleanup"
Nov 03 01:45:33.361186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.351613    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab2b38d-28f9-4676-85fb-4f315928c129" containerName="cleanup"
Nov 03 01:45:33.361186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.351938    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"
Nov 03 01:45:33.362155 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.362129    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"]
Nov 03 01:45:33.362275 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.362242    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"
Nov 03 01:45:33.368794 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.368765    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2t5c\" (UniqueName: \"kubernetes.io/projected/424066ff-90ea-4e4b-a556-9830b0835d26-kube-api-access-v2t5c\") pod \"payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j\" (UID: \"424066ff-90ea-4e4b-a556-9830b0835d26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"
Nov 03 01:45:33.368976 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.368905    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb\" (UID: \"c17e7840-e240-40eb-b0b0-8ba0a790dd8f\") " pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"
Nov 03 01:45:33.368976 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.368940    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424066ff-90ea-4e4b-a556-9830b0835d26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j\" (UID: \"424066ff-90ea-4e4b-a556-9830b0835d26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"
Nov 03 01:45:33.369132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.369119    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rjwv\" (UniqueName: \"kubernetes.io/projected/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-kube-api-access-5rjwv\") pod \"payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb\" (UID: \"c17e7840-e240-40eb-b0b0-8ba0a790dd8f\") " pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"
Nov 03 01:45:33.421387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.413035    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2b9f22c1\""
Nov 03 01:45:33.452101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.450655    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2b9f22c1-dockercfg-c2p49\""
Nov 03 01:45:33.487636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.479802    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb\" (UID: \"c17e7840-e240-40eb-b0b0-8ba0a790dd8f\") " pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"
Nov 03 01:45:33.487636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.479858    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424066ff-90ea-4e4b-a556-9830b0835d26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j\" (UID: \"424066ff-90ea-4e4b-a556-9830b0835d26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"
Nov 03 01:45:33.487636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.480047    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rjwv\" (UniqueName: \"kubernetes.io/projected/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-kube-api-access-5rjwv\") pod \"payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb\" (UID: \"c17e7840-e240-40eb-b0b0-8ba0a790dd8f\") " pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"
Nov 03 01:45:33.487636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.480152    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2t5c\" (UniqueName: \"kubernetes.io/projected/424066ff-90ea-4e4b-a556-9830b0835d26-kube-api-access-v2t5c\") pod \"payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j\" (UID: \"424066ff-90ea-4e4b-a556-9830b0835d26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"
Nov 03 01:45:33.487636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.483774    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-plugins-config-volume\") pod \"payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb\" (UID: \"c17e7840-e240-40eb-b0b0-8ba0a790dd8f\") " pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"
Nov 03 01:45:33.487636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.487231    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424066ff-90ea-4e4b-a556-9830b0835d26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j\" (UID: \"424066ff-90ea-4e4b-a556-9830b0835d26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"
Nov 03 01:45:33.537164 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.537137    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2t5c\" (UniqueName: \"kubernetes.io/projected/424066ff-90ea-4e4b-a556-9830b0835d26-kube-api-access-v2t5c\") pod \"payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j\" (UID: \"424066ff-90ea-4e4b-a556-9830b0835d26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"
Nov 03 01:45:33.539973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.539943    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rjwv\" (UniqueName: \"kubernetes.io/projected/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-kube-api-access-5rjwv\") pod \"payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb\" (UID: \"c17e7840-e240-40eb-b0b0-8ba0a790dd8f\") " pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"
Nov 03 01:45:33.717302 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.717249    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"
Nov 03 01:45:33.725537 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:33.725513    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"
Nov 03 01:45:34.158927 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:34.158889    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"]
Nov 03 01:45:34.203812 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:45:34.203779    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17e7840_e240_40eb_b0b0_8ba0a790dd8f.slice/crio-68ec6a0ad5b4654b2e42740ef89e8cd379fd8c5a2ba4461aa2eb33a7f96d6d58 WatchSource:0}: Error finding container 68ec6a0ad5b4654b2e42740ef89e8cd379fd8c5a2ba4461aa2eb33a7f96d6d58: Status 404 returned error can't find the container with id 68ec6a0ad5b4654b2e42740ef89e8cd379fd8c5a2ba4461aa2eb33a7f96d6d58
Nov 03 01:45:34.209823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:34.209733    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"]
Nov 03 01:45:34.217775 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:45:34.217742    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424066ff_90ea_4e4b_a556_9830b0835d26.slice/crio-2e2e03e0f531e3fea92828133b361ca564aceaab38fbc72de8731f33fb7ee2e3 WatchSource:0}: Error finding container 2e2e03e0f531e3fea92828133b361ca564aceaab38fbc72de8731f33fb7ee2e3: Status 404 returned error can't find the container with id 2e2e03e0f531e3fea92828133b361ca564aceaab38fbc72de8731f33fb7ee2e3
Nov 03 01:45:35.200876 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:35.200832    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j" event={"ID":"424066ff-90ea-4e4b-a556-9830b0835d26","Type":"ContainerStarted","Data":"82799efa8bbeed7572e5449a15cd1656a296530ec5fb6a2d89af122a0a4fd76a"}
Nov 03 01:45:35.201473 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:35.200962    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j" event={"ID":"424066ff-90ea-4e4b-a556-9830b0835d26","Type":"ContainerStarted","Data":"2e2e03e0f531e3fea92828133b361ca564aceaab38fbc72de8731f33fb7ee2e3"}
Nov 03 01:45:35.201473 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:35.200998    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"
Nov 03 01:45:35.203085 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:35.203050    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb" event={"ID":"c17e7840-e240-40eb-b0b0-8ba0a790dd8f","Type":"ContainerStarted","Data":"ff74c3d8827fba4726c600a2a43b5529aed388e76bb1bea3b188ff401d20d27b"}
Nov 03 01:45:35.203227 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:35.203114    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb" event={"ID":"c17e7840-e240-40eb-b0b0-8ba0a790dd8f","Type":"ContainerStarted","Data":"68ec6a0ad5b4654b2e42740ef89e8cd379fd8c5a2ba4461aa2eb33a7f96d6d58"}
Nov 03 01:45:35.203227 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:35.203151    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"
Nov 03 01:45:35.467436 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:35.463824    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb" podStartSLOduration=3.463801082 podStartE2EDuration="3.463801082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:35.450024226 +0000 UTC m=+1614.875630692" watchObservedRunningTime="2026-11-03 01:45:35.463801082 +0000 UTC m=+1614.889407548"
Nov 03 01:45:35.489436 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:35.468270    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j" podStartSLOduration=3.46825285 podStartE2EDuration="3.46825285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:35.335997924 +0000 UTC m=+1614.761604426" watchObservedRunningTime="2026-11-03 01:45:35.46825285 +0000 UTC m=+1614.893859316"
Nov 03 01:45:35.623095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:35.618936    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:45:36.061374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:36.050896    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"]
Nov 03 01:45:36.061374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:36.051830    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy" containerID="cri-o://375c972a106dfcba9f7f0db96a0e09179e38c055747bfe9777ee711d6a708467" gracePeriod=30
Nov 03 01:45:41.335113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.335034    5508 generic.go:356] "Generic (PLEG): container finished" podID="83390757-f37d-4198-b4ac-bc05f260fa82" containerID="375c972a106dfcba9f7f0db96a0e09179e38c055747bfe9777ee711d6a708467" exitCode=0
Nov 03 01:45:41.335113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.335104    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" event={"ID":"83390757-f37d-4198-b4ac-bc05f260fa82","Type":"ContainerDied","Data":"375c972a106dfcba9f7f0db96a0e09179e38c055747bfe9777ee711d6a708467"}
Nov 03 01:45:41.335611 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.335189    5508 scope.go:122] "RemoveContainer" containerID="54e9673589e42ee976169781ee949a322cdda03c97a34c0236981a90569c255c"
Nov 03 01:45:41.552324 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.552284    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"]
Nov 03 01:45:41.552590 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.552558    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb" podUID="c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" containerName="istio-proxy" containerID="cri-o://48fbcb4305404989e4b5b6da794443b9f9c31a4cb3841a48851187095cea6614" gracePeriod=30
Nov 03 01:45:41.598300 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.598219    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"]
Nov 03 01:45:41.598543 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.598517    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn" podUID="beeaa3a5-ee87-4129-a879-5f5f591ec803" containerName="istio-proxy" containerID="cri-o://848e6c08d801240b3d6542b1e3444d5806eec2310d5da171c771356535c16dfa" gracePeriod=30
Nov 03 01:45:41.602658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.602631    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"]
Nov 03 01:45:41.602852 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.602819    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg" podUID="af9497a7-0c68-4222-b517-fc47fa1bfaf3" containerName="istio-proxy" containerID="cri-o://e50578398abbd4b0193def7b21b69b9864b00f55efba4a2907661308a4032ccf" gracePeriod=30
Nov 03 01:45:41.649639 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.644713    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"]
Nov 03 01:45:41.649639 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.644988    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" podUID="53333dac-248a-4cf1-ac30-c1962d83d8b9" containerName="istio-proxy" containerID="cri-o://88ccdcfb3405094953e7c5668e14fb2e542c46a8fabd1d61e969b8331d437ded" gracePeriod=30
Nov 03 01:45:41.781367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.781327    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:45:41.899659 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.899626    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-data\") pod \"83390757-f37d-4198-b4ac-bc05f260fa82\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") "
Nov 03 01:45:41.899864 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.899697    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-kube-api-access-flhmd\" (UniqueName: \"kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-kube-api-access-flhmd\") pod \"83390757-f37d-4198-b4ac-bc05f260fa82\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") "
Nov 03 01:45:41.899864 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.899767    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83390757-f37d-4198-b4ac-bc05f260fa82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83390757-f37d-4198-b4ac-bc05f260fa82-istiod-ca-cert\") pod \"83390757-f37d-4198-b4ac-bc05f260fa82\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") "
Nov 03 01:45:41.899864 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.899825    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83390757-f37d-4198-b4ac-bc05f260fa82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83390757-f37d-4198-b4ac-bc05f260fa82-istio-podinfo\") pod \"83390757-f37d-4198-b4ac-bc05f260fa82\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") "
Nov 03 01:45:41.900045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.899911    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-socket\") pod \"83390757-f37d-4198-b4ac-bc05f260fa82\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") "
Nov 03 01:45:41.900045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.899936    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-data" pod "83390757-f37d-4198-b4ac-bc05f260fa82" (UID: "83390757-f37d-4198-b4ac-bc05f260fa82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:41.900045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.899974    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-certs\") pod \"83390757-f37d-4198-b4ac-bc05f260fa82\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") "
Nov 03 01:45:41.900045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.900014    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-credential-socket\") pod \"83390757-f37d-4198-b4ac-bc05f260fa82\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") "
Nov 03 01:45:41.900264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.900049    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-istio-token\" (UniqueName: \"kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-istio-token\") pod \"83390757-f37d-4198-b4ac-bc05f260fa82\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") "
Nov 03 01:45:41.900264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.900149    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-envoy\") pod \"83390757-f37d-4198-b4ac-bc05f260fa82\" (UID: \"83390757-f37d-4198-b4ac-bc05f260fa82\") "
Nov 03 01:45:41.900264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.900162    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83390757-f37d-4198-b4ac-bc05f260fa82-istiod-ca-cert" pod "83390757-f37d-4198-b4ac-bc05f260fa82" (UID: "83390757-f37d-4198-b4ac-bc05f260fa82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:41.900264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.900184    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-socket" pod "83390757-f37d-4198-b4ac-bc05f260fa82" (UID: "83390757-f37d-4198-b4ac-bc05f260fa82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:41.900264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.900220    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-certs" pod "83390757-f37d-4198-b4ac-bc05f260fa82" (UID: "83390757-f37d-4198-b4ac-bc05f260fa82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:41.900568 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.900406    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-credential-socket" pod "83390757-f37d-4198-b4ac-bc05f260fa82" (UID: "83390757-f37d-4198-b4ac-bc05f260fa82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:41.901040 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.901016    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:41.901040 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.901041    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:41.901211 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.901052    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:41.901211 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.901060    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:41.901211 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.901069    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83390757-f37d-4198-b4ac-bc05f260fa82-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:41.902453 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.902416    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-kube-api-access-flhmd" pod "83390757-f37d-4198-b4ac-bc05f260fa82" (UID: "83390757-f37d-4198-b4ac-bc05f260fa82"). InnerVolumeSpecName "kube-api-access-flhmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:41.902569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.902524    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83390757-f37d-4198-b4ac-bc05f260fa82-istio-podinfo" pod "83390757-f37d-4198-b4ac-bc05f260fa82" (UID: "83390757-f37d-4198-b4ac-bc05f260fa82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:45:41.902721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.902692    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-envoy" pod "83390757-f37d-4198-b4ac-bc05f260fa82" (UID: "83390757-f37d-4198-b4ac-bc05f260fa82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:41.903260 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:41.903239    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-istio-token" pod "83390757-f37d-4198-b4ac-bc05f260fa82" (UID: "83390757-f37d-4198-b4ac-bc05f260fa82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:42.002813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.002776    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flhmd\" (UniqueName: \"kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-kube-api-access-flhmd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:42.002813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.002807    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83390757-f37d-4198-b4ac-bc05f260fa82-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:42.002813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.002817    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83390757-f37d-4198-b4ac-bc05f260fa82-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:42.003061 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.002826    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83390757-f37d-4198-b4ac-bc05f260fa82-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:42.100094 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.100062    5508 scope.go:122] "RemoveContainer" containerID="6028e9394bc4069e737c5fe2011a983516e0ccdcffef8805422e55f8bdb65ce2"
Nov 03 01:45:42.112573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.112545    5508 scope.go:122] "RemoveContainer" containerID="9e9d9512a26cfb04d0de269e0b7c0ecb7366e99a12c1497672638c92ec6da9db"
Nov 03 01:45:42.124043 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.124024    5508 scope.go:122] "RemoveContainer" containerID="9d3f6b8fb4bcb8daca00f3a9a70d41b444dee7486dddf980eb7ad251df08860e"
Nov 03 01:45:42.135096 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.135073    5508 scope.go:122] "RemoveContainer" containerID="25eca983c72efcacb5f6293eb2bb77a5b911088d4d7fcd9994b1ca36a7f09fa9"
Nov 03 01:45:42.145715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.145695    5508 scope.go:122] "RemoveContainer" containerID="6ae38348099f20c71bac316a02d9c1026b09342a50b4ebf769c0312a2d201537"
Nov 03 01:45:42.157925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.157902    5508 scope.go:122] "RemoveContainer" containerID="523e48ed71911e11cdca79d41c6073e5d764d4fdadce898e6d524c396a6c9820"
Nov 03 01:45:42.168698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.168663    5508 scope.go:122] "RemoveContainer" containerID="95aa91b6484b637d674c1969f6340edf8dd39957cac33bb83bc2e67c44d7d071"
Nov 03 01:45:42.181058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.180589    5508 scope.go:122] "RemoveContainer" containerID="17ec610628dbe4f4f1d3eb1200304e11e09859cb6b43e860ef3a7bf852b94f08"
Nov 03 01:45:42.198649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.198613    5508 scope.go:122] "RemoveContainer" containerID="c47e7de2c1051f5e81141f8eceee757757aac202e1da35c24e19700354d6beac"
Nov 03 01:45:42.213821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.213792    5508 scope.go:122] "RemoveContainer" containerID="40f9f6a3322d15e852fee36860c32216de4cb36f4b6d8e59be452c6ae0092fe6"
Nov 03 01:45:42.348047 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.348021    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4" event={"ID":"83390757-f37d-4198-b4ac-bc05f260fa82","Type":"ContainerDied","Data":"eeb5b2167e05ba219ea238932d1bcec1a1a7598011995cac35d72e47ebedca2f"}
Nov 03 01:45:42.348047 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.348038    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"
Nov 03 01:45:42.348469 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.348088    5508 scope.go:122] "RemoveContainer" containerID="375c972a106dfcba9f7f0db96a0e09179e38c055747bfe9777ee711d6a708467"
Nov 03 01:45:42.404595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.404565    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"]
Nov 03 01:45:42.507655 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.507561    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"]
Nov 03 01:45:42.526426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.522635    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"]
Nov 03 01:45:42.529005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.528976    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-51bcf3-openshift-default-85d9658b4f-lklk4"]
Nov 03 01:45:42.700000 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:42.699962    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" path="/var/lib/kubelet/pods/83390757-f37d-4198-b4ac-bc05f260fa82/volumes"
Nov 03 01:45:43.172767 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:43.172734    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"]
Nov 03 01:45:43.173394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:43.173360    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy" containerID="cri-o://ed9656e1e22e25b2a91db70830a895a241a81f5c14127c35e13a2a02c3b2da92" gracePeriod=30
Nov 03 01:45:43.189723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:43.189665    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"]
Nov 03 01:45:43.190260 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:43.190220    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn" podUID="a07f4321-ff64-453e-a103-29676b99d594" containerName="istio-proxy" containerID="cri-o://5e5ae1738d5390a7c547061d3fcf18423e85f6d8a2377ad45fe0e232b3896f06" gracePeriod=30
Nov 03 01:45:43.950885 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:43.950843    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:45:44.524297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:44.524261    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:45:44.524495 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:44.524319    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg" podUID="af9497a7-0c68-4222-b517-fc47fa1bfaf3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:45:46.291904 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.291822    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"
Nov 03 01:45:46.291904 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.291891    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"
Nov 03 01:45:46.910592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.910287    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:45:46.939011 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.938982    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-socket\") pod \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") "
Nov 03 01:45:46.939145 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.939034    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-podinfo\") pod \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") "
Nov 03 01:45:46.939145 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.939071    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-kube-api-access-n5zdb\" (UniqueName: \"kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-kube-api-access-n5zdb\") pod \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") "
Nov 03 01:45:46.939252 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.939192    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istiod-ca-cert\") pod \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") "
Nov 03 01:45:46.939304 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.939293    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-envoy\") pod \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") "
Nov 03 01:45:46.939381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.939326    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-credential-socket\") pod \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") "
Nov 03 01:45:46.939457 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.939442    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-data\") pod \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") "
Nov 03 01:45:46.939522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.939507    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-token\" (UniqueName: \"kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-token\") pod \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") "
Nov 03 01:45:46.939597 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.939585    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-certs\") pod \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\" (UID: \"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2\") "
Nov 03 01:45:46.939904 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.939874    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-credential-socket" pod "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" (UID: "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:46.940160 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.940116    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-certs" pod "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" (UID: "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:46.940244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.940174    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-data" pod "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" (UID: "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:46.940379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.940338    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istiod-ca-cert" pod "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" (UID: "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:46.940490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.940463    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:46.940490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.940473    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-socket" pod "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" (UID: "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:46.940490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.940486    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:46.940603 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.940501    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:46.943332 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.943256    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-kube-api-access-n5zdb" pod "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" (UID: "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2"). InnerVolumeSpecName "kube-api-access-n5zdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:46.943332 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.943297    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-envoy" pod "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" (UID: "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:46.945057 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.945021    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-podinfo" pod "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" (UID: "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:45:46.945194 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:46.945171    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-token" pod "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" (UID: "c97fbb6e-fcbc-4ab8-9d3f-7299753616c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:47.019580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.019486    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:45:47.041591 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.041542    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-token\" (UniqueName: \"kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-token\") pod \"beeaa3a5-ee87-4129-a879-5f5f591ec803\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") "
Nov 03 01:45:47.041769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.041617    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-socket\") pod \"beeaa3a5-ee87-4129-a879-5f5f591ec803\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") "
Nov 03 01:45:47.041769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.041686    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-kube-api-access-vz9rf\" (UniqueName: \"kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-kube-api-access-vz9rf\") pod \"beeaa3a5-ee87-4129-a879-5f5f591ec803\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") "
Nov 03 01:45:47.041769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.041743    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-podinfo\") pod \"beeaa3a5-ee87-4129-a879-5f5f591ec803\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") "
Nov 03 01:45:47.041872 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.041843    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-envoy\") pod \"beeaa3a5-ee87-4129-a879-5f5f591ec803\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") "
Nov 03 01:45:47.041922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.041905    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-data\") pod \"beeaa3a5-ee87-4129-a879-5f5f591ec803\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") "
Nov 03 01:45:47.041991 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.041976    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-certs\") pod \"beeaa3a5-ee87-4129-a879-5f5f591ec803\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") "
Nov 03 01:45:47.042058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.042043    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beeaa3a5-ee87-4129-a879-5f5f591ec803-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beeaa3a5-ee87-4129-a879-5f5f591ec803-istiod-ca-cert\") pod \"beeaa3a5-ee87-4129-a879-5f5f591ec803\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") "
Nov 03 01:45:47.042127 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.042112    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-credential-socket\") pod \"beeaa3a5-ee87-4129-a879-5f5f591ec803\" (UID: \"beeaa3a5-ee87-4129-a879-5f5f591ec803\") "
Nov 03 01:45:47.042298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.042267    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-socket" pod "beeaa3a5-ee87-4129-a879-5f5f591ec803" (UID: "beeaa3a5-ee87-4129-a879-5f5f591ec803"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.042948 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.042922    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.043424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.043093    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beeaa3a5-ee87-4129-a879-5f5f591ec803-istiod-ca-cert" pod "beeaa3a5-ee87-4129-a879-5f5f591ec803" (UID: "beeaa3a5-ee87-4129-a879-5f5f591ec803"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:47.043424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.043101    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.043424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.043149    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.043424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.043168    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.043424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.043184    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.043424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.043201    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.043424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.043217    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5zdb\" (UniqueName: \"kubernetes.io/projected/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2-kube-api-access-n5zdb\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.043424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.043314    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-credential-socket" pod "beeaa3a5-ee87-4129-a879-5f5f591ec803" (UID: "beeaa3a5-ee87-4129-a879-5f5f591ec803"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.043424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.043392    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-data" pod "beeaa3a5-ee87-4129-a879-5f5f591ec803" (UID: "beeaa3a5-ee87-4129-a879-5f5f591ec803"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.044002 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.043974    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-certs" pod "beeaa3a5-ee87-4129-a879-5f5f591ec803" (UID: "beeaa3a5-ee87-4129-a879-5f5f591ec803"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.048915 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.048885    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-kube-api-access-vz9rf" pod "beeaa3a5-ee87-4129-a879-5f5f591ec803" (UID: "beeaa3a5-ee87-4129-a879-5f5f591ec803"). InnerVolumeSpecName "kube-api-access-vz9rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:47.049665 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.049641    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-envoy" pod "beeaa3a5-ee87-4129-a879-5f5f591ec803" (UID: "beeaa3a5-ee87-4129-a879-5f5f591ec803"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.049761 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.049647    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-token" pod "beeaa3a5-ee87-4129-a879-5f5f591ec803" (UID: "beeaa3a5-ee87-4129-a879-5f5f591ec803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:47.051526 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.051468    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-podinfo" pod "beeaa3a5-ee87-4129-a879-5f5f591ec803" (UID: "beeaa3a5-ee87-4129-a879-5f5f591ec803"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:45:47.069934 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.069846    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:45:47.079327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.079298    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.151638    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-envoy\") pod \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.151699    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-certs\") pod \"53333dac-248a-4cf1-ac30-c1962d83d8b9\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.151752    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-socket\") pod \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.151820    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-envoy\") pod \"53333dac-248a-4cf1-ac30-c1962d83d8b9\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.151876    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-certs\") pod \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.151936    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istiod-ca-cert\") pod \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.151985    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-podinfo\") pod \"53333dac-248a-4cf1-ac30-c1962d83d8b9\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152042    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-credential-socket\") pod \"53333dac-248a-4cf1-ac30-c1962d83d8b9\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152072    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-token\" (UniqueName: \"kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-token\") pod \"53333dac-248a-4cf1-ac30-c1962d83d8b9\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152115    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-credential-socket\") pod \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152181    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-token\" (UniqueName: \"kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-token\") pod \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152217    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-data\") pod \"53333dac-248a-4cf1-ac30-c1962d83d8b9\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152249    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53333dac-248a-4cf1-ac30-c1962d83d8b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53333dac-248a-4cf1-ac30-c1962d83d8b9-istiod-ca-cert\") pod \"53333dac-248a-4cf1-ac30-c1962d83d8b9\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152325    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-podinfo\") pod \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152451    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-socket\") pod \"53333dac-248a-4cf1-ac30-c1962d83d8b9\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152485    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-kube-api-access-5cj4j\" (UniqueName: \"kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-kube-api-access-5cj4j\") pod \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152570    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-socket" pod "af9497a7-0c68-4222-b517-fc47fa1bfaf3" (UID: "af9497a7-0c68-4222-b517-fc47fa1bfaf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152613    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-data\") pod \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\" (UID: \"af9497a7-0c68-4222-b517-fc47fa1bfaf3\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152648    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-kube-api-access-624p8\" (UniqueName: \"kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-kube-api-access-624p8\") pod \"53333dac-248a-4cf1-ac30-c1962d83d8b9\" (UID: \"53333dac-248a-4cf1-ac30-c1962d83d8b9\") "
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152651    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-certs" pod "53333dac-248a-4cf1-ac30-c1962d83d8b9" (UID: "53333dac-248a-4cf1-ac30-c1962d83d8b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152802    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-certs" pod "af9497a7-0c68-4222-b517-fc47fa1bfaf3" (UID: "af9497a7-0c68-4222-b517-fc47fa1bfaf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.152984    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-socket" pod "53333dac-248a-4cf1-ac30-c1962d83d8b9" (UID: "53333dac-248a-4cf1-ac30-c1962d83d8b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153003    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53333dac-248a-4cf1-ac30-c1962d83d8b9-istiod-ca-cert" pod "53333dac-248a-4cf1-ac30-c1962d83d8b9" (UID: "53333dac-248a-4cf1-ac30-c1962d83d8b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153470    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istiod-ca-cert" pod "af9497a7-0c68-4222-b517-fc47fa1bfaf3" (UID: "af9497a7-0c68-4222-b517-fc47fa1bfaf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153566    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz9rf\" (UniqueName: \"kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-kube-api-access-vz9rf\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153589    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153605    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153619    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153637    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153652    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153669    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153683    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153698    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beeaa3a5-ee87-4129-a879-5f5f591ec803-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153714    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53333dac-248a-4cf1-ac30-c1962d83d8b9-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153727    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beeaa3a5-ee87-4129-a879-5f5f591ec803-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153741    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153753    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beeaa3a5-ee87-4129-a879-5f5f591ec803-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.153790    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-credential-socket" pod "53333dac-248a-4cf1-ac30-c1962d83d8b9" (UID: "53333dac-248a-4cf1-ac30-c1962d83d8b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.155140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.154095    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-data" pod "af9497a7-0c68-4222-b517-fc47fa1bfaf3" (UID: "af9497a7-0c68-4222-b517-fc47fa1bfaf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.157181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.156286    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-token" pod "53333dac-248a-4cf1-ac30-c1962d83d8b9" (UID: "53333dac-248a-4cf1-ac30-c1962d83d8b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:47.157181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.156454    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-token" pod "af9497a7-0c68-4222-b517-fc47fa1bfaf3" (UID: "af9497a7-0c68-4222-b517-fc47fa1bfaf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:47.157181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.156522    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-data" pod "53333dac-248a-4cf1-ac30-c1962d83d8b9" (UID: "53333dac-248a-4cf1-ac30-c1962d83d8b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.157181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.156654    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-credential-socket" pod "af9497a7-0c68-4222-b517-fc47fa1bfaf3" (UID: "af9497a7-0c68-4222-b517-fc47fa1bfaf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.157598 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.157571    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-envoy" pod "af9497a7-0c68-4222-b517-fc47fa1bfaf3" (UID: "af9497a7-0c68-4222-b517-fc47fa1bfaf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.157768 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.157747    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-podinfo" pod "53333dac-248a-4cf1-ac30-c1962d83d8b9" (UID: "53333dac-248a-4cf1-ac30-c1962d83d8b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:45:47.157877 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.157857    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-podinfo" pod "af9497a7-0c68-4222-b517-fc47fa1bfaf3" (UID: "af9497a7-0c68-4222-b517-fc47fa1bfaf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:45:47.158777 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.158741    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-kube-api-access-5cj4j" pod "af9497a7-0c68-4222-b517-fc47fa1bfaf3" (UID: "af9497a7-0c68-4222-b517-fc47fa1bfaf3"). InnerVolumeSpecName "kube-api-access-5cj4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:47.159207 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.159182    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-kube-api-access-624p8" pod "53333dac-248a-4cf1-ac30-c1962d83d8b9" (UID: "53333dac-248a-4cf1-ac30-c1962d83d8b9"). InnerVolumeSpecName "kube-api-access-624p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:47.159644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.159621    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-envoy" pod "53333dac-248a-4cf1-ac30-c1962d83d8b9" (UID: "53333dac-248a-4cf1-ac30-c1962d83d8b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:47.271721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.270145    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.271721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.270178    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-624p8\" (UniqueName: \"kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-kube-api-access-624p8\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.271721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.270193    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.271721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.270206    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.271721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.270221    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.271721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.270235    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.271721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.270247    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.271721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.270260    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.271721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.270274    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9497a7-0c68-4222-b517-fc47fa1bfaf3-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.271721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.270290    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.271721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.270304    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53333dac-248a-4cf1-ac30-c1962d83d8b9-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.271721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.270317    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9497a7-0c68-4222-b517-fc47fa1bfaf3-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.271721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.270331    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cj4j\" (UniqueName: \"kubernetes.io/projected/af9497a7-0c68-4222-b517-fc47fa1bfaf3-kube-api-access-5cj4j\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.405461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.405327    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz" podUID="7c23153d-c0e2-4a72-a25a-67a4e61766bb" containerName="payload-processing" containerID="cri-o://f9db715e6ec6b47ef3131961c246b5ab49fc558e92b05aea28fbc208975e19a0" gracePeriod=25
Nov 03 01:45:47.439140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.439108    5508 generic.go:356] "Generic (PLEG): container finished" podID="c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" containerID="48fbcb4305404989e4b5b6da794443b9f9c31a4cb3841a48851187095cea6614" exitCode=0
Nov 03 01:45:47.439306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.439154    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"
Nov 03 01:45:47.439306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.439192    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb" event={"ID":"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2","Type":"ContainerDied","Data":"48fbcb4305404989e4b5b6da794443b9f9c31a4cb3841a48851187095cea6614"}
Nov 03 01:45:47.439306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.439253    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb" event={"ID":"c97fbb6e-fcbc-4ab8-9d3f-7299753616c2","Type":"ContainerDied","Data":"f6cd53f87c2458c32acb0c9382d49fb20e9290fe82e8c8fe06e03752762a253f"}
Nov 03 01:45:47.439306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.439275    5508 scope.go:122] "RemoveContainer" containerID="48fbcb4305404989e4b5b6da794443b9f9c31a4cb3841a48851187095cea6614"
Nov 03 01:45:47.440782 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.440758    5508 generic.go:356] "Generic (PLEG): container finished" podID="beeaa3a5-ee87-4129-a879-5f5f591ec803" containerID="848e6c08d801240b3d6542b1e3444d5806eec2310d5da171c771356535c16dfa" exitCode=0
Nov 03 01:45:47.440893 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.440801    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"
Nov 03 01:45:47.440893 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.440842    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn" event={"ID":"beeaa3a5-ee87-4129-a879-5f5f591ec803","Type":"ContainerDied","Data":"848e6c08d801240b3d6542b1e3444d5806eec2310d5da171c771356535c16dfa"}
Nov 03 01:45:47.440967 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.440899    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn" event={"ID":"beeaa3a5-ee87-4129-a879-5f5f591ec803","Type":"ContainerDied","Data":"6cddde5f1b97bb83151927ea1539054dd4562fbcdfb6b85fb5ddbb51bcf4448f"}
Nov 03 01:45:47.442637 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.442615    5508 generic.go:356] "Generic (PLEG): container finished" podID="af9497a7-0c68-4222-b517-fc47fa1bfaf3" containerID="e50578398abbd4b0193def7b21b69b9864b00f55efba4a2907661308a4032ccf" exitCode=0
Nov 03 01:45:47.442745 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.442673    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"
Nov 03 01:45:47.442745 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.442729    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg" event={"ID":"af9497a7-0c68-4222-b517-fc47fa1bfaf3","Type":"ContainerDied","Data":"e50578398abbd4b0193def7b21b69b9864b00f55efba4a2907661308a4032ccf"}
Nov 03 01:45:47.442862 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.442783    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg" event={"ID":"af9497a7-0c68-4222-b517-fc47fa1bfaf3","Type":"ContainerDied","Data":"74a41bfc01fd48ad360404d4edd9b9c06f830124b93bca5aaa83858bc9b327bf"}
Nov 03 01:45:47.470402 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.470309    5508 generic.go:356] "Generic (PLEG): container finished" podID="53333dac-248a-4cf1-ac30-c1962d83d8b9" containerID="88ccdcfb3405094953e7c5668e14fb2e542c46a8fabd1d61e969b8331d437ded" exitCode=0
Nov 03 01:45:47.470629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.470539    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" event={"ID":"53333dac-248a-4cf1-ac30-c1962d83d8b9","Type":"ContainerDied","Data":"88ccdcfb3405094953e7c5668e14fb2e542c46a8fabd1d61e969b8331d437ded"}
Nov 03 01:45:47.470699 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.470648    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8" event={"ID":"53333dac-248a-4cf1-ac30-c1962d83d8b9","Type":"ContainerDied","Data":"227b4f001481fe2acd8293b5e3e69d4b9b103ec7f14c7f57e0a89513f9a9ea23"}
Nov 03 01:45:47.470891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.470868    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"
Nov 03 01:45:47.508129 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.508097    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb" podUID="5556c0df-cf71-4528-ab71-71ef67e9ac3e" containerName="payload-pre-processing" containerID="cri-o://8359e5d50c7f2ed31d318e9d0995eb36bc3113b1f807c393c9184d0936a1e35a" gracePeriod=25
Nov 03 01:45:47.521067 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.521040    5508 scope.go:122] "RemoveContainer" containerID="48fbcb4305404989e4b5b6da794443b9f9c31a4cb3841a48851187095cea6614"
Nov 03 01:45:47.521497 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:47.521471    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48fbcb4305404989e4b5b6da794443b9f9c31a4cb3841a48851187095cea6614\": container with ID starting with 48fbcb4305404989e4b5b6da794443b9f9c31a4cb3841a48851187095cea6614 not found: ID does not exist" containerID="48fbcb4305404989e4b5b6da794443b9f9c31a4cb3841a48851187095cea6614"
Nov 03 01:45:47.521600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.521505    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48fbcb4305404989e4b5b6da794443b9f9c31a4cb3841a48851187095cea6614"} err="failed to get container status \"48fbcb4305404989e4b5b6da794443b9f9c31a4cb3841a48851187095cea6614\": rpc error: code = NotFound desc = could not find container \"48fbcb4305404989e4b5b6da794443b9f9c31a4cb3841a48851187095cea6614\": container with ID starting with 48fbcb4305404989e4b5b6da794443b9f9c31a4cb3841a48851187095cea6614 not found: ID does not exist"
Nov 03 01:45:47.521600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.521525    5508 scope.go:122] "RemoveContainer" containerID="848e6c08d801240b3d6542b1e3444d5806eec2310d5da171c771356535c16dfa"
Nov 03 01:45:47.557660 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.557640    5508 scope.go:122] "RemoveContainer" containerID="848e6c08d801240b3d6542b1e3444d5806eec2310d5da171c771356535c16dfa"
Nov 03 01:45:47.558086 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:47.558053    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848e6c08d801240b3d6542b1e3444d5806eec2310d5da171c771356535c16dfa\": container with ID starting with 848e6c08d801240b3d6542b1e3444d5806eec2310d5da171c771356535c16dfa not found: ID does not exist" containerID="848e6c08d801240b3d6542b1e3444d5806eec2310d5da171c771356535c16dfa"
Nov 03 01:45:47.558181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.558099    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848e6c08d801240b3d6542b1e3444d5806eec2310d5da171c771356535c16dfa"} err="failed to get container status \"848e6c08d801240b3d6542b1e3444d5806eec2310d5da171c771356535c16dfa\": rpc error: code = NotFound desc = could not find container \"848e6c08d801240b3d6542b1e3444d5806eec2310d5da171c771356535c16dfa\": container with ID starting with 848e6c08d801240b3d6542b1e3444d5806eec2310d5da171c771356535c16dfa not found: ID does not exist"
Nov 03 01:45:47.558181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.558125    5508 scope.go:122] "RemoveContainer" containerID="e50578398abbd4b0193def7b21b69b9864b00f55efba4a2907661308a4032ccf"
Nov 03 01:45:47.633379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.633210    5508 scope.go:122] "RemoveContainer" containerID="e50578398abbd4b0193def7b21b69b9864b00f55efba4a2907661308a4032ccf"
Nov 03 01:45:47.633918 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:47.633765    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50578398abbd4b0193def7b21b69b9864b00f55efba4a2907661308a4032ccf\": container with ID starting with e50578398abbd4b0193def7b21b69b9864b00f55efba4a2907661308a4032ccf not found: ID does not exist" containerID="e50578398abbd4b0193def7b21b69b9864b00f55efba4a2907661308a4032ccf"
Nov 03 01:45:47.633918 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.633806    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50578398abbd4b0193def7b21b69b9864b00f55efba4a2907661308a4032ccf"} err="failed to get container status \"e50578398abbd4b0193def7b21b69b9864b00f55efba4a2907661308a4032ccf\": rpc error: code = NotFound desc = could not find container \"e50578398abbd4b0193def7b21b69b9864b00f55efba4a2907661308a4032ccf\": container with ID starting with e50578398abbd4b0193def7b21b69b9864b00f55efba4a2907661308a4032ccf not found: ID does not exist"
Nov 03 01:45:47.633918 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.633827    5508 scope.go:122] "RemoveContainer" containerID="88ccdcfb3405094953e7c5668e14fb2e542c46a8fabd1d61e969b8331d437ded"
Nov 03 01:45:47.673720 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.673628    5508 scope.go:122] "RemoveContainer" containerID="88ccdcfb3405094953e7c5668e14fb2e542c46a8fabd1d61e969b8331d437ded"
Nov 03 01:45:47.675316 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:47.675246    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ccdcfb3405094953e7c5668e14fb2e542c46a8fabd1d61e969b8331d437ded\": container with ID starting with 88ccdcfb3405094953e7c5668e14fb2e542c46a8fabd1d61e969b8331d437ded not found: ID does not exist" containerID="88ccdcfb3405094953e7c5668e14fb2e542c46a8fabd1d61e969b8331d437ded"
Nov 03 01:45:47.675316 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.675279    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ccdcfb3405094953e7c5668e14fb2e542c46a8fabd1d61e969b8331d437ded"} err="failed to get container status \"88ccdcfb3405094953e7c5668e14fb2e542c46a8fabd1d61e969b8331d437ded\": rpc error: code = NotFound desc = could not find container \"88ccdcfb3405094953e7c5668e14fb2e542c46a8fabd1d61e969b8331d437ded\": container with ID starting with 88ccdcfb3405094953e7c5668e14fb2e542c46a8fabd1d61e969b8331d437ded not found: ID does not exist"
Nov 03 01:45:47.739572 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.739540    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"
Nov 03 01:45:47.752071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.752010    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"]
Nov 03 01:45:47.758837 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.758808    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rbwtb"]
Nov 03 01:45:47.801063 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.801027    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c23153d-c0e2-4a72-a25a-67a4e61766bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c23153d-c0e2-4a72-a25a-67a4e61766bb-plugins-config-volume\") pod \"7c23153d-c0e2-4a72-a25a-67a4e61766bb\" (UID: \"7c23153d-c0e2-4a72-a25a-67a4e61766bb\") "
Nov 03 01:45:47.801565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.801543    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c23153d-c0e2-4a72-a25a-67a4e61766bb-kube-api-access-ljwqn\" (UniqueName: \"kubernetes.io/projected/7c23153d-c0e2-4a72-a25a-67a4e61766bb-kube-api-access-ljwqn\") pod \"7c23153d-c0e2-4a72-a25a-67a4e61766bb\" (UID: \"7c23153d-c0e2-4a72-a25a-67a4e61766bb\") "
Nov 03 01:45:47.802852 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.802822    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c23153d-c0e2-4a72-a25a-67a4e61766bb-plugins-config-volume" pod "7c23153d-c0e2-4a72-a25a-67a4e61766bb" (UID: "7c23153d-c0e2-4a72-a25a-67a4e61766bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:47.805600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.805553    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c23153d-c0e2-4a72-a25a-67a4e61766bb-kube-api-access-ljwqn" pod "7c23153d-c0e2-4a72-a25a-67a4e61766bb" (UID: "7c23153d-c0e2-4a72-a25a-67a4e61766bb"). InnerVolumeSpecName "kube-api-access-ljwqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:47.862554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.862477    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"]
Nov 03 01:45:47.866903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.866868    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-k5pmn"]
Nov 03 01:45:47.903244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.903209    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljwqn\" (UniqueName: \"kubernetes.io/projected/7c23153d-c0e2-4a72-a25a-67a4e61766bb-kube-api-access-ljwqn\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.903244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.903240    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c23153d-c0e2-4a72-a25a-67a4e61766bb-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:47.906800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.906775    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"
Nov 03 01:45:47.969625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.969529    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"]
Nov 03 01:45:47.975815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:47.975782    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-6mxqg"]
Nov 03 01:45:48.004606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.004560    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5556c0df-cf71-4528-ab71-71ef67e9ac3e-kube-api-access-g5rdt\" (UniqueName: \"kubernetes.io/projected/5556c0df-cf71-4528-ab71-71ef67e9ac3e-kube-api-access-g5rdt\") pod \"5556c0df-cf71-4528-ab71-71ef67e9ac3e\" (UID: \"5556c0df-cf71-4528-ab71-71ef67e9ac3e\") "
Nov 03 01:45:48.004754 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.004687    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5556c0df-cf71-4528-ab71-71ef67e9ac3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5556c0df-cf71-4528-ab71-71ef67e9ac3e-plugins-config-volume\") pod \"5556c0df-cf71-4528-ab71-71ef67e9ac3e\" (UID: \"5556c0df-cf71-4528-ab71-71ef67e9ac3e\") "
Nov 03 01:45:48.005031 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.005008    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5556c0df-cf71-4528-ab71-71ef67e9ac3e-plugins-config-volume" pod "5556c0df-cf71-4528-ab71-71ef67e9ac3e" (UID: "5556c0df-cf71-4528-ab71-71ef67e9ac3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:48.005313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.005297    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5556c0df-cf71-4528-ab71-71ef67e9ac3e-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.006713 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.006691    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5556c0df-cf71-4528-ab71-71ef67e9ac3e-kube-api-access-g5rdt" pod "5556c0df-cf71-4528-ab71-71ef67e9ac3e" (UID: "5556c0df-cf71-4528-ab71-71ef67e9ac3e"). InnerVolumeSpecName "kube-api-access-g5rdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:48.073861 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.073781    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"]
Nov 03 01:45:48.078342 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.078302    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-kd6s8"]
Nov 03 01:45:48.106389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.106309    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5rdt\" (UniqueName: \"kubernetes.io/projected/5556c0df-cf71-4528-ab71-71ef67e9ac3e-kube-api-access-g5rdt\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.484618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.484578    5508 generic.go:356] "Generic (PLEG): container finished" podID="7c23153d-c0e2-4a72-a25a-67a4e61766bb" containerID="f9db715e6ec6b47ef3131961c246b5ab49fc558e92b05aea28fbc208975e19a0" exitCode=0
Nov 03 01:45:48.485158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.484674    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz" event={"ID":"7c23153d-c0e2-4a72-a25a-67a4e61766bb","Type":"ContainerDied","Data":"f9db715e6ec6b47ef3131961c246b5ab49fc558e92b05aea28fbc208975e19a0"}
Nov 03 01:45:48.485158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.484725    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz" event={"ID":"7c23153d-c0e2-4a72-a25a-67a4e61766bb","Type":"ContainerDied","Data":"ed016b52b2111c01454ca9cce329720fc966f99ae930b884f6d818da2ee37282"}
Nov 03 01:45:48.485158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.484752    5508 scope.go:122] "RemoveContainer" containerID="f9db715e6ec6b47ef3131961c246b5ab49fc558e92b05aea28fbc208975e19a0"
Nov 03 01:45:48.485158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.484869    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"
Nov 03 01:45:48.491845 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.491668    5508 generic.go:356] "Generic (PLEG): container finished" podID="a07f4321-ff64-453e-a103-29676b99d594" containerID="5e5ae1738d5390a7c547061d3fcf18423e85f6d8a2377ad45fe0e232b3896f06" exitCode=0
Nov 03 01:45:48.491845 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.491776    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn" event={"ID":"a07f4321-ff64-453e-a103-29676b99d594","Type":"ContainerDied","Data":"5e5ae1738d5390a7c547061d3fcf18423e85f6d8a2377ad45fe0e232b3896f06"}
Nov 03 01:45:48.500733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.497114    5508 generic.go:356] "Generic (PLEG): container finished" podID="5556c0df-cf71-4528-ab71-71ef67e9ac3e" containerID="8359e5d50c7f2ed31d318e9d0995eb36bc3113b1f807c393c9184d0936a1e35a" exitCode=0
Nov 03 01:45:48.500733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.497226    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb" event={"ID":"5556c0df-cf71-4528-ab71-71ef67e9ac3e","Type":"ContainerDied","Data":"8359e5d50c7f2ed31d318e9d0995eb36bc3113b1f807c393c9184d0936a1e35a"}
Nov 03 01:45:48.500733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.497298    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb" event={"ID":"5556c0df-cf71-4528-ab71-71ef67e9ac3e","Type":"ContainerDied","Data":"da6ec7000176a4abb745cb1befef4343b858c10733ecd91a3810131689e81e3a"}
Nov 03 01:45:48.500733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.497398    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"
Nov 03 01:45:48.515757 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.515629    5508 generic.go:356] "Generic (PLEG): container finished" podID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerID="ed9656e1e22e25b2a91db70830a895a241a81f5c14127c35e13a2a02c3b2da92" exitCode=0
Nov 03 01:45:48.515757 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.515666    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" event={"ID":"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2","Type":"ContainerDied","Data":"ed9656e1e22e25b2a91db70830a895a241a81f5c14127c35e13a2a02c3b2da92"}
Nov 03 01:45:48.528390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.528364    5508 scope.go:122] "RemoveContainer" containerID="f9db715e6ec6b47ef3131961c246b5ab49fc558e92b05aea28fbc208975e19a0"
Nov 03 01:45:48.528941 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:48.528914    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9db715e6ec6b47ef3131961c246b5ab49fc558e92b05aea28fbc208975e19a0\": container with ID starting with f9db715e6ec6b47ef3131961c246b5ab49fc558e92b05aea28fbc208975e19a0 not found: ID does not exist" containerID="f9db715e6ec6b47ef3131961c246b5ab49fc558e92b05aea28fbc208975e19a0"
Nov 03 01:45:48.529028 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.528953    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9db715e6ec6b47ef3131961c246b5ab49fc558e92b05aea28fbc208975e19a0"} err="failed to get container status \"f9db715e6ec6b47ef3131961c246b5ab49fc558e92b05aea28fbc208975e19a0\": rpc error: code = NotFound desc = could not find container \"f9db715e6ec6b47ef3131961c246b5ab49fc558e92b05aea28fbc208975e19a0\": container with ID starting with f9db715e6ec6b47ef3131961c246b5ab49fc558e92b05aea28fbc208975e19a0 not found: ID does not exist"
Nov 03 01:45:48.529028 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.528980    5508 scope.go:122] "RemoveContainer" containerID="8359e5d50c7f2ed31d318e9d0995eb36bc3113b1f807c393c9184d0936a1e35a"
Nov 03 01:45:48.540317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.540290    5508 scope.go:122] "RemoveContainer" containerID="8359e5d50c7f2ed31d318e9d0995eb36bc3113b1f807c393c9184d0936a1e35a"
Nov 03 01:45:48.540631 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:48.540609    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8359e5d50c7f2ed31d318e9d0995eb36bc3113b1f807c393c9184d0936a1e35a\": container with ID starting with 8359e5d50c7f2ed31d318e9d0995eb36bc3113b1f807c393c9184d0936a1e35a not found: ID does not exist" containerID="8359e5d50c7f2ed31d318e9d0995eb36bc3113b1f807c393c9184d0936a1e35a"
Nov 03 01:45:48.540681 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.540641    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8359e5d50c7f2ed31d318e9d0995eb36bc3113b1f807c393c9184d0936a1e35a"} err="failed to get container status \"8359e5d50c7f2ed31d318e9d0995eb36bc3113b1f807c393c9184d0936a1e35a\": rpc error: code = NotFound desc = could not find container \"8359e5d50c7f2ed31d318e9d0995eb36bc3113b1f807c393c9184d0936a1e35a\": container with ID starting with 8359e5d50c7f2ed31d318e9d0995eb36bc3113b1f807c393c9184d0936a1e35a not found: ID does not exist"
Nov 03 01:45:48.542029 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.542012    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:45:48.544966 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.544806    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:45:48.616608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.615376    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-credential-socket\") pod \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") "
Nov 03 01:45:48.617998 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.617488    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-kube-api-access-f599q\" (UniqueName: \"kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-kube-api-access-f599q\") pod \"a07f4321-ff64-453e-a103-29676b99d594\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") "
Nov 03 01:45:48.627516 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.617322    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-credential-socket" pod "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" (UID: "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.638747    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istiod-ca-cert" pod "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" (UID: "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.663457    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"]
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.627927    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istiod-ca-cert\") pod \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.664637    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-socket\") pod \"a07f4321-ff64-453e-a103-29676b99d594\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.664691    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-istio-token\" (UniqueName: \"kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-istio-token\") pod \"a07f4321-ff64-453e-a103-29676b99d594\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.664768    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-certs\") pod \"a07f4321-ff64-453e-a103-29676b99d594\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.664795    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-credential-socket\") pod \"a07f4321-ff64-453e-a103-29676b99d594\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.664869    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-envoy\") pod \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.664902    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-certs\") pod \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.664918    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-socket" pod "a07f4321-ff64-453e-a103-29676b99d594" (UID: "a07f4321-ff64-453e-a103-29676b99d594"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.664946    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-data\") pod \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.665023    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a07f4321-ff64-453e-a103-29676b99d594-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f4321-ff64-453e-a103-29676b99d594-istio-podinfo\") pod \"a07f4321-ff64-453e-a103-29676b99d594\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.665068    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-credential-socket" pod "a07f4321-ff64-453e-a103-29676b99d594" (UID: "a07f4321-ff64-453e-a103-29676b99d594"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.665100    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-token\" (UniqueName: \"kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-token\") pod \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.665136    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-kube-api-access-8spth\" (UniqueName: \"kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-kube-api-access-8spth\") pod \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.665164    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-socket\") pod \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.665209    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-data\") pod \"a07f4321-ff64-453e-a103-29676b99d594\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.665241    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-podinfo\") pod \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\" (UID: \"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2\") "
Nov 03 01:45:48.665367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.665318    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07f4321-ff64-453e-a103-29676b99d594-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f4321-ff64-453e-a103-29676b99d594-istiod-ca-cert\") pod \"a07f4321-ff64-453e-a103-29676b99d594\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") "
Nov 03 01:45:48.666506 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.665411    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-envoy\") pod \"a07f4321-ff64-453e-a103-29676b99d594\" (UID: \"a07f4321-ff64-453e-a103-29676b99d594\") "
Nov 03 01:45:48.666506 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.666123    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.666506 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.666146    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.666506 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.666159    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.666506 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.666173    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.670059 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.669549    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-certs" pod "a07f4321-ff64-453e-a103-29676b99d594" (UID: "a07f4321-ff64-453e-a103-29676b99d594"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:48.671326 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.670835    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-socket" pod "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" (UID: "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:48.671326 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.671042    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-certs" pod "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" (UID: "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:48.671709 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.671679    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-data" pod "a07f4321-ff64-453e-a103-29676b99d594" (UID: "a07f4321-ff64-453e-a103-29676b99d594"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:48.672103 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.671837    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-data" pod "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" (UID: "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:48.672103 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.672030    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07f4321-ff64-453e-a103-29676b99d594-istiod-ca-cert" pod "a07f4321-ff64-453e-a103-29676b99d594" (UID: "a07f4321-ff64-453e-a103-29676b99d594"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:48.676213 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.676184    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-012d31-7f958f5655-sfkxz"]
Nov 03 01:45:48.682725 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.682678    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-kube-api-access-8spth" pod "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" (UID: "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2"). InnerVolumeSpecName "kube-api-access-8spth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:48.682725 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.682703    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-token" pod "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" (UID: "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:48.682894 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.682721    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-envoy" pod "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" (UID: "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:48.682894 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.682760    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-podinfo" pod "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" (UID: "1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:45:48.683001 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.682888    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-envoy" pod "a07f4321-ff64-453e-a103-29676b99d594" (UID: "a07f4321-ff64-453e-a103-29676b99d594"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:48.683063 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.683003    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-kube-api-access-f599q" pod "a07f4321-ff64-453e-a103-29676b99d594" (UID: "a07f4321-ff64-453e-a103-29676b99d594"). InnerVolumeSpecName "kube-api-access-f599q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:48.683124 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.683069    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-istio-token" pod "a07f4321-ff64-453e-a103-29676b99d594" (UID: "a07f4321-ff64-453e-a103-29676b99d594"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:48.684788 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.684487    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a07f4321-ff64-453e-a103-29676b99d594-istio-podinfo" pod "a07f4321-ff64-453e-a103-29676b99d594" (UID: "a07f4321-ff64-453e-a103-29676b99d594"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:45:48.696436 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.696410    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53333dac-248a-4cf1-ac30-c1962d83d8b9" path="/var/lib/kubelet/pods/53333dac-248a-4cf1-ac30-c1962d83d8b9/volumes"
Nov 03 01:45:48.697397 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.696848    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c23153d-c0e2-4a72-a25a-67a4e61766bb" path="/var/lib/kubelet/pods/7c23153d-c0e2-4a72-a25a-67a4e61766bb/volumes"
Nov 03 01:45:48.697397 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.697253    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9497a7-0c68-4222-b517-fc47fa1bfaf3" path="/var/lib/kubelet/pods/af9497a7-0c68-4222-b517-fc47fa1bfaf3/volumes"
Nov 03 01:45:48.698152 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.698130    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beeaa3a5-ee87-4129-a879-5f5f591ec803" path="/var/lib/kubelet/pods/beeaa3a5-ee87-4129-a879-5f5f591ec803/volumes"
Nov 03 01:45:48.699382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.698652    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" path="/var/lib/kubelet/pods/c97fbb6e-fcbc-4ab8-9d3f-7299753616c2/volumes"
Nov 03 01:45:48.771178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.771116    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"]
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.771766    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.775994    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.776014    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.776028    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.776041    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.776054    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a07f4321-ff64-453e-a103-29676b99d594-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.776067    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.776081    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8spth\" (UniqueName: \"kubernetes.io/projected/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-kube-api-access-8spth\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.776094    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.776107    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.776119    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.776131    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a07f4321-ff64-453e-a103-29676b99d594-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.776144    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a07f4321-ff64-453e-a103-29676b99d594-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.776193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.776160    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f599q\" (UniqueName: \"kubernetes.io/projected/a07f4321-ff64-453e-a103-29676b99d594-kube-api-access-f599q\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:48.779239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:48.779137    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-012d31-54dccd5cf9-cbvzb"]
Nov 03 01:45:49.146913 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.145379    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"]
Nov 03 01:45:49.146913 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.145632    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n" podUID="0896f72f-e85a-4e3c-9a25-876e0a78f4b6" containerName="istio-proxy" containerID="cri-o://a9f22498b67b15f50e5544c5254b1cfc167e12ec401715705adc27f5bc973a08" gracePeriod=30
Nov 03 01:45:49.150886 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.150856    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"]
Nov 03 01:45:49.151437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.151400    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl" podUID="db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" containerName="istio-proxy" containerID="cri-o://c0aea2a4031aeadd85ae40f7e66f415f583bfe21b702349e1018985b6d0d7394" gracePeriod=30
Nov 03 01:45:49.156295 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.156262    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"]
Nov 03 01:45:49.156552 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.156521    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" podUID="a0e25b47-aaa7-4776-8757-c242e4441da2" containerName="istio-proxy" containerID="cri-o://8abaf68244b8e32bf3c6292a0848f65a116b03320e1e13b7eae49869635964d0" gracePeriod=30
Nov 03 01:45:49.532121 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.532038    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"
Nov 03 01:45:49.532549 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.532045    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc" event={"ID":"1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2","Type":"ContainerDied","Data":"ca10a8663d88a61959d525b16720d5ca847059a7ef34ef7e0247e24ad69739af"}
Nov 03 01:45:49.532549 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.532177    5508 scope.go:122] "RemoveContainer" containerID="ed9656e1e22e25b2a91db70830a895a241a81f5c14127c35e13a2a02c3b2da92"
Nov 03 01:45:49.534962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.534935    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"
Nov 03 01:45:49.535084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.534967    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn" event={"ID":"a07f4321-ff64-453e-a103-29676b99d594","Type":"ContainerDied","Data":"e6144ce799a400f42af171fc7401b6e43420b5f3d6ad0f0801e6930f268b8109"}
Nov 03 01:45:49.543236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.543213    5508 scope.go:122] "RemoveContainer" containerID="5e5ae1738d5390a7c547061d3fcf18423e85f6d8a2377ad45fe0e232b3896f06"
Nov 03 01:45:49.689831 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.689792    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"]
Nov 03 01:45:49.696613 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.696582    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-2dsmn"]
Nov 03 01:45:49.819507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.819411    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"]
Nov 03 01:45:49.825179 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:49.825144    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-7rgrc"]
Nov 03 01:45:50.086635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.086562    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"]
Nov 03 01:45:50.087213 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087199    5508 cpu_manager.go:414] "containerMap: removing container" podUID="beeaa3a5-ee87-4129-a879-5f5f591ec803" containerName="istio-proxy"
Nov 03 01:45:50.087271 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087215    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beeaa3a5-ee87-4129-a879-5f5f591ec803" containerName="istio-proxy"
Nov 03 01:45:50.087271 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087225    5508 cpu_manager.go:414] "containerMap: removing container" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy"
Nov 03 01:45:50.087271 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087230    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy"
Nov 03 01:45:50.087271 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087236    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7c23153d-c0e2-4a72-a25a-67a4e61766bb" containerName="payload-processing"
Nov 03 01:45:50.087271 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087242    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c23153d-c0e2-4a72-a25a-67a4e61766bb" containerName="payload-processing"
Nov 03 01:45:50.087271 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087261    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" containerName="istio-proxy"
Nov 03 01:45:50.087271 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087266    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087274    5508 cpu_manager.go:414] "containerMap: removing container" podUID="53333dac-248a-4cf1-ac30-c1962d83d8b9" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087278    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53333dac-248a-4cf1-ac30-c1962d83d8b9" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087290    5508 cpu_manager.go:414] "containerMap: removing container" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087294    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087304    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a07f4321-ff64-453e-a103-29676b99d594" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087308    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07f4321-ff64-453e-a103-29676b99d594" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087314    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5556c0df-cf71-4528-ab71-71ef67e9ac3e" containerName="payload-pre-processing"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087319    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5556c0df-cf71-4528-ab71-71ef67e9ac3e" containerName="payload-pre-processing"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087339    5508 cpu_manager.go:414] "containerMap: removing container" podUID="af9497a7-0c68-4222-b517-fc47fa1bfaf3" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087343    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9497a7-0c68-4222-b517-fc47fa1bfaf3" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087370    5508 cpu_manager.go:414] "containerMap: removing container" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087374    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087458    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087468    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07f4321-ff64-453e-a103-29676b99d594" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087474    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5556c0df-cf71-4528-ab71-71ef67e9ac3e" containerName="payload-pre-processing"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087484    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9497a7-0c68-4222-b517-fc47fa1bfaf3" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087490    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087495    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="beeaa3a5-ee87-4129-a879-5f5f591ec803" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087499    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="83390757-f37d-4198-b4ac-bc05f260fa82" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087506    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c23153d-c0e2-4a72-a25a-67a4e61766bb" containerName="payload-processing"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087515    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97fbb6e-fcbc-4ab8-9d3f-7299753616c2" containerName="istio-proxy"
Nov 03 01:45:50.087569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.087521    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="53333dac-248a-4cf1-ac30-c1962d83d8b9" containerName="istio-proxy"
Nov 03 01:45:50.092882 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.092859    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.122413 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.122344    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-podinfo\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.122618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.122450    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istiod-ca-cert\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.122618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.122602    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-token\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.122704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.122643    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-envoy\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.122704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.122680    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-socket\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.122779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.122724    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f2kl\" (UniqueName: \"kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-kube-api-access-2f2kl\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.122815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.122789    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-data\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.122855 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.122841    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-credential-socket\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.123027 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.123009    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-certs\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.142815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.142763    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-109234-openshift-default-dockercfg-vfxdr\""
Nov 03 01:45:50.148982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.148955    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"]
Nov 03 01:45:50.251507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.249470    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-certs\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.251507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.249570    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-podinfo\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.251507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.249620    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istiod-ca-cert\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.251507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.249681    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-token\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.251507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.249705    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-envoy\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.251507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.249738    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-socket\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.251507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.249780    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2kl\" (UniqueName: \"kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-kube-api-access-2f2kl\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.251507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.249845    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-data\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.251507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.249899    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-credential-socket\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.251507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.250645    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-credential-socket\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.251507 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.250904    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-certs\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.290339 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.290296    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istiod-ca-cert\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.296293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.296248    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-envoy\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.296815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.296787    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-socket\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.297754 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.297723    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-data\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.298678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.298653    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-podinfo\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.350086 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.350025    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-token\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.354510 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.354474    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2kl\" (UniqueName: \"kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-kube-api-access-2f2kl\") pod \"e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") " pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.408433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.408397    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:50.696458 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.696425    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2" path="/var/lib/kubelet/pods/1bff3dbb-1081-4c8d-a558-d8db9ea4a2c2/volumes"
Nov 03 01:45:50.696883 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.696868    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5556c0df-cf71-4528-ab71-71ef67e9ac3e" path="/var/lib/kubelet/pods/5556c0df-cf71-4528-ab71-71ef67e9ac3e/volumes"
Nov 03 01:45:50.697391 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.697376    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07f4321-ff64-453e-a103-29676b99d594" path="/var/lib/kubelet/pods/a07f4321-ff64-453e-a103-29676b99d594/volumes"
Nov 03 01:45:50.845146 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.845108    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"]
Nov 03 01:45:50.851206 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:45:50.851177    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e96865_e9ca_4040_bfe1_27d0c745ca85.slice/crio-5acfd0e3f03be014cc6c6569a63b21196b9d9ef71799f01ba6288f090c085068 WatchSource:0}: Error finding container 5acfd0e3f03be014cc6c6569a63b21196b9d9ef71799f01ba6288f090c085068: Status 404 returned error can't find the container with id 5acfd0e3f03be014cc6c6569a63b21196b9d9ef71799f01ba6288f090c085068
Nov 03 01:45:50.854007 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.853974    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:45:50.854082 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.854044    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:45:50.854082 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:50.854076    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:45:51.616371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:51.608918    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" event={"ID":"d7e96865-e9ca-4040-bfe1-27d0c745ca85","Type":"ContainerStarted","Data":"bb0bcf5c0f6c82e59ce87cd1f457690c80d9e800c602b4af418edacea49e6d3e"}
Nov 03 01:45:51.616371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:51.608988    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" event={"ID":"d7e96865-e9ca-4040-bfe1-27d0c745ca85","Type":"ContainerStarted","Data":"5acfd0e3f03be014cc6c6569a63b21196b9d9ef71799f01ba6288f090c085068"}
Nov 03 01:45:51.724872 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:51.724801    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" podStartSLOduration=1.724781219 podStartE2EDuration="1.724781219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:51.724221382 +0000 UTC m=+1631.149827862" watchObservedRunningTime="2026-11-03 01:45:51.724781219 +0000 UTC m=+1631.150387684"
Nov 03 01:45:52.436369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:52.425520    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:52.447641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:52.447299    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:52.617966 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:52.617935    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:52.619129 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:52.619105    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:45:53.988256 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:53.988214    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:45:53.988774 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:53.988283    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n" podUID="0896f72f-e85a-4e3c-9a25-876e0a78f4b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:45:54.619090 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.619062    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:45:54.642904 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.642876    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:45:54.664179 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.664146    5508 generic.go:356] "Generic (PLEG): container finished" podID="a0e25b47-aaa7-4776-8757-c242e4441da2" containerID="8abaf68244b8e32bf3c6292a0848f65a116b03320e1e13b7eae49869635964d0" exitCode=0
Nov 03 01:45:54.664374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.664214    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"
Nov 03 01:45:54.664374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.664231    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" event={"ID":"a0e25b47-aaa7-4776-8757-c242e4441da2","Type":"ContainerDied","Data":"8abaf68244b8e32bf3c6292a0848f65a116b03320e1e13b7eae49869635964d0"}
Nov 03 01:45:54.664374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.664282    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz" event={"ID":"a0e25b47-aaa7-4776-8757-c242e4441da2","Type":"ContainerDied","Data":"40993728c0504d0cdbc4c184f7b393d7cda010443619cc1c702d80e1ba7ea12d"}
Nov 03 01:45:54.664374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.664303    5508 scope.go:122] "RemoveContainer" containerID="8abaf68244b8e32bf3c6292a0848f65a116b03320e1e13b7eae49869635964d0"
Nov 03 01:45:54.666305 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.666281    5508 generic.go:356] "Generic (PLEG): container finished" podID="db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" containerID="c0aea2a4031aeadd85ae40f7e66f415f583bfe21b702349e1018985b6d0d7394" exitCode=0
Nov 03 01:45:54.666538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.666518    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"
Nov 03 01:45:54.666700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.666679    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl" event={"ID":"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1","Type":"ContainerDied","Data":"c0aea2a4031aeadd85ae40f7e66f415f583bfe21b702349e1018985b6d0d7394"}
Nov 03 01:45:54.666760 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.666727    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl" event={"ID":"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1","Type":"ContainerDied","Data":"33145e5155b35cc3b37f167f7afbb21c42ffe7631dcdc298f1c38c6b5e52b015"}
Nov 03 01:45:54.668539 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.668518    5508 generic.go:356] "Generic (PLEG): container finished" podID="0896f72f-e85a-4e3c-9a25-876e0a78f4b6" containerID="a9f22498b67b15f50e5544c5254b1cfc167e12ec401715705adc27f5bc973a08" exitCode=0
Nov 03 01:45:54.669499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.669476    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n" event={"ID":"0896f72f-e85a-4e3c-9a25-876e0a78f4b6","Type":"ContainerDied","Data":"a9f22498b67b15f50e5544c5254b1cfc167e12ec401715705adc27f5bc973a08"}
Nov 03 01:45:54.678025 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.678007    5508 scope.go:122] "RemoveContainer" containerID="8abaf68244b8e32bf3c6292a0848f65a116b03320e1e13b7eae49869635964d0"
Nov 03 01:45:54.678383 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:54.678322    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8abaf68244b8e32bf3c6292a0848f65a116b03320e1e13b7eae49869635964d0\": container with ID starting with 8abaf68244b8e32bf3c6292a0848f65a116b03320e1e13b7eae49869635964d0 not found: ID does not exist" containerID="8abaf68244b8e32bf3c6292a0848f65a116b03320e1e13b7eae49869635964d0"
Nov 03 01:45:54.678462 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.678400    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8abaf68244b8e32bf3c6292a0848f65a116b03320e1e13b7eae49869635964d0"} err="failed to get container status \"8abaf68244b8e32bf3c6292a0848f65a116b03320e1e13b7eae49869635964d0\": rpc error: code = NotFound desc = could not find container \"8abaf68244b8e32bf3c6292a0848f65a116b03320e1e13b7eae49869635964d0\": container with ID starting with 8abaf68244b8e32bf3c6292a0848f65a116b03320e1e13b7eae49869635964d0 not found: ID does not exist"
Nov 03 01:45:54.678462 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.678425    5508 scope.go:122] "RemoveContainer" containerID="c0aea2a4031aeadd85ae40f7e66f415f583bfe21b702349e1018985b6d0d7394"
Nov 03 01:45:54.690390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.690331    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:45:54.698152 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.698130    5508 scope.go:122] "RemoveContainer" containerID="c0aea2a4031aeadd85ae40f7e66f415f583bfe21b702349e1018985b6d0d7394"
Nov 03 01:45:54.700487 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:45:54.700459    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0aea2a4031aeadd85ae40f7e66f415f583bfe21b702349e1018985b6d0d7394\": container with ID starting with c0aea2a4031aeadd85ae40f7e66f415f583bfe21b702349e1018985b6d0d7394 not found: ID does not exist" containerID="c0aea2a4031aeadd85ae40f7e66f415f583bfe21b702349e1018985b6d0d7394"
Nov 03 01:45:54.700593 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.700500    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0aea2a4031aeadd85ae40f7e66f415f583bfe21b702349e1018985b6d0d7394"} err="failed to get container status \"c0aea2a4031aeadd85ae40f7e66f415f583bfe21b702349e1018985b6d0d7394\": rpc error: code = NotFound desc = could not find container \"c0aea2a4031aeadd85ae40f7e66f415f583bfe21b702349e1018985b6d0d7394\": container with ID starting with c0aea2a4031aeadd85ae40f7e66f415f583bfe21b702349e1018985b6d0d7394 not found: ID does not exist"
Nov 03 01:45:54.731411 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731379    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-credential-socket\") pod \"a0e25b47-aaa7-4776-8757-c242e4441da2\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") "
Nov 03 01:45:54.731580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731428    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-data\") pod \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") "
Nov 03 01:45:54.731580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731486    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-data\") pod \"a0e25b47-aaa7-4776-8757-c242e4441da2\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") "
Nov 03 01:45:54.731580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731521    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-data\") pod \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") "
Nov 03 01:45:54.731580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731553    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-token\" (UniqueName: \"kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-token\") pod \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") "
Nov 03 01:45:54.731789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731582    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-socket\") pod \"a0e25b47-aaa7-4776-8757-c242e4441da2\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") "
Nov 03 01:45:54.731789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731627    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-kube-api-access-29n6n\" (UniqueName: \"kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-kube-api-access-29n6n\") pod \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") "
Nov 03 01:45:54.731789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731708    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-certs\") pod \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") "
Nov 03 01:45:54.731789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731716    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-credential-socket" pod "a0e25b47-aaa7-4776-8757-c242e4441da2" (UID: "a0e25b47-aaa7-4776-8757-c242e4441da2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.731789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731698    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-data" pod "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" (UID: "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.731789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731737    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-token\" (UniqueName: \"kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-token\") pod \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") "
Nov 03 01:45:54.731789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731759    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-data" pod "a0e25b47-aaa7-4776-8757-c242e4441da2" (UID: "a0e25b47-aaa7-4776-8757-c242e4441da2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.732120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731791    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-kube-api-access-f82j8\" (UniqueName: \"kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-kube-api-access-f82j8\") pod \"a0e25b47-aaa7-4776-8757-c242e4441da2\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") "
Nov 03 01:45:54.732120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731828    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istiod-ca-cert\") pod \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") "
Nov 03 01:45:54.732120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731892    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-podinfo\") pod \"a0e25b47-aaa7-4776-8757-c242e4441da2\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") "
Nov 03 01:45:54.732120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731940    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-kube-api-access-7mvq4\" (UniqueName: \"kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-kube-api-access-7mvq4\") pod \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") "
Nov 03 01:45:54.732120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731978    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-socket\") pod \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") "
Nov 03 01:45:54.732120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.731999    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-data" pod "0896f72f-e85a-4e3c-9a25-876e0a78f4b6" (UID: "0896f72f-e85a-4e3c-9a25-876e0a78f4b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.732120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732042    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-certs\") pod \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") "
Nov 03 01:45:54.732120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732101    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-certs\") pod \"a0e25b47-aaa7-4776-8757-c242e4441da2\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") "
Nov 03 01:45:54.732530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732136    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-credential-socket\") pod \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") "
Nov 03 01:45:54.732530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732170    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-socket\") pod \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") "
Nov 03 01:45:54.732530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732204    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-token\" (UniqueName: \"kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-token\") pod \"a0e25b47-aaa7-4776-8757-c242e4441da2\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") "
Nov 03 01:45:54.732530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732241    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-podinfo\") pod \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") "
Nov 03 01:45:54.732530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732269    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istiod-ca-cert" pod "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" (UID: "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:54.732530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732275    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-envoy\") pod \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") "
Nov 03 01:45:54.732530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732310    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-envoy\") pod \"a0e25b47-aaa7-4776-8757-c242e4441da2\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") "
Nov 03 01:45:54.732530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732383    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-envoy\") pod \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") "
Nov 03 01:45:54.732530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732416    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e25b47-aaa7-4776-8757-c242e4441da2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e25b47-aaa7-4776-8757-c242e4441da2-istiod-ca-cert\") pod \"a0e25b47-aaa7-4776-8757-c242e4441da2\" (UID: \"a0e25b47-aaa7-4776-8757-c242e4441da2\") "
Nov 03 01:45:54.732530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732464    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-certs" pod "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" (UID: "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.732530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732474    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istiod-ca-cert\") pod \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\" (UID: \"0896f72f-e85a-4e3c-9a25-876e0a78f4b6\") "
Nov 03 01:45:54.732530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732502    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-podinfo\") pod \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") "
Nov 03 01:45:54.733103 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732539    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-certs" pod "a0e25b47-aaa7-4776-8757-c242e4441da2" (UID: "a0e25b47-aaa7-4776-8757-c242e4441da2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.733103 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732547    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-credential-socket\") pod \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\" (UID: \"db2b4424-3a33-4f86-bb7d-28b6cf5f33c1\") "
Nov 03 01:45:54.733103 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732745    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-socket" pod "a0e25b47-aaa7-4776-8757-c242e4441da2" (UID: "a0e25b47-aaa7-4776-8757-c242e4441da2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.733103 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.732966    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-socket" pod "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" (UID: "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.733325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.733138    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-credential-socket" pod "0896f72f-e85a-4e3c-9a25-876e0a78f4b6" (UID: "0896f72f-e85a-4e3c-9a25-876e0a78f4b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.733695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.733516    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.733695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.733541    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.733695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.733558    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.733695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.733573    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.733695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.733589    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.733695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.733603    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.733695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.733619    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.733695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.733632    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.733695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.733645    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.733695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.733658    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.734926 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.734897    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-token" pod "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" (UID: "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:54.736268 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.735983    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-podinfo" pod "a0e25b47-aaa7-4776-8757-c242e4441da2" (UID: "a0e25b47-aaa7-4776-8757-c242e4441da2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:45:54.736268 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.736062    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-socket" pod "0896f72f-e85a-4e3c-9a25-876e0a78f4b6" (UID: "0896f72f-e85a-4e3c-9a25-876e0a78f4b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.736822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.736587    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-credential-socket" pod "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" (UID: "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.736890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.736867    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-kube-api-access-f82j8" pod "a0e25b47-aaa7-4776-8757-c242e4441da2" (UID: "a0e25b47-aaa7-4776-8757-c242e4441da2"). InnerVolumeSpecName "kube-api-access-f82j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:54.736952 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.736916    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e25b47-aaa7-4776-8757-c242e4441da2-istiod-ca-cert" pod "a0e25b47-aaa7-4776-8757-c242e4441da2" (UID: "a0e25b47-aaa7-4776-8757-c242e4441da2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:54.737033 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.737015    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-envoy" pod "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" (UID: "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.737104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.737086    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istiod-ca-cert" pod "0896f72f-e85a-4e3c-9a25-876e0a78f4b6" (UID: "0896f72f-e85a-4e3c-9a25-876e0a78f4b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:45:54.737207 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.737184    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-certs" pod "0896f72f-e85a-4e3c-9a25-876e0a78f4b6" (UID: "0896f72f-e85a-4e3c-9a25-876e0a78f4b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.737370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.737335    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-token" pod "0896f72f-e85a-4e3c-9a25-876e0a78f4b6" (UID: "0896f72f-e85a-4e3c-9a25-876e0a78f4b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:54.737747 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.737723    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-kube-api-access-29n6n" pod "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" (UID: "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1"). InnerVolumeSpecName "kube-api-access-29n6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:54.737881 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.737863    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-podinfo" pod "0896f72f-e85a-4e3c-9a25-876e0a78f4b6" (UID: "0896f72f-e85a-4e3c-9a25-876e0a78f4b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:45:54.737959 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.737861    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-podinfo" pod "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" (UID: "db2b4424-3a33-4f86-bb7d-28b6cf5f33c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:45:54.737998 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.737967    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-envoy" pod "a0e25b47-aaa7-4776-8757-c242e4441da2" (UID: "a0e25b47-aaa7-4776-8757-c242e4441da2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.738999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.738982    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-envoy" pod "0896f72f-e85a-4e3c-9a25-876e0a78f4b6" (UID: "0896f72f-e85a-4e3c-9a25-876e0a78f4b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:45:54.739426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.739404    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-token" pod "a0e25b47-aaa7-4776-8757-c242e4441da2" (UID: "a0e25b47-aaa7-4776-8757-c242e4441da2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:54.739728 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.739710    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-kube-api-access-7mvq4" pod "0896f72f-e85a-4e3c-9a25-876e0a78f4b6" (UID: "0896f72f-e85a-4e3c-9a25-876e0a78f4b6"). InnerVolumeSpecName "kube-api-access-7mvq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:45:54.835065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835023    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29n6n\" (UniqueName: \"kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-kube-api-access-29n6n\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835055    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835065    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f82j8\" (UniqueName: \"kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-kube-api-access-f82j8\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835074    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835083    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mvq4\" (UniqueName: \"kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-kube-api-access-7mvq4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835091    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835099    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835109    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835117    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835124    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835134    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e25b47-aaa7-4776-8757-c242e4441da2-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835141    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835149    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e25b47-aaa7-4776-8757-c242e4441da2-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835157    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835164    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835174    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:54.835465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:54.835181    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0896f72f-e85a-4e3c-9a25-876e0a78f4b6-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:45:55.176646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:55.176606    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"]
Nov 03 01:45:55.249892 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:55.249811    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-5mpzl"]
Nov 03 01:45:55.296104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:55.296067    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"]
Nov 03 01:45:55.302242 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:55.302208    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-79mtz"]
Nov 03 01:45:55.687790 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:55.687756    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n" event={"ID":"0896f72f-e85a-4e3c-9a25-876e0a78f4b6","Type":"ContainerDied","Data":"293d11ed5f9bbbcce13a33aba0a82e65b4aec4f260dbcb056ae4487b1f9e4539"}
Nov 03 01:45:55.687965 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:55.687803    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"
Nov 03 01:45:55.687965 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:55.687811    5508 scope.go:122] "RemoveContainer" containerID="a9f22498b67b15f50e5544c5254b1cfc167e12ec401715705adc27f5bc973a08"
Nov 03 01:45:55.843735 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:55.843704    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"]
Nov 03 01:45:55.850004 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:55.849976    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-82q5n"]
Nov 03 01:45:56.693091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:56.693057    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0896f72f-e85a-4e3c-9a25-876e0a78f4b6" path="/var/lib/kubelet/pods/0896f72f-e85a-4e3c-9a25-876e0a78f4b6/volumes"
Nov 03 01:45:56.693651 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:56.693635    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e25b47-aaa7-4776-8757-c242e4441da2" path="/var/lib/kubelet/pods/a0e25b47-aaa7-4776-8757-c242e4441da2/volumes"
Nov 03 01:45:56.694092 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:56.694078    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" path="/var/lib/kubelet/pods/db2b4424-3a33-4f86-bb7d-28b6cf5f33c1/volumes"
Nov 03 01:45:58.112427 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.112380    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"]
Nov 03 01:45:58.113296 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.113255    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a0e25b47-aaa7-4776-8757-c242e4441da2" containerName="istio-proxy"
Nov 03 01:45:58.113368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.113297    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e25b47-aaa7-4776-8757-c242e4441da2" containerName="istio-proxy"
Nov 03 01:45:58.113368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.113322    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0896f72f-e85a-4e3c-9a25-876e0a78f4b6" containerName="istio-proxy"
Nov 03 01:45:58.113368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.113330    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0896f72f-e85a-4e3c-9a25-876e0a78f4b6" containerName="istio-proxy"
Nov 03 01:45:58.113490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.113370    5508 cpu_manager.go:414] "containerMap: removing container" podUID="db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" containerName="istio-proxy"
Nov 03 01:45:58.113490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.113378    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" containerName="istio-proxy"
Nov 03 01:45:58.113590 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.113578    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e25b47-aaa7-4776-8757-c242e4441da2" containerName="istio-proxy"
Nov 03 01:45:58.113633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.113597    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0896f72f-e85a-4e3c-9a25-876e0a78f4b6" containerName="istio-proxy"
Nov 03 01:45:58.113633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.113608    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2b4424-3a33-4f86-bb7d-28b6cf5f33c1" containerName="istio-proxy"
Nov 03 01:45:58.118535 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.118512    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"
Nov 03 01:45:58.168649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.168620    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"]
Nov 03 01:45:58.214370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.213534    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t64kg\" (UniqueName: \"kubernetes.io/projected/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-kube-api-access-t64kg\") pod \"payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5\" (UID: \"f91d5bd0-8541-4a70-a6e7-06db5c88e2df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"
Nov 03 01:45:58.214370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.213632    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5\" (UID: \"f91d5bd0-8541-4a70-a6e7-06db5c88e2df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"
Nov 03 01:45:58.261370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.260646    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-109234-dockercfg-2tlqc\""
Nov 03 01:45:58.261370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.260760    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-109234\""
Nov 03 01:45:58.328809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.321370    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t64kg\" (UniqueName: \"kubernetes.io/projected/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-kube-api-access-t64kg\") pod \"payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5\" (UID: \"f91d5bd0-8541-4a70-a6e7-06db5c88e2df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"
Nov 03 01:45:58.328809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.321474    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5\" (UID: \"f91d5bd0-8541-4a70-a6e7-06db5c88e2df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"
Nov 03 01:45:58.328809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.322443    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5\" (UID: \"f91d5bd0-8541-4a70-a6e7-06db5c88e2df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"
Nov 03 01:45:58.475950 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.423673    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"]
Nov 03 01:45:58.483839 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.483807    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t64kg\" (UniqueName: \"kubernetes.io/projected/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-kube-api-access-t64kg\") pod \"payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5\" (UID: \"f91d5bd0-8541-4a70-a6e7-06db5c88e2df\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"
Nov 03 01:45:58.506403 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.498657    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"
Nov 03 01:45:58.517190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.517153    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"]
Nov 03 01:45:58.524681 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.524640    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-729b7\" (UniqueName: \"kubernetes.io/projected/305befcb-aafc-48ca-8469-49697d33d49e-kube-api-access-729b7\") pod \"payload-processing-e2e-api-b-109234-5676c44df9-9b6mt\" (UID: \"305befcb-aafc-48ca-8469-49697d33d49e\") " pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"
Nov 03 01:45:58.524977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.524957    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/305befcb-aafc-48ca-8469-49697d33d49e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-109234-5676c44df9-9b6mt\" (UID: \"305befcb-aafc-48ca-8469-49697d33d49e\") " pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"
Nov 03 01:45:58.525131 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.525116    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"]
Nov 03 01:45:58.525526 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.525496    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" podUID="916e646a-6298-47da-a869-a863a7c410fc" containerName="istio-proxy" containerID="cri-o://7418960d236921a254f8e659a62ddd6b5a69a9c69f793205450fb9a3e31bb4da" gracePeriod=30
Nov 03 01:45:58.717371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.716774    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-729b7\" (UniqueName: \"kubernetes.io/projected/305befcb-aafc-48ca-8469-49697d33d49e-kube-api-access-729b7\") pod \"payload-processing-e2e-api-b-109234-5676c44df9-9b6mt\" (UID: \"305befcb-aafc-48ca-8469-49697d33d49e\") " pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"
Nov 03 01:45:58.717371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.717031    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/305befcb-aafc-48ca-8469-49697d33d49e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-109234-5676c44df9-9b6mt\" (UID: \"305befcb-aafc-48ca-8469-49697d33d49e\") " pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"
Nov 03 01:45:58.726475 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.718000    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/305befcb-aafc-48ca-8469-49697d33d49e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-109234-5676c44df9-9b6mt\" (UID: \"305befcb-aafc-48ca-8469-49697d33d49e\") " pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"
Nov 03 01:45:58.775385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.767604    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"
Nov 03 01:45:58.805569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.805502    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-729b7\" (UniqueName: \"kubernetes.io/projected/305befcb-aafc-48ca-8469-49697d33d49e-kube-api-access-729b7\") pod \"payload-processing-e2e-api-b-109234-5676c44df9-9b6mt\" (UID: \"305befcb-aafc-48ca-8469-49697d33d49e\") " pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"
Nov 03 01:45:58.849360 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:58.849315    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"
Nov 03 01:45:59.004785 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:59.004704    5508 patch_prober.go:29] interesting pod/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:45:59.004785 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:59.004762    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" podUID="916e646a-6298-47da-a869-a863a7c410fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:45:59.184967 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:59.184933    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"]
Nov 03 01:45:59.190199 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:45:59.190165    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91d5bd0_8541_4a70_a6e7_06db5c88e2df.slice/crio-6843bb5e28e8dfcb36c7791caa2c01aa7ea067116b577a675548e9263d969008 WatchSource:0}: Error finding container 6843bb5e28e8dfcb36c7791caa2c01aa7ea067116b577a675548e9263d969008: Status 404 returned error can't find the container with id 6843bb5e28e8dfcb36c7791caa2c01aa7ea067116b577a675548e9263d969008
Nov 03 01:45:59.299552 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:59.299479    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"]
Nov 03 01:45:59.306515 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:45:59.306473    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305befcb_aafc_48ca_8469_49697d33d49e.slice/crio-2a8033f851325ce2bf136c3d1536237a08bde74c2f9fdfbf756e9c53aad1c89a WatchSource:0}: Error finding container 2a8033f851325ce2bf136c3d1536237a08bde74c2f9fdfbf756e9c53aad1c89a: Status 404 returned error can't find the container with id 2a8033f851325ce2bf136c3d1536237a08bde74c2f9fdfbf756e9c53aad1c89a
Nov 03 01:45:59.820672 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:59.820624    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5" event={"ID":"f91d5bd0-8541-4a70-a6e7-06db5c88e2df","Type":"ContainerStarted","Data":"09409ce142a65af149a7ecf1d2cc343995ac6ddc7b915e7bbd9e90a4ef22c93c"}
Nov 03 01:45:59.820874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:59.820725    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5" event={"ID":"f91d5bd0-8541-4a70-a6e7-06db5c88e2df","Type":"ContainerStarted","Data":"6843bb5e28e8dfcb36c7791caa2c01aa7ea067116b577a675548e9263d969008"}
Nov 03 01:45:59.820874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:59.820865    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"
Nov 03 01:45:59.822512 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:59.822469    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt" event={"ID":"305befcb-aafc-48ca-8469-49697d33d49e","Type":"ContainerStarted","Data":"cbc04ddac35bd879adfa55f4ddc01731273f7670aee4f8bd3163b714738edb76"}
Nov 03 01:45:59.822618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:59.822525    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt" event={"ID":"305befcb-aafc-48ca-8469-49697d33d49e","Type":"ContainerStarted","Data":"2a8033f851325ce2bf136c3d1536237a08bde74c2f9fdfbf756e9c53aad1c89a"}
Nov 03 01:45:59.822618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:59.822566    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"
Nov 03 01:45:59.931476 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:45:59.931406    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5" podStartSLOduration=1.9313871649999999 podStartE2EDuration="1.931387165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:59.9288327 +0000 UTC m=+1639.354439185" watchObservedRunningTime="2026-11-03 01:45:59.931387165 +0000 UTC m=+1639.356993644"
Nov 03 01:46:00.085860 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:00.085732    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt" podStartSLOduration=2.085708757 podStartE2EDuration="2.085708757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:46:00.068317866 +0000 UTC m=+1639.493924335" watchObservedRunningTime="2026-11-03 01:46:00.085708757 +0000 UTC m=+1639.511315223"
Nov 03 01:46:03.827034 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.827008    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:46:03.877678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.876486    5508 generic.go:356] "Generic (PLEG): container finished" podID="916e646a-6298-47da-a869-a863a7c410fc" containerID="7418960d236921a254f8e659a62ddd6b5a69a9c69f793205450fb9a3e31bb4da" exitCode=0
Nov 03 01:46:03.877678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.876581    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" event={"ID":"916e646a-6298-47da-a869-a863a7c410fc","Type":"ContainerDied","Data":"7418960d236921a254f8e659a62ddd6b5a69a9c69f793205450fb9a3e31bb4da"}
Nov 03 01:46:03.877678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.876709    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"
Nov 03 01:46:03.877678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.876732    5508 scope.go:122] "RemoveContainer" containerID="7418960d236921a254f8e659a62ddd6b5a69a9c69f793205450fb9a3e31bb4da"
Nov 03 01:46:03.877678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.876717    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr" event={"ID":"916e646a-6298-47da-a869-a863a7c410fc","Type":"ContainerDied","Data":"7ca1b3805eb01abf8a0d21852148d2c248d88faf94794b26cb401bd92832ed87"}
Nov 03 01:46:03.903475 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.903444    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-certs\") pod \"916e646a-6298-47da-a869-a863a7c410fc\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") "
Nov 03 01:46:03.904085 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.904064    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-data\") pod \"916e646a-6298-47da-a869-a863a7c410fc\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") "
Nov 03 01:46:03.904255 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.904240    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/916e646a-6298-47da-a869-a863a7c410fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916e646a-6298-47da-a869-a863a7c410fc-istio-podinfo\") pod \"916e646a-6298-47da-a869-a863a7c410fc\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") "
Nov 03 01:46:03.904440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.904423    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-socket\") pod \"916e646a-6298-47da-a869-a863a7c410fc\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") "
Nov 03 01:46:03.904579 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.904554    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-istio-token\" (UniqueName: \"kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-istio-token\") pod \"916e646a-6298-47da-a869-a863a7c410fc\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") "
Nov 03 01:46:03.904677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.904619    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-envoy\") pod \"916e646a-6298-47da-a869-a863a7c410fc\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") "
Nov 03 01:46:03.904677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.904668    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-credential-socket\") pod \"916e646a-6298-47da-a869-a863a7c410fc\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") "
Nov 03 01:46:03.904784 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.904716    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-data" pod "916e646a-6298-47da-a869-a863a7c410fc" (UID: "916e646a-6298-47da-a869-a863a7c410fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:03.904784 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.904761    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-kube-api-access-c5j4s\" (UniqueName: \"kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-kube-api-access-c5j4s\") pod \"916e646a-6298-47da-a869-a863a7c410fc\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") "
Nov 03 01:46:03.904892 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.904871    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916e646a-6298-47da-a869-a863a7c410fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916e646a-6298-47da-a869-a863a7c410fc-istiod-ca-cert\") pod \"916e646a-6298-47da-a869-a863a7c410fc\" (UID: \"916e646a-6298-47da-a869-a863a7c410fc\") "
Nov 03 01:46:03.904966 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.904943    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-certs" pod "916e646a-6298-47da-a869-a863a7c410fc" (UID: "916e646a-6298-47da-a869-a863a7c410fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:03.906827 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.906803    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:03.906919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.906829    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:03.909336 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.909311    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-istio-token" pod "916e646a-6298-47da-a869-a863a7c410fc" (UID: "916e646a-6298-47da-a869-a863a7c410fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:46:03.909591 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.909564    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-credential-socket" pod "916e646a-6298-47da-a869-a863a7c410fc" (UID: "916e646a-6298-47da-a869-a863a7c410fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:03.909669 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.909630    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916e646a-6298-47da-a869-a863a7c410fc-istiod-ca-cert" pod "916e646a-6298-47da-a869-a863a7c410fc" (UID: "916e646a-6298-47da-a869-a863a7c410fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:46:03.909829 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.909809    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-socket" pod "916e646a-6298-47da-a869-a863a7c410fc" (UID: "916e646a-6298-47da-a869-a863a7c410fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:03.913770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.913746    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-kube-api-access-c5j4s" pod "916e646a-6298-47da-a869-a863a7c410fc" (UID: "916e646a-6298-47da-a869-a863a7c410fc"). InnerVolumeSpecName "kube-api-access-c5j4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:46:03.915273 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.915251    5508 scope.go:122] "RemoveContainer" containerID="368a31001287facf53ada9a06e66ae576581db3f0d1627395416ad4f393644e6"
Nov 03 01:46:03.925404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.925375    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-envoy" pod "916e646a-6298-47da-a869-a863a7c410fc" (UID: "916e646a-6298-47da-a869-a863a7c410fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:03.926756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.926730    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/916e646a-6298-47da-a869-a863a7c410fc-istio-podinfo" pod "916e646a-6298-47da-a869-a863a7c410fc" (UID: "916e646a-6298-47da-a869-a863a7c410fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:46:03.943822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.943796    5508 scope.go:122] "RemoveContainer" containerID="7418960d236921a254f8e659a62ddd6b5a69a9c69f793205450fb9a3e31bb4da"
Nov 03 01:46:03.944167 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:46:03.944140    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7418960d236921a254f8e659a62ddd6b5a69a9c69f793205450fb9a3e31bb4da\": container with ID starting with 7418960d236921a254f8e659a62ddd6b5a69a9c69f793205450fb9a3e31bb4da not found: ID does not exist" containerID="7418960d236921a254f8e659a62ddd6b5a69a9c69f793205450fb9a3e31bb4da"
Nov 03 01:46:03.944244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.944181    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7418960d236921a254f8e659a62ddd6b5a69a9c69f793205450fb9a3e31bb4da"} err="failed to get container status \"7418960d236921a254f8e659a62ddd6b5a69a9c69f793205450fb9a3e31bb4da\": rpc error: code = NotFound desc = could not find container \"7418960d236921a254f8e659a62ddd6b5a69a9c69f793205450fb9a3e31bb4da\": container with ID starting with 7418960d236921a254f8e659a62ddd6b5a69a9c69f793205450fb9a3e31bb4da not found: ID does not exist"
Nov 03 01:46:03.944244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.944207    5508 scope.go:122] "RemoveContainer" containerID="368a31001287facf53ada9a06e66ae576581db3f0d1627395416ad4f393644e6"
Nov 03 01:46:03.944556 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:46:03.944539    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368a31001287facf53ada9a06e66ae576581db3f0d1627395416ad4f393644e6\": container with ID starting with 368a31001287facf53ada9a06e66ae576581db3f0d1627395416ad4f393644e6 not found: ID does not exist" containerID="368a31001287facf53ada9a06e66ae576581db3f0d1627395416ad4f393644e6"
Nov 03 01:46:03.944629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:03.944562    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368a31001287facf53ada9a06e66ae576581db3f0d1627395416ad4f393644e6"} err="failed to get container status \"368a31001287facf53ada9a06e66ae576581db3f0d1627395416ad4f393644e6\": rpc error: code = NotFound desc = could not find container \"368a31001287facf53ada9a06e66ae576581db3f0d1627395416ad4f393644e6\": container with ID starting with 368a31001287facf53ada9a06e66ae576581db3f0d1627395416ad4f393644e6 not found: ID does not exist"
Nov 03 01:46:04.009368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:04.007927    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/916e646a-6298-47da-a869-a863a7c410fc-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:04.009368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:04.007963    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:04.009368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:04.007979    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:04.009368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:04.007992    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:04.009368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:04.008006    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/916e646a-6298-47da-a869-a863a7c410fc-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:04.009368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:04.008019    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5j4s\" (UniqueName: \"kubernetes.io/projected/916e646a-6298-47da-a869-a863a7c410fc-kube-api-access-c5j4s\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:04.009368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:04.008033    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/916e646a-6298-47da-a869-a863a7c410fc-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:04.297241 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:04.297201    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"]
Nov 03 01:46:04.300757 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:04.300728    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-012d31-openshift-default-7cfdd6685-l2krr"]
Nov 03 01:46:04.694897 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:04.694863    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916e646a-6298-47da-a869-a863a7c410fc" path="/var/lib/kubelet/pods/916e646a-6298-47da-a869-a863a7c410fc/volumes"
Nov 03 01:46:10.670589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:10.670556    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:46:10.835221 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:10.835189    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"
Nov 03 01:46:10.835428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:10.835259    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"
Nov 03 01:46:13.812823 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:46:13.812788    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:46:20.512983 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:20.512936    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"]
Nov 03 01:46:20.513493 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:20.513191    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf" podUID="b9367258-844a-46ac-a685-3be44068b879" containerName="istio-proxy" containerID="cri-o://c9113027a1dcdd7d89d9b4b65e2b846948fa08db9fe5d7c92f5a755313dd9b5f" gracePeriod=30
Nov 03 01:46:20.561432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:20.561396    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"]
Nov 03 01:46:20.561704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:20.561659    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5" podUID="895db3e8-7f0d-40fd-9572-66659bc5887e" containerName="istio-proxy" containerID="cri-o://d2010e4421c35147212188611d8eb5df1513bb770214fbe52fb94138d04aa9c2" gracePeriod=30
Nov 03 01:46:21.048269 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:21.048234    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"]
Nov 03 01:46:21.053116 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:21.053082    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a13fd92-gfwn4"]
Nov 03 01:46:22.061032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:22.060995    5508 generic.go:356] "Generic (PLEG): container finished" podID="f930409c-7286-4a06-bb5f-93015858b97e" containerID="482a99fc36dbee9778bb283835a122cf1dd20ef208186f85d2297de03c2df9de" exitCode=0
Nov 03 01:46:22.061591 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:22.061062    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b" event={"ID":"f930409c-7286-4a06-bb5f-93015858b97e","Type":"ContainerDied","Data":"482a99fc36dbee9778bb283835a122cf1dd20ef208186f85d2297de03c2df9de"}
Nov 03 01:46:22.061682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:22.061633    5508 scope.go:122] "RemoveContainer" containerID="482a99fc36dbee9778bb283835a122cf1dd20ef208186f85d2297de03c2df9de"
Nov 03 01:46:22.063487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:22.063455    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:46:22.063575 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:22.063542    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:46:22.063623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:22.063610    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:46:22.692745 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:22.692715    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53364c1e-cdcc-4f5f-9f62-434b6823b4ef" path="/var/lib/kubelet/pods/53364c1e-cdcc-4f5f-9f62-434b6823b4ef/volumes"
Nov 03 01:46:23.081051 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:23.080959    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b" event={"ID":"f930409c-7286-4a06-bb5f-93015858b97e","Type":"ContainerStarted","Data":"b149004c5b6a469ab40d04d78926d4ccd813ff21eaed14513b2144d8393854c5"}
Nov 03 01:46:23.442218 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:23.442185    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:46:23.442419 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:23.442359    5508 patch_prober.go:29] interesting pod/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 01:46:23.442419 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:23.442403    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b" podUID="f930409c-7286-4a06-bb5f-93015858b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 01:46:24.442238 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:24.442204    5508 patch_prober.go:29] interesting pod/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 01:46:24.442691 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:24.442259    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b" podUID="f930409c-7286-4a06-bb5f-93015858b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 01:46:25.442658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:25.442623    5508 patch_prober.go:29] interesting pod/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 01:46:25.443027 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:25.442699    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b" podUID="f930409c-7286-4a06-bb5f-93015858b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 01:46:26.009612 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.009541    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:46:26.082649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.082609    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:46:26.108587 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.108550    5508 generic.go:356] "Generic (PLEG): container finished" podID="895db3e8-7f0d-40fd-9572-66659bc5887e" containerID="d2010e4421c35147212188611d8eb5df1513bb770214fbe52fb94138d04aa9c2" exitCode=0
Nov 03 01:46:26.108772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.108616    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"
Nov 03 01:46:26.108772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.108669    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5" event={"ID":"895db3e8-7f0d-40fd-9572-66659bc5887e","Type":"ContainerDied","Data":"d2010e4421c35147212188611d8eb5df1513bb770214fbe52fb94138d04aa9c2"}
Nov 03 01:46:26.108772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.108704    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5" event={"ID":"895db3e8-7f0d-40fd-9572-66659bc5887e","Type":"ContainerDied","Data":"34af4f095287b57befec88241c0963ff70341dff1399f6335c5c4d09eaa29b00"}
Nov 03 01:46:26.108772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.108725    5508 scope.go:122] "RemoveContainer" containerID="d2010e4421c35147212188611d8eb5df1513bb770214fbe52fb94138d04aa9c2"
Nov 03 01:46:26.110294 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.110273    5508 generic.go:356] "Generic (PLEG): container finished" podID="b9367258-844a-46ac-a685-3be44068b879" containerID="c9113027a1dcdd7d89d9b4b65e2b846948fa08db9fe5d7c92f5a755313dd9b5f" exitCode=0
Nov 03 01:46:26.110420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.110319    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"
Nov 03 01:46:26.110420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.110364    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf" event={"ID":"b9367258-844a-46ac-a685-3be44068b879","Type":"ContainerDied","Data":"c9113027a1dcdd7d89d9b4b65e2b846948fa08db9fe5d7c92f5a755313dd9b5f"}
Nov 03 01:46:26.110420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.110405    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf" event={"ID":"b9367258-844a-46ac-a685-3be44068b879","Type":"ContainerDied","Data":"db7683f8a3d938611e632b1c79d96f174fae72b3d66f63ddd604942dbf56ad55"}
Nov 03 01:46:26.130565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.130539    5508 scope.go:122] "RemoveContainer" containerID="d2010e4421c35147212188611d8eb5df1513bb770214fbe52fb94138d04aa9c2"
Nov 03 01:46:26.130995 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:46:26.130975    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2010e4421c35147212188611d8eb5df1513bb770214fbe52fb94138d04aa9c2\": container with ID starting with d2010e4421c35147212188611d8eb5df1513bb770214fbe52fb94138d04aa9c2 not found: ID does not exist" containerID="d2010e4421c35147212188611d8eb5df1513bb770214fbe52fb94138d04aa9c2"
Nov 03 01:46:26.131060 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.131006    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2010e4421c35147212188611d8eb5df1513bb770214fbe52fb94138d04aa9c2"} err="failed to get container status \"d2010e4421c35147212188611d8eb5df1513bb770214fbe52fb94138d04aa9c2\": rpc error: code = NotFound desc = could not find container \"d2010e4421c35147212188611d8eb5df1513bb770214fbe52fb94138d04aa9c2\": container with ID starting with d2010e4421c35147212188611d8eb5df1513bb770214fbe52fb94138d04aa9c2 not found: ID does not exist"
Nov 03 01:46:26.131060 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.131024    5508 scope.go:122] "RemoveContainer" containerID="c9113027a1dcdd7d89d9b4b65e2b846948fa08db9fe5d7c92f5a755313dd9b5f"
Nov 03 01:46:26.134911 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.134886    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-istio-token\" (UniqueName: \"kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-istio-token\") pod \"b9367258-844a-46ac-a685-3be44068b879\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") "
Nov 03 01:46:26.135043 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.134925    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9367258-844a-46ac-a685-3be44068b879-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9367258-844a-46ac-a685-3be44068b879-istiod-ca-cert\") pod \"b9367258-844a-46ac-a685-3be44068b879\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") "
Nov 03 01:46:26.135043 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.134997    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-credential-socket\") pod \"b9367258-844a-46ac-a685-3be44068b879\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") "
Nov 03 01:46:26.135157 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135062    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-certs\") pod \"895db3e8-7f0d-40fd-9572-66659bc5887e\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") "
Nov 03 01:46:26.135157 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135091    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-kube-api-access-gsh92\" (UniqueName: \"kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-kube-api-access-gsh92\") pod \"b9367258-844a-46ac-a685-3be44068b879\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") "
Nov 03 01:46:26.135253 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135196    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-token\" (UniqueName: \"kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-token\") pod \"895db3e8-7f0d-40fd-9572-66659bc5887e\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") "
Nov 03 01:46:26.135421 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135401    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-data\") pod \"895db3e8-7f0d-40fd-9572-66659bc5887e\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") "
Nov 03 01:46:26.135506 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135461    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-podinfo\") pod \"895db3e8-7f0d-40fd-9572-66659bc5887e\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") "
Nov 03 01:46:26.135571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135509    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-envoy\") pod \"895db3e8-7f0d-40fd-9572-66659bc5887e\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") "
Nov 03 01:46:26.135571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135538    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-certs\") pod \"b9367258-844a-46ac-a685-3be44068b879\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") "
Nov 03 01:46:26.135672 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135586    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-credential-socket\") pod \"895db3e8-7f0d-40fd-9572-66659bc5887e\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") "
Nov 03 01:46:26.135672 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135635    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-socket\") pod \"b9367258-844a-46ac-a685-3be44068b879\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") "
Nov 03 01:46:26.135776 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135505    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-credential-socket" pod "b9367258-844a-46ac-a685-3be44068b879" (UID: "b9367258-844a-46ac-a685-3be44068b879"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:26.135776 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135722    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-data" pod "895db3e8-7f0d-40fd-9572-66659bc5887e" (UID: "895db3e8-7f0d-40fd-9572-66659bc5887e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:26.135776 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135727    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-envoy\") pod \"b9367258-844a-46ac-a685-3be44068b879\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") "
Nov 03 01:46:26.135776 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135772    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-kube-api-access-d2nbd\" (UniqueName: \"kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-kube-api-access-d2nbd\") pod \"895db3e8-7f0d-40fd-9572-66659bc5887e\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") "
Nov 03 01:46:26.135986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135804    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895db3e8-7f0d-40fd-9572-66659bc5887e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895db3e8-7f0d-40fd-9572-66659bc5887e-istiod-ca-cert\") pod \"895db3e8-7f0d-40fd-9572-66659bc5887e\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") "
Nov 03 01:46:26.135986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135840    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9367258-844a-46ac-a685-3be44068b879-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9367258-844a-46ac-a685-3be44068b879-istio-podinfo\") pod \"b9367258-844a-46ac-a685-3be44068b879\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") "
Nov 03 01:46:26.135986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135889    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-data\") pod \"b9367258-844a-46ac-a685-3be44068b879\" (UID: \"b9367258-844a-46ac-a685-3be44068b879\") "
Nov 03 01:46:26.135986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135913    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-credential-socket" pod "895db3e8-7f0d-40fd-9572-66659bc5887e" (UID: "895db3e8-7f0d-40fd-9572-66659bc5887e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:26.135986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.135929    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-socket\") pod \"895db3e8-7f0d-40fd-9572-66659bc5887e\" (UID: \"895db3e8-7f0d-40fd-9572-66659bc5887e\") "
Nov 03 01:46:26.136246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.136082    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-certs" pod "b9367258-844a-46ac-a685-3be44068b879" (UID: "b9367258-844a-46ac-a685-3be44068b879"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:26.136437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.136414    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-certs" pod "895db3e8-7f0d-40fd-9572-66659bc5887e" (UID: "895db3e8-7f0d-40fd-9572-66659bc5887e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:26.136585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.136570    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.136663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.136591    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.136727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.136705    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.136727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.136702    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-socket" pod "895db3e8-7f0d-40fd-9572-66659bc5887e" (UID: "895db3e8-7f0d-40fd-9572-66659bc5887e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:26.136727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.136722    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.136879 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.136736    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.137210 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.136932    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-socket" pod "b9367258-844a-46ac-a685-3be44068b879" (UID: "b9367258-844a-46ac-a685-3be44068b879"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:26.137210 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.137126    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-data" pod "b9367258-844a-46ac-a685-3be44068b879" (UID: "b9367258-844a-46ac-a685-3be44068b879"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:26.137210 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.137185    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895db3e8-7f0d-40fd-9572-66659bc5887e-istiod-ca-cert" pod "895db3e8-7f0d-40fd-9572-66659bc5887e" (UID: "895db3e8-7f0d-40fd-9572-66659bc5887e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:46:26.137678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.137525    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-istio-token" pod "b9367258-844a-46ac-a685-3be44068b879" (UID: "b9367258-844a-46ac-a685-3be44068b879"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:46:26.137783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.137767    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9367258-844a-46ac-a685-3be44068b879-istiod-ca-cert" pod "b9367258-844a-46ac-a685-3be44068b879" (UID: "b9367258-844a-46ac-a685-3be44068b879"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:46:26.139065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.138886    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-podinfo" pod "895db3e8-7f0d-40fd-9572-66659bc5887e" (UID: "895db3e8-7f0d-40fd-9572-66659bc5887e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:46:26.139065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.139012    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-kube-api-access-gsh92" pod "b9367258-844a-46ac-a685-3be44068b879" (UID: "b9367258-844a-46ac-a685-3be44068b879"). InnerVolumeSpecName "kube-api-access-gsh92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:46:26.139065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.139040    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-token" pod "895db3e8-7f0d-40fd-9572-66659bc5887e" (UID: "895db3e8-7f0d-40fd-9572-66659bc5887e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:46:26.139439 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.139413    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9367258-844a-46ac-a685-3be44068b879-istio-podinfo" pod "b9367258-844a-46ac-a685-3be44068b879" (UID: "b9367258-844a-46ac-a685-3be44068b879"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:46:26.139866 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.139846    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-envoy" pod "b9367258-844a-46ac-a685-3be44068b879" (UID: "b9367258-844a-46ac-a685-3be44068b879"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:26.139919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.139848    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-envoy" pod "895db3e8-7f0d-40fd-9572-66659bc5887e" (UID: "895db3e8-7f0d-40fd-9572-66659bc5887e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:46:26.140547 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.140525    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-kube-api-access-d2nbd" pod "895db3e8-7f0d-40fd-9572-66659bc5887e" (UID: "895db3e8-7f0d-40fd-9572-66659bc5887e"). InnerVolumeSpecName "kube-api-access-d2nbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:46:26.144518 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.144498    5508 scope.go:122] "RemoveContainer" containerID="c9113027a1dcdd7d89d9b4b65e2b846948fa08db9fe5d7c92f5a755313dd9b5f"
Nov 03 01:46:26.144873 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:46:26.144854    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9113027a1dcdd7d89d9b4b65e2b846948fa08db9fe5d7c92f5a755313dd9b5f\": container with ID starting with c9113027a1dcdd7d89d9b4b65e2b846948fa08db9fe5d7c92f5a755313dd9b5f not found: ID does not exist" containerID="c9113027a1dcdd7d89d9b4b65e2b846948fa08db9fe5d7c92f5a755313dd9b5f"
Nov 03 01:46:26.144928 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.144883    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9113027a1dcdd7d89d9b4b65e2b846948fa08db9fe5d7c92f5a755313dd9b5f"} err="failed to get container status \"c9113027a1dcdd7d89d9b4b65e2b846948fa08db9fe5d7c92f5a755313dd9b5f\": rpc error: code = NotFound desc = could not find container \"c9113027a1dcdd7d89d9b4b65e2b846948fa08db9fe5d7c92f5a755313dd9b5f\": container with ID starting with c9113027a1dcdd7d89d9b4b65e2b846948fa08db9fe5d7c92f5a755313dd9b5f not found: ID does not exist"
Nov 03 01:46:26.238421 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.238386    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.238421 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.238422    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.238701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.238438    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2nbd\" (UniqueName: \"kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-kube-api-access-d2nbd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.238701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.238456    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895db3e8-7f0d-40fd-9572-66659bc5887e-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.238701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.238472    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9367258-844a-46ac-a685-3be44068b879-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.238701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.238485    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9367258-844a-46ac-a685-3be44068b879-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.238701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.238510    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.238701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.238524    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.238701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.238537    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9367258-844a-46ac-a685-3be44068b879-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.238701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.238551    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsh92\" (UniqueName: \"kubernetes.io/projected/b9367258-844a-46ac-a685-3be44068b879-kube-api-access-gsh92\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.238701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.238564    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.238701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.238579    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.238701 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.238592    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895db3e8-7f0d-40fd-9572-66659bc5887e-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:26.442231 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.442203    5508 patch_prober.go:29] interesting pod/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 01:46:26.442388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.442256    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b" podUID="f930409c-7286-4a06-bb5f-93015858b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 01:46:26.530730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.530693    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"]
Nov 03 01:46:26.534167 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.534140    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-xf8cf"]
Nov 03 01:46:26.636248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.636208    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"]
Nov 03 01:46:26.642193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.642166    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-gc8n5"]
Nov 03 01:46:26.692753 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.692678    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895db3e8-7f0d-40fd-9572-66659bc5887e" path="/var/lib/kubelet/pods/895db3e8-7f0d-40fd-9572-66659bc5887e/volumes"
Nov 03 01:46:26.693226 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:26.693168    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9367258-844a-46ac-a685-3be44068b879" path="/var/lib/kubelet/pods/b9367258-844a-46ac-a685-3be44068b879/volumes"
Nov 03 01:46:27.442725 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:27.442691    5508 patch_prober.go:29] interesting pod/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Nov 03 01:46:27.442897 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:27.442748    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b" podUID="f930409c-7286-4a06-bb5f-93015858b97e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Nov 03 01:46:28.447215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.447183    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:46:28.447661 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.447627    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:46:28.448306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.448293    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:46:28.673596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.673557    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"]
Nov 03 01:46:28.674466 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.674449    5508 cpu_manager.go:414] "containerMap: removing container" podUID="916e646a-6298-47da-a869-a863a7c410fc" containerName="istio-proxy"
Nov 03 01:46:28.674544 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.674469    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916e646a-6298-47da-a869-a863a7c410fc" containerName="istio-proxy"
Nov 03 01:46:28.674544 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.674512    5508 cpu_manager.go:414] "containerMap: removing container" podUID="895db3e8-7f0d-40fd-9572-66659bc5887e" containerName="istio-proxy"
Nov 03 01:46:28.674544 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.674519    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895db3e8-7f0d-40fd-9572-66659bc5887e" containerName="istio-proxy"
Nov 03 01:46:28.674639 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.674544    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b9367258-844a-46ac-a685-3be44068b879" containerName="istio-proxy"
Nov 03 01:46:28.674639 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.674551    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9367258-844a-46ac-a685-3be44068b879" containerName="istio-proxy"
Nov 03 01:46:28.674699 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.674657    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="895db3e8-7f0d-40fd-9572-66659bc5887e" containerName="istio-proxy"
Nov 03 01:46:28.674699 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.674668    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="916e646a-6298-47da-a869-a863a7c410fc" containerName="istio-proxy"
Nov 03 01:46:28.674699 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.674680    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9367258-844a-46ac-a685-3be44068b879" containerName="istio-proxy"
Nov 03 01:46:28.678956 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.678938    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"
Nov 03 01:46:28.754523 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.754422    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"]
Nov 03 01:46:28.778080 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.778029    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:46:28.778080 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.778065    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:46:28.778330 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.778309    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:46:28.778522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.778396    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:46:28.792734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.792697    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwgcj\" (UniqueName: \"kubernetes.io/projected/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-kube-api-access-fwgcj\") pod \"maas-api-revoke-sub-ca6bf3f1-lgdr4\" (UID: \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"
Nov 03 01:46:28.792910 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.792884    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca6bf3f1-lgdr4\" (UID: \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"
Nov 03 01:46:28.793068 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.792961    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca6bf3f1-lgdr4\" (UID: \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"
Nov 03 01:46:28.894625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.894595    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwgcj\" (UniqueName: \"kubernetes.io/projected/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-kube-api-access-fwgcj\") pod \"maas-api-revoke-sub-ca6bf3f1-lgdr4\" (UID: \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"
Nov 03 01:46:28.894800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.894688    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca6bf3f1-lgdr4\" (UID: \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"
Nov 03 01:46:28.894800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.894727    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca6bf3f1-lgdr4\" (UID: \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"
Nov 03 01:46:28.895372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.895331    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca6bf3f1-lgdr4\" (UID: \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"
Nov 03 01:46:28.896946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.896922    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca6bf3f1-lgdr4\" (UID: \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"
Nov 03 01:46:28.952452 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.952419    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwgcj\" (UniqueName: \"kubernetes.io/projected/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-kube-api-access-fwgcj\") pod \"maas-api-revoke-sub-ca6bf3f1-lgdr4\" (UID: \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"
Nov 03 01:46:28.993521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:28.993483    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"
Nov 03 01:46:29.375134 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:29.375048    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"]
Nov 03 01:46:29.382072 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:46:29.382042    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod109c5dc1_dc70_4db4_9563_65f27bc8e6b7.slice/crio-4d7aa26d4dea798540373b940c3a07c14cda5102a1a9cc1118459f997e79461c WatchSource:0}: Error finding container 4d7aa26d4dea798540373b940c3a07c14cda5102a1a9cc1118459f997e79461c: Status 404 returned error can't find the container with id 4d7aa26d4dea798540373b940c3a07c14cda5102a1a9cc1118459f997e79461c
Nov 03 01:46:30.148081 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:30.148044    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4" event={"ID":"109c5dc1-dc70-4db4-9563-65f27bc8e6b7","Type":"ContainerStarted","Data":"4c6587d9f4198080c233977e6219a2605e9f813228cc126ee40647aeeddde026"}
Nov 03 01:46:30.148518 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:30.148113    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4" event={"ID":"109c5dc1-dc70-4db4-9563-65f27bc8e6b7","Type":"ContainerStarted","Data":"4d7aa26d4dea798540373b940c3a07c14cda5102a1a9cc1118459f997e79461c"}
Nov 03 01:46:30.258062 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:30.258010    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4" podStartSLOduration=2.257993922 podStartE2EDuration="2.257993922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:46:30.25612559 +0000 UTC m=+1669.681732053" watchObservedRunningTime="2026-11-03 01:46:30.257993922 +0000 UTC m=+1669.683600386"
Nov 03 01:46:31.157206 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:31.157175    5508 generic.go:356] "Generic (PLEG): container finished" podID="109c5dc1-dc70-4db4-9563-65f27bc8e6b7" containerID="4c6587d9f4198080c233977e6219a2605e9f813228cc126ee40647aeeddde026" exitCode=0
Nov 03 01:46:31.157631 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:31.157270    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4" event={"ID":"109c5dc1-dc70-4db4-9563-65f27bc8e6b7","Type":"ContainerDied","Data":"4c6587d9f4198080c233977e6219a2605e9f813228cc126ee40647aeeddde026"}
Nov 03 01:46:32.326935 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:32.326901    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"
Nov 03 01:46:32.447451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:32.447334    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-kube-api-access-fwgcj\" (UniqueName: \"kubernetes.io/projected/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-kube-api-access-fwgcj\") pod \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\" (UID: \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\") "
Nov 03 01:46:32.447623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:32.447528    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-openshift-ca\") pod \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\" (UID: \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\") "
Nov 03 01:46:32.447623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:32.447587    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-xks-ca\") pod \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\" (UID: \"109c5dc1-dc70-4db4-9563-65f27bc8e6b7\") "
Nov 03 01:46:32.447960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:32.447933    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-openshift-ca" pod "109c5dc1-dc70-4db4-9563-65f27bc8e6b7" (UID: "109c5dc1-dc70-4db4-9563-65f27bc8e6b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:46:32.448331 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:32.448298    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:32.449647 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:32.449624    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-kube-api-access-fwgcj" pod "109c5dc1-dc70-4db4-9563-65f27bc8e6b7" (UID: "109c5dc1-dc70-4db4-9563-65f27bc8e6b7"). InnerVolumeSpecName "kube-api-access-fwgcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:46:32.450307 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:32.450283    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-xks-ca" pod "109c5dc1-dc70-4db4-9563-65f27bc8e6b7" (UID: "109c5dc1-dc70-4db4-9563-65f27bc8e6b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:46:32.550199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:32.550164    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwgcj\" (UniqueName: \"kubernetes.io/projected/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-kube-api-access-fwgcj\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:32.550199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:32.550195    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/109c5dc1-dc70-4db4-9563-65f27bc8e6b7-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:33.175995 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:33.175959    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4" event={"ID":"109c5dc1-dc70-4db4-9563-65f27bc8e6b7","Type":"ContainerDied","Data":"4d7aa26d4dea798540373b940c3a07c14cda5102a1a9cc1118459f997e79461c"}
Nov 03 01:46:33.176213 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:33.176024    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d7aa26d4dea798540373b940c3a07c14cda5102a1a9cc1118459f997e79461c"
Nov 03 01:46:33.176213 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:33.175965    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"
Nov 03 01:46:42.327571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:42.326433    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:46:42.418563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:42.418507    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 03 01:46:42.432707 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:42.428646    5508 scope.go:122] "RemoveContainer" containerID="b2db03d474eeb41f8e23113c59ad748eb7e51a7c734cbb56ed086bf78652bef8"
Nov 03 01:46:42.451472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:42.451428    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:46:42.498630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:42.493530    5508 scope.go:122] "RemoveContainer" containerID="9d27ad3774f857bc2c4cfa3759d1aefa44e83b9601156a63dae3fcbd665919df"
Nov 03 01:46:42.532962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:42.532629    5508 scope.go:122] "RemoveContainer" containerID="47f60cabfa207eabfe30a6ae02ad120e0aef92bda9a8f6fa0f516febe34ad6e9"
Nov 03 01:46:42.548118 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:42.548093    5508 scope.go:122] "RemoveContainer" containerID="dbf078c877bac15827a95997b71a25c96d3ed3bf004c451953062be4cfba81fc"
Nov 03 01:46:42.680819 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:42.680784    5508 scope.go:122] "RemoveContainer" containerID="104a0f259c0f7a018ac257a74f33323e3d42bf8252e5252ca6b1f4abd4cc2f0c"
Nov 03 01:46:42.761215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:42.760460    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Nov 03 01:46:42.789682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:42.789537    5508 scope.go:122] "RemoveContainer" containerID="fce1be557495f130b639e14c1e6af19dbdfe0627def3263a3cfcc6fb6720a0c3"
Nov 03 01:46:43.057694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:43.057661    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"]
Nov 03 01:46:43.197444 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:43.196408    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"]
Nov 03 01:46:43.422692 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:43.422642    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:46:43.423429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:43.423376    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:46:43.680604 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:43.680039    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 01:46:43.957847 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:46:43.945547    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:46:48.063794 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.063724    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72" podUID="0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e" containerName="payload-processing" containerID="cri-o://500a916a59c09aa7f38de8cc54a793d1ac67697eaf6a89708336d70fca2fbbd4" gracePeriod=25
Nov 03 01:46:48.196987 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.196930    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25" podUID="1f34bcde-6db6-468d-b473-4f41de31149a" containerName="payload-pre-processing" containerID="cri-o://50b78491a51e210ca463c1223b765ff9bacf4f4ca4466c3c79fbd6803a9e93dd" gracePeriod=25
Nov 03 01:46:48.355777 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.355533    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"
Nov 03 01:46:48.389905 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.388840    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-kube-api-access-qgd62\" (UniqueName: \"kubernetes.io/projected/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-kube-api-access-qgd62\") pod \"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e\" (UID: \"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e\") "
Nov 03 01:46:48.389905 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.389157    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-plugins-config-volume\") pod \"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e\" (UID: \"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e\") "
Nov 03 01:46:48.390736 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.390315    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-plugins-config-volume" pod "0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e" (UID: "0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:46:48.404418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.397553    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-kube-api-access-qgd62" pod "0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e" (UID: "0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e"). InnerVolumeSpecName "kube-api-access-qgd62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:46:48.480626 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.480599    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"
Nov 03 01:46:48.491906 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.491872    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:48.491906 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.491907    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgd62\" (UniqueName: \"kubernetes.io/projected/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e-kube-api-access-qgd62\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:48.592396 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.592305    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f34bcde-6db6-468d-b473-4f41de31149a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f34bcde-6db6-468d-b473-4f41de31149a-plugins-config-volume\") pod \"1f34bcde-6db6-468d-b473-4f41de31149a\" (UID: \"1f34bcde-6db6-468d-b473-4f41de31149a\") "
Nov 03 01:46:48.592550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.592485    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f34bcde-6db6-468d-b473-4f41de31149a-kube-api-access-mr4jw\" (UniqueName: \"kubernetes.io/projected/1f34bcde-6db6-468d-b473-4f41de31149a-kube-api-access-mr4jw\") pod \"1f34bcde-6db6-468d-b473-4f41de31149a\" (UID: \"1f34bcde-6db6-468d-b473-4f41de31149a\") "
Nov 03 01:46:48.592725 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.592703    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f34bcde-6db6-468d-b473-4f41de31149a-plugins-config-volume" pod "1f34bcde-6db6-468d-b473-4f41de31149a" (UID: "1f34bcde-6db6-468d-b473-4f41de31149a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:46:48.593059 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.593039    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f34bcde-6db6-468d-b473-4f41de31149a-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:48.594746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.594721    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f34bcde-6db6-468d-b473-4f41de31149a-kube-api-access-mr4jw" pod "1f34bcde-6db6-468d-b473-4f41de31149a" (UID: "1f34bcde-6db6-468d-b473-4f41de31149a"). InnerVolumeSpecName "kube-api-access-mr4jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:46:48.694232 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.694185    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr4jw\" (UniqueName: \"kubernetes.io/projected/1f34bcde-6db6-468d-b473-4f41de31149a-kube-api-access-mr4jw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:46:48.807537 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.807502    5508 generic.go:356] "Generic (PLEG): container finished" podID="0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e" containerID="500a916a59c09aa7f38de8cc54a793d1ac67697eaf6a89708336d70fca2fbbd4" exitCode=0
Nov 03 01:46:48.807722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.807565    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"
Nov 03 01:46:48.807722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.807589    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72" event={"ID":"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e","Type":"ContainerDied","Data":"500a916a59c09aa7f38de8cc54a793d1ac67697eaf6a89708336d70fca2fbbd4"}
Nov 03 01:46:48.807722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.807654    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72" event={"ID":"0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e","Type":"ContainerDied","Data":"5cd01ff7136250f9af5a24b57c5dd843d258d160a5221faeebd0f440a903d46d"}
Nov 03 01:46:48.807722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.807685    5508 scope.go:122] "RemoveContainer" containerID="500a916a59c09aa7f38de8cc54a793d1ac67697eaf6a89708336d70fca2fbbd4"
Nov 03 01:46:48.809302 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.809230    5508 generic.go:356] "Generic (PLEG): container finished" podID="1f34bcde-6db6-468d-b473-4f41de31149a" containerID="50b78491a51e210ca463c1223b765ff9bacf4f4ca4466c3c79fbd6803a9e93dd" exitCode=0
Nov 03 01:46:48.809437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.809313    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"
Nov 03 01:46:48.809437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.809334    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25" event={"ID":"1f34bcde-6db6-468d-b473-4f41de31149a","Type":"ContainerDied","Data":"50b78491a51e210ca463c1223b765ff9bacf4f4ca4466c3c79fbd6803a9e93dd"}
Nov 03 01:46:48.809437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.809422    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25" event={"ID":"1f34bcde-6db6-468d-b473-4f41de31149a","Type":"ContainerDied","Data":"b636508f7957f5d24223181640a06f603c6ae0f90ab733190908d5757cd9c767"}
Nov 03 01:46:48.819461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.819436    5508 scope.go:122] "RemoveContainer" containerID="500a916a59c09aa7f38de8cc54a793d1ac67697eaf6a89708336d70fca2fbbd4"
Nov 03 01:46:48.819785 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:46:48.819768    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500a916a59c09aa7f38de8cc54a793d1ac67697eaf6a89708336d70fca2fbbd4\": container with ID starting with 500a916a59c09aa7f38de8cc54a793d1ac67697eaf6a89708336d70fca2fbbd4 not found: ID does not exist" containerID="500a916a59c09aa7f38de8cc54a793d1ac67697eaf6a89708336d70fca2fbbd4"
Nov 03 01:46:48.819850 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.819792    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500a916a59c09aa7f38de8cc54a793d1ac67697eaf6a89708336d70fca2fbbd4"} err="failed to get container status \"500a916a59c09aa7f38de8cc54a793d1ac67697eaf6a89708336d70fca2fbbd4\": rpc error: code = NotFound desc = could not find container \"500a916a59c09aa7f38de8cc54a793d1ac67697eaf6a89708336d70fca2fbbd4\": container with ID starting with 500a916a59c09aa7f38de8cc54a793d1ac67697eaf6a89708336d70fca2fbbd4 not found: ID does not exist"
Nov 03 01:46:48.819850 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.819808    5508 scope.go:122] "RemoveContainer" containerID="50b78491a51e210ca463c1223b765ff9bacf4f4ca4466c3c79fbd6803a9e93dd"
Nov 03 01:46:48.830517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.830482    5508 scope.go:122] "RemoveContainer" containerID="50b78491a51e210ca463c1223b765ff9bacf4f4ca4466c3c79fbd6803a9e93dd"
Nov 03 01:46:48.830839 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:46:48.830807    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b78491a51e210ca463c1223b765ff9bacf4f4ca4466c3c79fbd6803a9e93dd\": container with ID starting with 50b78491a51e210ca463c1223b765ff9bacf4f4ca4466c3c79fbd6803a9e93dd not found: ID does not exist" containerID="50b78491a51e210ca463c1223b765ff9bacf4f4ca4466c3c79fbd6803a9e93dd"
Nov 03 01:46:48.830918 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.830849    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b78491a51e210ca463c1223b765ff9bacf4f4ca4466c3c79fbd6803a9e93dd"} err="failed to get container status \"50b78491a51e210ca463c1223b765ff9bacf4f4ca4466c3c79fbd6803a9e93dd\": rpc error: code = NotFound desc = could not find container \"50b78491a51e210ca463c1223b765ff9bacf4f4ca4466c3c79fbd6803a9e93dd\": container with ID starting with 50b78491a51e210ca463c1223b765ff9bacf4f4ca4466c3c79fbd6803a9e93dd not found: ID does not exist"
Nov 03 01:46:48.964980 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.964934    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"]
Nov 03 01:46:48.970622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:48.970594    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-948470-6cb5fcbc8c-qhf25"]
Nov 03 01:46:49.072363 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:49.072323    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"]
Nov 03 01:46:49.076815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:49.076783    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-948470-cb959cb87-d2w72"]
Nov 03 01:46:50.702964 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:50.702932    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e" path="/var/lib/kubelet/pods/0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e/volumes"
Nov 03 01:46:50.703390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:50.703376    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f34bcde-6db6-468d-b473-4f41de31149a" path="/var/lib/kubelet/pods/1f34bcde-6db6-468d-b473-4f41de31149a/volumes"
Nov 03 01:46:55.477073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.477036    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"]
Nov 03 01:46:55.477726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.477709    5508 cpu_manager.go:414] "containerMap: removing container" podUID="1f34bcde-6db6-468d-b473-4f41de31149a" containerName="payload-pre-processing"
Nov 03 01:46:55.477773 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.477727    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f34bcde-6db6-468d-b473-4f41de31149a" containerName="payload-pre-processing"
Nov 03 01:46:55.477773 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.477758    5508 cpu_manager.go:414] "containerMap: removing container" podUID="916e646a-6298-47da-a869-a863a7c410fc" containerName="istio-proxy"
Nov 03 01:46:55.477773 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.477764    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916e646a-6298-47da-a869-a863a7c410fc" containerName="istio-proxy"
Nov 03 01:46:55.477880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.477774    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e" containerName="payload-processing"
Nov 03 01:46:55.477880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.477779    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e" containerName="payload-processing"
Nov 03 01:46:55.477880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.477800    5508 cpu_manager.go:414] "containerMap: removing container" podUID="109c5dc1-dc70-4db4-9563-65f27bc8e6b7" containerName="revoke-keys"
Nov 03 01:46:55.477880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.477805    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="109c5dc1-dc70-4db4-9563-65f27bc8e6b7" containerName="revoke-keys"
Nov 03 01:46:55.478008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.477896    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="916e646a-6298-47da-a869-a863a7c410fc" containerName="istio-proxy"
Nov 03 01:46:55.478008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.477907    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="109c5dc1-dc70-4db4-9563-65f27bc8e6b7" containerName="revoke-keys"
Nov 03 01:46:55.478008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.477916    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f34bcde-6db6-468d-b473-4f41de31149a" containerName="payload-pre-processing"
Nov 03 01:46:55.478008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.477926    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf383e9-e6fa-4cd8-b46a-8b3924c9fb8e" containerName="payload-processing"
Nov 03 01:46:55.482825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.482810    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:55.578677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.578612    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 01:46:55.578677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.578638    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 01:46:55.578677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.578662    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-109234-serving-cert\""
Nov 03 01:46:55.578677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.578616    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 01:46:55.578958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.578736    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sxqdt\""
Nov 03 01:46:55.581488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.581464    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e0d034-4613-46a3-b1a9-0b6d12ef8801-openshift-service-ca\") pod \"maas-api-e2e-api-b-109234-b64dcd57c-v8qzr\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:55.581624 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.581555    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-metrics-certs\") pod \"maas-api-e2e-api-b-109234-b64dcd57c-v8qzr\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:55.581713 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.581692    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-maas-api-tls\") pod \"maas-api-e2e-api-b-109234-b64dcd57c-v8qzr\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:55.581796 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.581769    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l699h\" (UniqueName: \"kubernetes.io/projected/48e0d034-4613-46a3-b1a9-0b6d12ef8801-kube-api-access-l699h\") pod \"maas-api-e2e-api-b-109234-b64dcd57c-v8qzr\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:55.582538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.582519    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"]
Nov 03 01:46:55.682858 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.682827    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e0d034-4613-46a3-b1a9-0b6d12ef8801-openshift-service-ca\") pod \"maas-api-e2e-api-b-109234-b64dcd57c-v8qzr\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:55.683056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.682930    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-metrics-certs\") pod \"maas-api-e2e-api-b-109234-b64dcd57c-v8qzr\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:55.683056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.682988    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-maas-api-tls\") pod \"maas-api-e2e-api-b-109234-b64dcd57c-v8qzr\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:55.683056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.683010    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l699h\" (UniqueName: \"kubernetes.io/projected/48e0d034-4613-46a3-b1a9-0b6d12ef8801-kube-api-access-l699h\") pod \"maas-api-e2e-api-b-109234-b64dcd57c-v8qzr\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:55.683219 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:46:55.683099    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-109234-serving-cert: secret "maas-api-e2e-api-b-109234-serving-cert" not found
Nov 03 01:46:55.683219 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:46:55.683172    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-maas-api-tls podName:48e0d034-4613-46a3-b1a9-0b6d12ef8801 nodeName:}" failed. No retries permitted until 2026-11-03 01:46:56.183149392 +0000 UTC m=+1695.608755849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-maas-api-tls") pod "maas-api-e2e-api-b-109234-b64dcd57c-v8qzr" (UID: "48e0d034-4613-46a3-b1a9-0b6d12ef8801") : secret "maas-api-e2e-api-b-109234-serving-cert" not found
Nov 03 01:46:55.683610 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.683589    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e0d034-4613-46a3-b1a9-0b6d12ef8801-openshift-service-ca\") pod \"maas-api-e2e-api-b-109234-b64dcd57c-v8qzr\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:55.685547 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.685517    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-metrics-certs\") pod \"maas-api-e2e-api-b-109234-b64dcd57c-v8qzr\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:55.741177 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:55.741099    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l699h\" (UniqueName: \"kubernetes.io/projected/48e0d034-4613-46a3-b1a9-0b6d12ef8801-kube-api-access-l699h\") pod \"maas-api-e2e-api-b-109234-b64dcd57c-v8qzr\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:56.198664 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:56.198622    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-maas-api-tls\") pod \"maas-api-e2e-api-b-109234-b64dcd57c-v8qzr\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:56.201194 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:56.201162    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-maas-api-tls\") pod \"maas-api-e2e-api-b-109234-b64dcd57c-v8qzr\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:56.396568 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:56.396533    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:46:56.776183 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:56.776099    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"]
Nov 03 01:46:56.782731 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:46:56.782698    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e0d034_4613_46a3_b1a9_0b6d12ef8801.slice/crio-690d28564667c16cdbc139b7e656d01d0735dc441e76067850c60ad567c47362 WatchSource:0}: Error finding container 690d28564667c16cdbc139b7e656d01d0735dc441e76067850c60ad567c47362: Status 404 returned error can't find the container with id 690d28564667c16cdbc139b7e656d01d0735dc441e76067850c60ad567c47362
Nov 03 01:46:56.784229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:56.784208    5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:46:56.875208 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:56.875174    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr" event={"ID":"48e0d034-4613-46a3-b1a9-0b6d12ef8801","Type":"ContainerStarted","Data":"690d28564667c16cdbc139b7e656d01d0735dc441e76067850c60ad567c47362"}
Nov 03 01:46:58.884690 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:58.884658    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"]
Nov 03 01:46:58.889478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:58.889454    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.041835 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.041795    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.042012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.041865    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.042012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.041976    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4wln\" (UniqueName: \"kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-kube-api-access-w4wln\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.042100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.042050    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.042154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.042112    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.042154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.042145    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.042229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.042164    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.042229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.042202    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.042293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.042254    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.096467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.096419    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"]
Nov 03 01:46:59.144065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.143974    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.144241 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.144076    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.144241 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.144162    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.144241 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.144195    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4wln\" (UniqueName: \"kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-kube-api-access-w4wln\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.144442 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.144284    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.144442 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.144364    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.144442 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.144400    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.144604 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.144447    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.144604 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.144475    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.144604 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.144538    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.144755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.144641    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.144853 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.144830    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.144914 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.144890    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.144914 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.144900    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.147190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.147168    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.147292 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.147276    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.201723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.201663    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.201884 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.201830    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4wln\" (UniqueName: \"kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-kube-api-access-w4wln\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.209586 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.209558    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:46:59.713794 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.712217    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"]
Nov 03 01:46:59.719164 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:46:59.719137    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7c9be9_74ca_43af_a8f2_0667cf1d5346.slice/crio-36bcb8fa31412b07e10b2b13548c9fdd99a0e704ac551d9a71cfee43c7f19623 WatchSource:0}: Error finding container 36bcb8fa31412b07e10b2b13548c9fdd99a0e704ac551d9a71cfee43c7f19623: Status 404 returned error can't find the container with id 36bcb8fa31412b07e10b2b13548c9fdd99a0e704ac551d9a71cfee43c7f19623
Nov 03 01:46:59.722281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.722246    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:46:59.722410 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.722315    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:46:59.722410 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.722360    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:46:59.907411 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.907339    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n" event={"ID":"fd7c9be9-74ca-43af-a8f2-0667cf1d5346","Type":"ContainerStarted","Data":"fe83c79f52a5c2d5ce3abe9a14de80419db3d4cc44bcf0a8bf74654070dc2418"}
Nov 03 01:46:59.907892 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:46:59.907439    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n" event={"ID":"fd7c9be9-74ca-43af-a8f2-0667cf1d5346","Type":"ContainerStarted","Data":"36bcb8fa31412b07e10b2b13548c9fdd99a0e704ac551d9a71cfee43c7f19623"}
Nov 03 01:47:00.024133 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:00.024016    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n" podStartSLOduration=2.023994722 podStartE2EDuration="2.023994722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:00.019498871 +0000 UTC m=+1699.445105358" watchObservedRunningTime="2026-11-03 01:47:00.023994722 +0000 UTC m=+1699.449601187"
Nov 03 01:47:00.210445 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:00.210407    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:47:00.212144 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:00.212117    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 01:47:00.212295 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:00.212165    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n" podUID="fd7c9be9-74ca-43af-a8f2-0667cf1d5346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 01:47:00.916359 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:00.916296    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr" event={"ID":"48e0d034-4613-46a3-b1a9-0b6d12ef8801","Type":"ContainerStarted","Data":"d5c2a87af82916e212faa502a90c2d6be8b012df9b26459ec56ab2e2b8637c09"}
Nov 03 01:47:00.916860 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:00.916392    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:47:01.030817 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:01.030761    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr" podStartSLOduration=2.188908845 podStartE2EDuration="6.03074683s" totalImagesPullingTime="3.841837985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:01.027899025 +0000 UTC m=+1700.453505516" watchObservedRunningTime="2026-11-03 01:47:01.03074683 +0000 UTC m=+1700.456353292"
Nov 03 01:47:01.216423 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:01.215833    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 01:47:01.216423 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:01.215904    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n" podUID="fd7c9be9-74ca-43af-a8f2-0667cf1d5346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 01:47:02.210321 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:02.210281    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 01:47:02.210854 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:02.210339    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n" podUID="fd7c9be9-74ca-43af-a8f2-0667cf1d5346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 01:47:03.045599 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:03.045558    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"]
Nov 03 01:47:03.092178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:03.092134    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06e5a21f-qqmpq"]
Nov 03 01:47:03.210153 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:03.210119    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 01:47:03.210311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:03.210174    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n" podUID="fd7c9be9-74ca-43af-a8f2-0667cf1d5346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 01:47:03.987156 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:03.987075    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:47:04.210132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:04.210093    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 01:47:04.210316 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:04.210144    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n" podUID="fd7c9be9-74ca-43af-a8f2-0667cf1d5346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 01:47:04.702986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:04.702946    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb153706-24e6-4f66-bd5b-c45f1c25de1c" path="/var/lib/kubelet/pods/fb153706-24e6-4f66-bd5b-c45f1c25de1c/volumes"
Nov 03 01:47:05.210973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:05.210932    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 01:47:05.211520 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:05.210999    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n" podUID="fd7c9be9-74ca-43af-a8f2-0667cf1d5346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 01:47:06.135557 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:06.135514    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"]
Nov 03 01:47:06.210489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:06.210455    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 01:47:06.210666 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:06.210510    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n" podUID="fd7c9be9-74ca-43af-a8f2-0667cf1d5346" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 01:47:06.224390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:06.224339    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"]
Nov 03 01:47:06.928377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:06.928324    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:47:07.217414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:07.217320    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:47:07.217799 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:07.217770    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:47:07.218462 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:07.218443    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:47:11.136388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.136330    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn" podUID="4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d" containerName="payload-processing" containerID="cri-o://90e74c0765c0fc5856bab77ba8f0aa314f89d95aa73adf6e5eb54b97bf8621e5" gracePeriod=25
Nov 03 01:47:11.225371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.225292    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb" podUID="49255ba1-cf88-4f8a-8d5f-84f260eb2d9b" containerName="payload-pre-processing" containerID="cri-o://7a8f820afc84a7db72f64805a8fae290dd063b67efca84dfd657171cb3de73af" gracePeriod=25
Nov 03 01:47:11.658169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.656901    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"
Nov 03 01:47:11.695648 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.694825    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-kube-api-access-jkqfr\" (UniqueName: \"kubernetes.io/projected/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-kube-api-access-jkqfr\") pod \"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d\" (UID: \"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d\") "
Nov 03 01:47:11.695648 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.695003    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-plugins-config-volume\") pod \"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d\" (UID: \"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d\") "
Nov 03 01:47:11.696054 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.696025    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-plugins-config-volume" pod "4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d" (UID: "4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:47:11.698525 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.698490    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-kube-api-access-jkqfr" pod "4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d" (UID: "4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d"). InnerVolumeSpecName "kube-api-access-jkqfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:47:11.746920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.746896    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"
Nov 03 01:47:11.796669 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.796634    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-plugins-config-volume\") pod \"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b\" (UID: \"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b\") "
Nov 03 01:47:11.796857 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.796739    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-kube-api-access-d65sd\" (UniqueName: \"kubernetes.io/projected/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-kube-api-access-d65sd\") pod \"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b\" (UID: \"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b\") "
Nov 03 01:47:11.797076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.797049    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-plugins-config-volume" pod "49255ba1-cf88-4f8a-8d5f-84f260eb2d9b" (UID: "49255ba1-cf88-4f8a-8d5f-84f260eb2d9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:47:11.797760 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.797732    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkqfr\" (UniqueName: \"kubernetes.io/projected/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-kube-api-access-jkqfr\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:11.797877 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.797763    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:11.797877 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.797779    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:11.799426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.799403    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-kube-api-access-d65sd" pod "49255ba1-cf88-4f8a-8d5f-84f260eb2d9b" (UID: "49255ba1-cf88-4f8a-8d5f-84f260eb2d9b"). InnerVolumeSpecName "kube-api-access-d65sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:47:11.899891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:11.899853    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d65sd\" (UniqueName: \"kubernetes.io/projected/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b-kube-api-access-d65sd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:12.006317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.006229    5508 generic.go:356] "Generic (PLEG): container finished" podID="49255ba1-cf88-4f8a-8d5f-84f260eb2d9b" containerID="7a8f820afc84a7db72f64805a8fae290dd063b67efca84dfd657171cb3de73af" exitCode=0
Nov 03 01:47:12.006317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.006276    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"
Nov 03 01:47:12.006556 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.006313    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb" event={"ID":"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b","Type":"ContainerDied","Data":"7a8f820afc84a7db72f64805a8fae290dd063b67efca84dfd657171cb3de73af"}
Nov 03 01:47:12.006556 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.006445    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb" event={"ID":"49255ba1-cf88-4f8a-8d5f-84f260eb2d9b","Type":"ContainerDied","Data":"7183970f79606d633662bb4b36900eb5d6bd52aa2e61c61698332cfd1879ff3c"}
Nov 03 01:47:12.006556 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.006483    5508 scope.go:122] "RemoveContainer" containerID="7a8f820afc84a7db72f64805a8fae290dd063b67efca84dfd657171cb3de73af"
Nov 03 01:47:12.007953 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.007925    5508 generic.go:356] "Generic (PLEG): container finished" podID="4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d" containerID="90e74c0765c0fc5856bab77ba8f0aa314f89d95aa73adf6e5eb54b97bf8621e5" exitCode=0
Nov 03 01:47:12.008058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.007968    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"
Nov 03 01:47:12.008058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.007984    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn" event={"ID":"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d","Type":"ContainerDied","Data":"90e74c0765c0fc5856bab77ba8f0aa314f89d95aa73adf6e5eb54b97bf8621e5"}
Nov 03 01:47:12.008058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.008033    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn" event={"ID":"4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d","Type":"ContainerDied","Data":"f9c31ad5ce9bee19becabafe00a9cf067f448ffe55837f38617fc1e78578ff13"}
Nov 03 01:47:12.016738 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.016718    5508 scope.go:122] "RemoveContainer" containerID="7a8f820afc84a7db72f64805a8fae290dd063b67efca84dfd657171cb3de73af"
Nov 03 01:47:12.017098 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:12.017072    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8f820afc84a7db72f64805a8fae290dd063b67efca84dfd657171cb3de73af\": container with ID starting with 7a8f820afc84a7db72f64805a8fae290dd063b67efca84dfd657171cb3de73af not found: ID does not exist" containerID="7a8f820afc84a7db72f64805a8fae290dd063b67efca84dfd657171cb3de73af"
Nov 03 01:47:12.017181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.017103    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8f820afc84a7db72f64805a8fae290dd063b67efca84dfd657171cb3de73af"} err="failed to get container status \"7a8f820afc84a7db72f64805a8fae290dd063b67efca84dfd657171cb3de73af\": rpc error: code = NotFound desc = could not find container \"7a8f820afc84a7db72f64805a8fae290dd063b67efca84dfd657171cb3de73af\": container with ID starting with 7a8f820afc84a7db72f64805a8fae290dd063b67efca84dfd657171cb3de73af not found: ID does not exist"
Nov 03 01:47:12.017181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.017119    5508 scope.go:122] "RemoveContainer" containerID="90e74c0765c0fc5856bab77ba8f0aa314f89d95aa73adf6e5eb54b97bf8621e5"
Nov 03 01:47:12.026378 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.026359    5508 scope.go:122] "RemoveContainer" containerID="90e74c0765c0fc5856bab77ba8f0aa314f89d95aa73adf6e5eb54b97bf8621e5"
Nov 03 01:47:12.026631 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:12.026613    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e74c0765c0fc5856bab77ba8f0aa314f89d95aa73adf6e5eb54b97bf8621e5\": container with ID starting with 90e74c0765c0fc5856bab77ba8f0aa314f89d95aa73adf6e5eb54b97bf8621e5 not found: ID does not exist" containerID="90e74c0765c0fc5856bab77ba8f0aa314f89d95aa73adf6e5eb54b97bf8621e5"
Nov 03 01:47:12.026669 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.026639    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e74c0765c0fc5856bab77ba8f0aa314f89d95aa73adf6e5eb54b97bf8621e5"} err="failed to get container status \"90e74c0765c0fc5856bab77ba8f0aa314f89d95aa73adf6e5eb54b97bf8621e5\": rpc error: code = NotFound desc = could not find container \"90e74c0765c0fc5856bab77ba8f0aa314f89d95aa73adf6e5eb54b97bf8621e5\": container with ID starting with 90e74c0765c0fc5856bab77ba8f0aa314f89d95aa73adf6e5eb54b97bf8621e5 not found: ID does not exist"
Nov 03 01:47:12.165562 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.163857    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"]
Nov 03 01:47:12.169067 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.169032    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5e048b-6f9cfc7d7-wm5vn"]
Nov 03 01:47:12.269683 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.269584    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"]
Nov 03 01:47:12.277437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.277403    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5e048b-959465f47-wppnb"]
Nov 03 01:47:12.696530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.696496    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49255ba1-cf88-4f8a-8d5f-84f260eb2d9b" path="/var/lib/kubelet/pods/49255ba1-cf88-4f8a-8d5f-84f260eb2d9b/volumes"
Nov 03 01:47:12.697084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:12.697065    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d" path="/var/lib/kubelet/pods/4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d/volumes"
Nov 03 01:47:13.857031 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:13.856991    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:47:14.004197 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.004159    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"]
Nov 03 01:47:14.004834 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.004817    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d" containerName="payload-processing"
Nov 03 01:47:14.004896 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.004837    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d" containerName="payload-processing"
Nov 03 01:47:14.004896 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.004851    5508 cpu_manager.go:414] "containerMap: removing container" podUID="49255ba1-cf88-4f8a-8d5f-84f260eb2d9b" containerName="payload-pre-processing"
Nov 03 01:47:14.004896 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.004856    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49255ba1-cf88-4f8a-8d5f-84f260eb2d9b" containerName="payload-pre-processing"
Nov 03 01:47:14.004995 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.004985    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2a3aba-9a9c-4add-a2bd-0e7a5c5b376d" containerName="payload-processing"
Nov 03 01:47:14.005025 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.004997    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="49255ba1-cf88-4f8a-8d5f-84f260eb2d9b" containerName="payload-pre-processing"
Nov 03 01:47:14.010200 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.010175    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.138093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.138007    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.138093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.138044    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.138093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.138078    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.138445 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.138169    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.138445 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.138205    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.138445 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.138234    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wlps\" (UniqueName: \"kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-kube-api-access-6wlps\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.138445 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.138428    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.138618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.138478    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.138618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.138517    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.218460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.218423    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"]
Nov 03 01:47:14.218722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.218692    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy" containerID="cri-o://c835ef9b6656ddda6b0e8ba6a06ed2b27e4fe3b8f10a8cba8f91f8a64b7becf7" gracePeriod=30
Nov 03 01:47:14.222946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.222907    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"]
Nov 03 01:47:14.242666 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.242273    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wlps\" (UniqueName: \"kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-kube-api-access-6wlps\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.242978 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.242951    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.243117 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.243102    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.243230 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.243215    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.243545 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.243527    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.243675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.243661    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.243793 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.243776    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.243920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.243905    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.244037 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.244021    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.247044 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.244536    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.247942 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.245692    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.248063 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.246640    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.248163 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.246913    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.248248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.247884    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.260424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.260391    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.262153 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.262095    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.305020 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.304941    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wlps\" (UniqueName: \"kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-kube-api-access-6wlps\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.382747 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.374194    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:14.625483 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:14.625449    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:15.077262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:15.077208    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"]
Nov 03 01:47:15.083283 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:47:15.083246    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a8b440_97a3_47a1_83b0_fd5627a39f8d.slice/crio-2d9b2b5d9024f03a2845d03116eacb4ee4e76fe15e3280191933badd358dba87 WatchSource:0}: Error finding container 2d9b2b5d9024f03a2845d03116eacb4ee4e76fe15e3280191933badd358dba87: Status 404 returned error can't find the container with id 2d9b2b5d9024f03a2845d03116eacb4ee4e76fe15e3280191933badd358dba87
Nov 03 01:47:15.086237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:15.086201    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:15.086322 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:15.086270    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:15.086322 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:15.086303    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:16.040577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:16.040392    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s" event={"ID":"48a8b440-97a3-47a1-83b0-fd5627a39f8d","Type":"ContainerStarted","Data":"e45ace429a16f5cf20057de9961636dd6486cb83c6002021228e2b652a57ca2e"}
Nov 03 01:47:16.040577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:16.040455    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s" event={"ID":"48a8b440-97a3-47a1-83b0-fd5627a39f8d","Type":"ContainerStarted","Data":"2d9b2b5d9024f03a2845d03116eacb4ee4e76fe15e3280191933badd358dba87"}
Nov 03 01:47:16.179385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:16.175865    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s" podStartSLOduration=3.17584175 podStartE2EDuration="3.17584175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:16.169835883 +0000 UTC m=+1715.595442366" watchObservedRunningTime="2026-11-03 01:47:16.17584175 +0000 UTC m=+1715.601448215"
Nov 03 01:47:16.625935 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:16.625893    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:16.627625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:16.627601    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 01:47:16.627756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:16.627677    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s" podUID="48a8b440-97a3-47a1-83b0-fd5627a39f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 01:47:17.632371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:17.626729    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 01:47:17.632371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:17.626805    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s" podUID="48a8b440-97a3-47a1-83b0-fd5627a39f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 01:47:18.626243 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:18.626209    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 01:47:18.626458 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:18.626280    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s" podUID="48a8b440-97a3-47a1-83b0-fd5627a39f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 01:47:19.569958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.569571    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:47:19.626665 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.626633    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 01:47:19.626819 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.626683    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s" podUID="48a8b440-97a3-47a1-83b0-fd5627a39f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 01:47:19.677876 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.677837    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9618bd52-f012-4dbb-b707-53a666005df7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9618bd52-f012-4dbb-b707-53a666005df7-istio-podinfo\") pod \"9618bd52-f012-4dbb-b707-53a666005df7\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") "
Nov 03 01:47:19.678089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.677925    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-credential-socket\") pod \"9618bd52-f012-4dbb-b707-53a666005df7\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") "
Nov 03 01:47:19.678089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.678004    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-envoy\") pod \"9618bd52-f012-4dbb-b707-53a666005df7\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") "
Nov 03 01:47:19.678089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.678075    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-socket\") pod \"9618bd52-f012-4dbb-b707-53a666005df7\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") "
Nov 03 01:47:19.678277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.678120    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-certs\") pod \"9618bd52-f012-4dbb-b707-53a666005df7\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") "
Nov 03 01:47:19.678277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.678173    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-istio-token\" (UniqueName: \"kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-istio-token\") pod \"9618bd52-f012-4dbb-b707-53a666005df7\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") "
Nov 03 01:47:19.678432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.678274    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9618bd52-f012-4dbb-b707-53a666005df7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9618bd52-f012-4dbb-b707-53a666005df7-istiod-ca-cert\") pod \"9618bd52-f012-4dbb-b707-53a666005df7\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") "
Nov 03 01:47:19.678432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.678335    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-kube-api-access-npkcg\" (UniqueName: \"kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-kube-api-access-npkcg\") pod \"9618bd52-f012-4dbb-b707-53a666005df7\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") "
Nov 03 01:47:19.678532 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.678468    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-data\") pod \"9618bd52-f012-4dbb-b707-53a666005df7\" (UID: \"9618bd52-f012-4dbb-b707-53a666005df7\") "
Nov 03 01:47:19.679203 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.678897    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-credential-socket" pod "9618bd52-f012-4dbb-b707-53a666005df7" (UID: "9618bd52-f012-4dbb-b707-53a666005df7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:47:19.679203 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.679151    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:19.679616 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.679491    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9618bd52-f012-4dbb-b707-53a666005df7-istiod-ca-cert" pod "9618bd52-f012-4dbb-b707-53a666005df7" (UID: "9618bd52-f012-4dbb-b707-53a666005df7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:47:19.679616 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.679523    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-certs" pod "9618bd52-f012-4dbb-b707-53a666005df7" (UID: "9618bd52-f012-4dbb-b707-53a666005df7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:47:19.679760 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.679629    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-socket" pod "9618bd52-f012-4dbb-b707-53a666005df7" (UID: "9618bd52-f012-4dbb-b707-53a666005df7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:47:19.679760 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.679694    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-data" pod "9618bd52-f012-4dbb-b707-53a666005df7" (UID: "9618bd52-f012-4dbb-b707-53a666005df7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:47:19.681643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.681414    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9618bd52-f012-4dbb-b707-53a666005df7-istio-podinfo" pod "9618bd52-f012-4dbb-b707-53a666005df7" (UID: "9618bd52-f012-4dbb-b707-53a666005df7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:47:19.681643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.681488    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-kube-api-access-npkcg" pod "9618bd52-f012-4dbb-b707-53a666005df7" (UID: "9618bd52-f012-4dbb-b707-53a666005df7"). InnerVolumeSpecName "kube-api-access-npkcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:47:19.681643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.681611    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-istio-token" pod "9618bd52-f012-4dbb-b707-53a666005df7" (UID: "9618bd52-f012-4dbb-b707-53a666005df7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:47:19.681868 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.681844    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-envoy" pod "9618bd52-f012-4dbb-b707-53a666005df7" (UID: "9618bd52-f012-4dbb-b707-53a666005df7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:47:19.780404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.780302    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9618bd52-f012-4dbb-b707-53a666005df7-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:19.780404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.780331    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npkcg\" (UniqueName: \"kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-kube-api-access-npkcg\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:19.780404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.780341    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:19.780404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.780368    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9618bd52-f012-4dbb-b707-53a666005df7-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:19.780404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.780377    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:19.780404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.780386    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:19.780404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.780395    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9618bd52-f012-4dbb-b707-53a666005df7-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:19.780404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:19.780405    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9618bd52-f012-4dbb-b707-53a666005df7-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:20.099141 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.099043    5508 generic.go:356] "Generic (PLEG): container finished" podID="9618bd52-f012-4dbb-b707-53a666005df7" containerID="c835ef9b6656ddda6b0e8ba6a06ed2b27e4fe3b8f10a8cba8f91f8a64b7becf7" exitCode=0
Nov 03 01:47:20.099141 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.099097    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"
Nov 03 01:47:20.099141 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.099122    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" event={"ID":"9618bd52-f012-4dbb-b707-53a666005df7","Type":"ContainerDied","Data":"c835ef9b6656ddda6b0e8ba6a06ed2b27e4fe3b8f10a8cba8f91f8a64b7becf7"}
Nov 03 01:47:20.099499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.099178    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx" event={"ID":"9618bd52-f012-4dbb-b707-53a666005df7","Type":"ContainerDied","Data":"2feeace6974a6f910691b1dd4668fbbcb27c2b2b3044e0fefab02ebabc46ab5e"}
Nov 03 01:47:20.099499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.099199    5508 scope.go:122] "RemoveContainer" containerID="c835ef9b6656ddda6b0e8ba6a06ed2b27e4fe3b8f10a8cba8f91f8a64b7becf7"
Nov 03 01:47:20.111401 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.111384    5508 scope.go:122] "RemoveContainer" containerID="0dd8244730903977d68c22882139c63e3b29674e456ada8b20bd9529a59dd183"
Nov 03 01:47:20.121807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.121781    5508 scope.go:122] "RemoveContainer" containerID="c835ef9b6656ddda6b0e8ba6a06ed2b27e4fe3b8f10a8cba8f91f8a64b7becf7"
Nov 03 01:47:20.122079 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:20.122059    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c835ef9b6656ddda6b0e8ba6a06ed2b27e4fe3b8f10a8cba8f91f8a64b7becf7\": container with ID starting with c835ef9b6656ddda6b0e8ba6a06ed2b27e4fe3b8f10a8cba8f91f8a64b7becf7 not found: ID does not exist" containerID="c835ef9b6656ddda6b0e8ba6a06ed2b27e4fe3b8f10a8cba8f91f8a64b7becf7"
Nov 03 01:47:20.122162 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.122086    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c835ef9b6656ddda6b0e8ba6a06ed2b27e4fe3b8f10a8cba8f91f8a64b7becf7"} err="failed to get container status \"c835ef9b6656ddda6b0e8ba6a06ed2b27e4fe3b8f10a8cba8f91f8a64b7becf7\": rpc error: code = NotFound desc = could not find container \"c835ef9b6656ddda6b0e8ba6a06ed2b27e4fe3b8f10a8cba8f91f8a64b7becf7\": container with ID starting with c835ef9b6656ddda6b0e8ba6a06ed2b27e4fe3b8f10a8cba8f91f8a64b7becf7 not found: ID does not exist"
Nov 03 01:47:20.122162 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.122103    5508 scope.go:122] "RemoveContainer" containerID="0dd8244730903977d68c22882139c63e3b29674e456ada8b20bd9529a59dd183"
Nov 03 01:47:20.122386 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:20.122341    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd8244730903977d68c22882139c63e3b29674e456ada8b20bd9529a59dd183\": container with ID starting with 0dd8244730903977d68c22882139c63e3b29674e456ada8b20bd9529a59dd183 not found: ID does not exist" containerID="0dd8244730903977d68c22882139c63e3b29674e456ada8b20bd9529a59dd183"
Nov 03 01:47:20.122429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.122392    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd8244730903977d68c22882139c63e3b29674e456ada8b20bd9529a59dd183"} err="failed to get container status \"0dd8244730903977d68c22882139c63e3b29674e456ada8b20bd9529a59dd183\": rpc error: code = NotFound desc = could not find container \"0dd8244730903977d68c22882139c63e3b29674e456ada8b20bd9529a59dd183\": container with ID starting with 0dd8244730903977d68c22882139c63e3b29674e456ada8b20bd9529a59dd183 not found: ID does not exist"
Nov 03 01:47:20.244709 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.244677    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"]
Nov 03 01:47:20.245474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.245454    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy"
Nov 03 01:47:20.245474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.245473    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy"
Nov 03 01:47:20.245588 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.245489    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy"
Nov 03 01:47:20.245588 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.245494    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy"
Nov 03 01:47:20.245677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.245605    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy"
Nov 03 01:47:20.258678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.258652    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.285438 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.285406    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"]
Nov 03 01:47:20.292033 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.292005    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-948470-openshift-default-5c4bd969b5-bb5fx"]
Nov 03 01:47:20.307975 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.307950    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-23e673-openshift-default-dockercfg-dx7mx\""
Nov 03 01:47:20.348293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.348255    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"]
Nov 03 01:47:20.394641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.385020    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"]
Nov 03 01:47:20.394641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.388922    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-podinfo\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.394641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.388953    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-data\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.394641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.388973    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-certs\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.394641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.388996    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-credential-socket\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.394641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.389011    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-token\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.394641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.389038    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhw9q\" (UniqueName: \"kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-kube-api-access-bhw9q\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.394641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.389058    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-socket\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.394641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.389092    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-envoy\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.394641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.389112    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istiod-ca-cert\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.496702 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.496663    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-podinfo\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.501517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.501479    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-data\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.501771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.501752    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-certs\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.501906 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.501893    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-credential-socket\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.502000 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.501988    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-token\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.502192 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.502178    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhw9q\" (UniqueName: \"kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-kube-api-access-bhw9q\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.502309 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.502296    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-socket\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.502485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.502468    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-envoy\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.502601 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.502587    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istiod-ca-cert\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.521371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.512027    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-data\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.521371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.512518    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-certs\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.521371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.513018    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-credential-socket\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.521371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.518963    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-socket\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.526175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.523029    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istiod-ca-cert\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.600368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.594655    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-envoy\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.681998 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.681957    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 01:47:20.687467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.687423    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s" podUID="48a8b440-97a3-47a1-83b0-fd5627a39f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 01:47:20.950458 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.936725    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhw9q\" (UniqueName: \"kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-kube-api-access-bhw9q\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.950662 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.950227    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-token\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.997455 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.986964    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-podinfo\") pod \"e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") " pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:20.997455 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:20.987457    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:21.014097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:21.014052    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9618bd52-f012-4dbb-b707-53a666005df7" path="/var/lib/kubelet/pods/9618bd52-f012-4dbb-b707-53a666005df7/volumes"
Nov 03 01:47:21.454085 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:21.454057    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"]
Nov 03 01:47:21.456965 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:47:21.456937    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774d3ce9_baeb_4de2_837b_62b8a559b7b3.slice/crio-216a9d82958aa5399eb56c0c18d5ce433359713bb398a2b67aa617522a10c277 WatchSource:0}: Error finding container 216a9d82958aa5399eb56c0c18d5ce433359713bb398a2b67aa617522a10c277: Status 404 returned error can't find the container with id 216a9d82958aa5399eb56c0c18d5ce433359713bb398a2b67aa617522a10c277
Nov 03 01:47:21.460205 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:21.460173    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:21.460315 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:21.460238    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:21.460315 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:21.460266    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:21.632677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:21.632641    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:21.633787 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:21.633607    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:21.634827 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:21.634804    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:47:22.172735 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:22.172698    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" event={"ID":"774d3ce9-baeb-4de2-837b-62b8a559b7b3","Type":"ContainerStarted","Data":"50fba1e546916d1e949d2ddd2234df40bb647848220f5e96ea1aefb0f2069702"}
Nov 03 01:47:22.172938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:22.172761    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" event={"ID":"774d3ce9-baeb-4de2-837b-62b8a559b7b3","Type":"ContainerStarted","Data":"216a9d82958aa5399eb56c0c18d5ce433359713bb398a2b67aa617522a10c277"}
Nov 03 01:47:22.304118 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:22.303998    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" podStartSLOduration=2.303976611 podStartE2EDuration="2.303976611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:22.285451065 +0000 UTC m=+1721.711057530" watchObservedRunningTime="2026-11-03 01:47:22.303976611 +0000 UTC m=+1721.729583074"
Nov 03 01:47:22.990373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:22.989389    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:23.010281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:23.006129    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:23.180769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:23.180735    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:23.181788 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:23.181768    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:47:25.385666 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:25.385622    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs" podUID="636315c8-c8fd-44b4-9ae5-5052b1b0af27" containerName="payload-pre-processing" containerID="cri-o://c4d70704128f7c8e9932536f5917bfa14bdcd3a76575bf861f2cb306c7f0ee95" gracePeriod=25
Nov 03 01:47:25.646654 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:25.646627    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"
Nov 03 01:47:25.731669 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:25.731628    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636315c8-c8fd-44b4-9ae5-5052b1b0af27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636315c8-c8fd-44b4-9ae5-5052b1b0af27-plugins-config-volume\") pod \"636315c8-c8fd-44b4-9ae5-5052b1b0af27\" (UID: \"636315c8-c8fd-44b4-9ae5-5052b1b0af27\") "
Nov 03 01:47:25.731854 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:25.731767    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636315c8-c8fd-44b4-9ae5-5052b1b0af27-kube-api-access-xpccw\" (UniqueName: \"kubernetes.io/projected/636315c8-c8fd-44b4-9ae5-5052b1b0af27-kube-api-access-xpccw\") pod \"636315c8-c8fd-44b4-9ae5-5052b1b0af27\" (UID: \"636315c8-c8fd-44b4-9ae5-5052b1b0af27\") "
Nov 03 01:47:25.732037 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:25.732014    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636315c8-c8fd-44b4-9ae5-5052b1b0af27-plugins-config-volume" pod "636315c8-c8fd-44b4-9ae5-5052b1b0af27" (UID: "636315c8-c8fd-44b4-9ae5-5052b1b0af27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:47:25.732324 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:25.732304    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636315c8-c8fd-44b4-9ae5-5052b1b0af27-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:25.734013 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:25.733978    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636315c8-c8fd-44b4-9ae5-5052b1b0af27-kube-api-access-xpccw" pod "636315c8-c8fd-44b4-9ae5-5052b1b0af27" (UID: "636315c8-c8fd-44b4-9ae5-5052b1b0af27"). InnerVolumeSpecName "kube-api-access-xpccw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:47:25.833389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:25.833304    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpccw\" (UniqueName: \"kubernetes.io/projected/636315c8-c8fd-44b4-9ae5-5052b1b0af27-kube-api-access-xpccw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:26.203798 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:26.203759    5508 generic.go:356] "Generic (PLEG): container finished" podID="636315c8-c8fd-44b4-9ae5-5052b1b0af27" containerID="c4d70704128f7c8e9932536f5917bfa14bdcd3a76575bf861f2cb306c7f0ee95" exitCode=0
Nov 03 01:47:26.203980 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:26.203815    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"
Nov 03 01:47:26.203980 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:26.203839    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs" event={"ID":"636315c8-c8fd-44b4-9ae5-5052b1b0af27","Type":"ContainerDied","Data":"c4d70704128f7c8e9932536f5917bfa14bdcd3a76575bf861f2cb306c7f0ee95"}
Nov 03 01:47:26.203980 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:26.203901    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs" event={"ID":"636315c8-c8fd-44b4-9ae5-5052b1b0af27","Type":"ContainerDied","Data":"265840442cc15b24de2edeebae0da7792fb49b5cdc95309891f95d18b1030cca"}
Nov 03 01:47:26.203980 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:26.203929    5508 scope.go:122] "RemoveContainer" containerID="c4d70704128f7c8e9932536f5917bfa14bdcd3a76575bf861f2cb306c7f0ee95"
Nov 03 01:47:26.215065 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:26.215040    5508 scope.go:122] "RemoveContainer" containerID="c4d70704128f7c8e9932536f5917bfa14bdcd3a76575bf861f2cb306c7f0ee95"
Nov 03 01:47:26.215416 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:26.215397    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d70704128f7c8e9932536f5917bfa14bdcd3a76575bf861f2cb306c7f0ee95\": container with ID starting with c4d70704128f7c8e9932536f5917bfa14bdcd3a76575bf861f2cb306c7f0ee95 not found: ID does not exist" containerID="c4d70704128f7c8e9932536f5917bfa14bdcd3a76575bf861f2cb306c7f0ee95"
Nov 03 01:47:26.215474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:26.215425    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d70704128f7c8e9932536f5917bfa14bdcd3a76575bf861f2cb306c7f0ee95"} err="failed to get container status \"c4d70704128f7c8e9932536f5917bfa14bdcd3a76575bf861f2cb306c7f0ee95\": rpc error: code = NotFound desc = could not find container \"c4d70704128f7c8e9932536f5917bfa14bdcd3a76575bf861f2cb306c7f0ee95\": container with ID starting with c4d70704128f7c8e9932536f5917bfa14bdcd3a76575bf861f2cb306c7f0ee95 not found: ID does not exist"
Nov 03 01:47:26.358101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:26.358067    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"]
Nov 03 01:47:26.404395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:26.404342    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4952b1-64684dc575-bdrbs"]
Nov 03 01:47:26.693046 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:26.693013    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636315c8-c8fd-44b4-9ae5-5052b1b0af27" path="/var/lib/kubelet/pods/636315c8-c8fd-44b4-9ae5-5052b1b0af27/volumes"
Nov 03 01:47:28.925373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:28.918174    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"]
Nov 03 01:47:28.925373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:28.919343    5508 cpu_manager.go:414] "containerMap: removing container" podUID="636315c8-c8fd-44b4-9ae5-5052b1b0af27" containerName="payload-pre-processing"
Nov 03 01:47:28.925373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:28.919381    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636315c8-c8fd-44b4-9ae5-5052b1b0af27" containerName="payload-pre-processing"
Nov 03 01:47:28.925373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:28.919538    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="636315c8-c8fd-44b4-9ae5-5052b1b0af27" containerName="payload-pre-processing"
Nov 03 01:47:28.925373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:28.919561    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9618bd52-f012-4dbb-b707-53a666005df7" containerName="istio-proxy"
Nov 03 01:47:28.939374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:28.937651    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"
Nov 03 01:47:28.996833 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:28.996793    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"]
Nov 03 01:47:29.020737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.020703    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-23e673\""
Nov 03 01:47:29.033369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.021096    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-23e673-dockercfg-qvplw\""
Nov 03 01:47:29.051300 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.051263    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl\" (UID: \"4982d2d4-a1e9-492e-b5aa-b599eb7ae954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"
Nov 03 01:47:29.053796 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.053763    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz88w\" (UniqueName: \"kubernetes.io/projected/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-kube-api-access-kz88w\") pod \"payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl\" (UID: \"4982d2d4-a1e9-492e-b5aa-b599eb7ae954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"
Nov 03 01:47:29.109124 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.106480    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"]
Nov 03 01:47:29.114894 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.114862    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"
Nov 03 01:47:29.136100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.136063    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"]
Nov 03 01:47:29.148320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.148275    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.155502 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.155470    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl\" (UID: \"4982d2d4-a1e9-492e-b5aa-b599eb7ae954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"
Nov 03 01:47:29.155747 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.155725    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz88w\" (UniqueName: \"kubernetes.io/projected/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-kube-api-access-kz88w\") pod \"payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl\" (UID: \"4982d2d4-a1e9-492e-b5aa-b599eb7ae954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"
Nov 03 01:47:29.156530 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.156494    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl\" (UID: \"4982d2d4-a1e9-492e-b5aa-b599eb7ae954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"
Nov 03 01:47:29.165561 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.165533    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"]
Nov 03 01:47:29.213262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.213178    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz88w\" (UniqueName: \"kubernetes.io/projected/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-kube-api-access-kz88w\") pod \"payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl\" (UID: \"4982d2d4-a1e9-492e-b5aa-b599eb7ae954\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"
Nov 03 01:47:29.236263 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.236227    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"]
Nov 03 01:47:29.257613 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.257573    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.257806 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.257630    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.257851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.257801    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.257885 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.257867    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.257920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.257896    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27e6feb-5948-4fa9-b295-b73779b7291f-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.257999 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.257981    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwcjz\" (UniqueName: \"kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-kube-api-access-kwcjz\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.258042 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.258028    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.258087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.258072    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rnqm\" (UniqueName: \"kubernetes.io/projected/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-kube-api-access-2rnqm\") pod \"payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv\" (UID: \"6ab1ecd5-69b4-4aec-a116-32fd0e14db26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"
Nov 03 01:47:29.258139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.258126    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv\" (UID: \"6ab1ecd5-69b4-4aec-a116-32fd0e14db26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"
Nov 03 01:47:29.258175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.258164    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.258212 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.258192    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.300286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.300242    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"
Nov 03 01:47:29.360440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.359246    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwcjz\" (UniqueName: \"kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-kube-api-access-kwcjz\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.360440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.359309    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.360440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.359364    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rnqm\" (UniqueName: \"kubernetes.io/projected/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-kube-api-access-2rnqm\") pod \"payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv\" (UID: \"6ab1ecd5-69b4-4aec-a116-32fd0e14db26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"
Nov 03 01:47:29.360440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.359414    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv\" (UID: \"6ab1ecd5-69b4-4aec-a116-32fd0e14db26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"
Nov 03 01:47:29.360440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.359448    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.360440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.359476    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.360440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.359572    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.360440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.359612    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.360440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.359729    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.360440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.359779    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.360440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.359807    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27e6feb-5948-4fa9-b295-b73779b7291f-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.360916 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.360674    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27e6feb-5948-4fa9-b295-b73779b7291f-istiod-ca-cert\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.360958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.360914    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-certs\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.363385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.362211    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.363385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.362646    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv\" (UID: \"6ab1ecd5-69b4-4aec-a116-32fd0e14db26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"
Nov 03 01:47:29.363385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.362754    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-data\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.363385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.362955    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-credential-socket\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.367087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.366708    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-envoy\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.367293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.367254    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-podinfo\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.424393 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.423731    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwcjz\" (UniqueName: \"kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-kube-api-access-kwcjz\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.424393 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.423760    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-token\") pod \"e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") " pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.424393 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.423874    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rnqm\" (UniqueName: \"kubernetes.io/projected/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-kube-api-access-2rnqm\") pod \"payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv\" (UID: \"6ab1ecd5-69b4-4aec-a116-32fd0e14db26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"
Nov 03 01:47:29.454894 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.454828    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"
Nov 03 01:47:29.472818 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.472791    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:29.682203 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.682167    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"]
Nov 03 01:47:29.688993 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:47:29.688957    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4982d2d4_a1e9_492e_b5aa_b599eb7ae954.slice/crio-3b19208683342b115b402f10bede404b0395429c534f85c274361fb2d1736ae2 WatchSource:0}: Error finding container 3b19208683342b115b402f10bede404b0395429c534f85c274361fb2d1736ae2: Status 404 returned error can't find the container with id 3b19208683342b115b402f10bede404b0395429c534f85c274361fb2d1736ae2
Nov 03 01:47:29.846594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.846559    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"]
Nov 03 01:47:29.852491 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:47:29.852459    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab1ecd5_69b4_4aec_a116_32fd0e14db26.slice/crio-17ae558b0c9cad1b281d4ab548ca28f3f1f20646b9fe9235651b6e30e23453e1 WatchSource:0}: Error finding container 17ae558b0c9cad1b281d4ab548ca28f3f1f20646b9fe9235651b6e30e23453e1: Status 404 returned error can't find the container with id 17ae558b0c9cad1b281d4ab548ca28f3f1f20646b9fe9235651b6e30e23453e1
Nov 03 01:47:29.896930 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.896898    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"]
Nov 03 01:47:29.903982 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:47:29.903950    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27e6feb_5948_4fa9_b295_b73779b7291f.slice/crio-5368a612f4798abd8b4b5d9787abfbb29731f84de785281cf107409e5c02bbe2 WatchSource:0}: Error finding container 5368a612f4798abd8b4b5d9787abfbb29731f84de785281cf107409e5c02bbe2: Status 404 returned error can't find the container with id 5368a612f4798abd8b4b5d9787abfbb29731f84de785281cf107409e5c02bbe2
Nov 03 01:47:29.906762 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.906732    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:29.906849 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.906799    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:29.906849 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:29.906830    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:30.271735 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.271688    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl" event={"ID":"4982d2d4-a1e9-492e-b5aa-b599eb7ae954","Type":"ContainerStarted","Data":"e68ed68def97bbbd6a95a4a156a84a7338ef476688744073e715e6496531a760"}
Nov 03 01:47:30.272244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.271755    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl" event={"ID":"4982d2d4-a1e9-492e-b5aa-b599eb7ae954","Type":"ContainerStarted","Data":"3b19208683342b115b402f10bede404b0395429c534f85c274361fb2d1736ae2"}
Nov 03 01:47:30.272244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.271809    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"
Nov 03 01:47:30.274315 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.274284    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb" event={"ID":"d27e6feb-5948-4fa9-b295-b73779b7291f","Type":"ContainerStarted","Data":"9a8478cd73f9536a8e34201562aac9b1e41d24a4d34de5b32249cf91533e2174"}
Nov 03 01:47:30.274514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.274364    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb" event={"ID":"d27e6feb-5948-4fa9-b295-b73779b7291f","Type":"ContainerStarted","Data":"5368a612f4798abd8b4b5d9787abfbb29731f84de785281cf107409e5c02bbe2"}
Nov 03 01:47:30.277217 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.276474    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv" event={"ID":"6ab1ecd5-69b4-4aec-a116-32fd0e14db26","Type":"ContainerStarted","Data":"980494dc2b851ad996aba109b7a0fff44e99c94eaec0116a07ee0c84302a100e"}
Nov 03 01:47:30.277217 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.276560    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"
Nov 03 01:47:30.277217 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.276586    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv" event={"ID":"6ab1ecd5-69b4-4aec-a116-32fd0e14db26","Type":"ContainerStarted","Data":"17ae558b0c9cad1b281d4ab548ca28f3f1f20646b9fe9235651b6e30e23453e1"}
Nov 03 01:47:30.385526 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.385458    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl" podStartSLOduration=2.385436344 podStartE2EDuration="2.385436344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:30.383695732 +0000 UTC m=+1729.809302198" watchObservedRunningTime="2026-11-03 01:47:30.385436344 +0000 UTC m=+1729.811042810"
Nov 03 01:47:30.411842 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.411808    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7002-310580"]
Nov 03 01:47:30.416280 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.416259    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-310580"
Nov 03 01:47:30.465174 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.465142    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-882bv\""
Nov 03 01:47:30.468758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.468727    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7002-310580"]
Nov 03 01:47:30.473511 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.473477    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:30.475203 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.475173    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:47:30.475341 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.475232    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb" podUID="d27e6feb-5948-4fa9-b295-b73779b7291f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:47:30.489601 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.489557    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp4q6\" (UniqueName: \"kubernetes.io/projected/ca541bc3-73f5-4255-95fd-78556757bc51-kube-api-access-dp4q6\") pod \"test-curl-7002-310580\" (UID: \"ca541bc3-73f5-4255-95fd-78556757bc51\") " pod="opendatahub/test-curl-7002-310580"
Nov 03 01:47:30.512371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.507486    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv" podStartSLOduration=1.507463683 podStartE2EDuration="1.507463683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:30.50446762 +0000 UTC m=+1729.930074106" watchObservedRunningTime="2026-11-03 01:47:30.507463683 +0000 UTC m=+1729.933070148"
Nov 03 01:47:30.597423 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.591236    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp4q6\" (UniqueName: \"kubernetes.io/projected/ca541bc3-73f5-4255-95fd-78556757bc51-kube-api-access-dp4q6\") pod \"test-curl-7002-310580\" (UID: \"ca541bc3-73f5-4255-95fd-78556757bc51\") " pod="opendatahub/test-curl-7002-310580"
Nov 03 01:47:30.630370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.626694    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb" podStartSLOduration=1.6266745569999999 podStartE2EDuration="1.626674557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:30.619857924 +0000 UTC m=+1730.045464417" watchObservedRunningTime="2026-11-03 01:47:30.626674557 +0000 UTC m=+1730.052281023"
Nov 03 01:47:30.668372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.667620    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp4q6\" (UniqueName: \"kubernetes.io/projected/ca541bc3-73f5-4255-95fd-78556757bc51-kube-api-access-dp4q6\") pod \"test-curl-7002-310580\" (UID: \"ca541bc3-73f5-4255-95fd-78556757bc51\") " pod="opendatahub/test-curl-7002-310580"
Nov 03 01:47:30.730375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:30.729723    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-310580"
Nov 03 01:47:31.217193 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:31.217156    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7002-310580"]
Nov 03 01:47:31.223678 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:47:31.223643    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca541bc3_73f5_4255_95fd_78556757bc51.slice/crio-3dd7376383d659186b8b7e4c5bb6be1fd56ae08527c979a6cd9ce4252c7b760f WatchSource:0}: Error finding container 3dd7376383d659186b8b7e4c5bb6be1fd56ae08527c979a6cd9ce4252c7b760f: Status 404 returned error can't find the container with id 3dd7376383d659186b8b7e4c5bb6be1fd56ae08527c979a6cd9ce4252c7b760f
Nov 03 01:47:31.290816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:31.290776    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7002-310580" event={"ID":"ca541bc3-73f5-4255-95fd-78556757bc51","Type":"ContainerStarted","Data":"3dd7376383d659186b8b7e4c5bb6be1fd56ae08527c979a6cd9ce4252c7b760f"}
Nov 03 01:47:31.475447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:31.474784    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:47:31.475447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:31.474851    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb" podUID="d27e6feb-5948-4fa9-b295-b73779b7291f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:47:32.475894 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:32.475850    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:47:32.476553 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:32.476515    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb" podUID="d27e6feb-5948-4fa9-b295-b73779b7291f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:47:33.476417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:33.475777    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:47:33.476417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:33.475839    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb" podUID="d27e6feb-5948-4fa9-b295-b73779b7291f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:47:34.505370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:34.475756    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:47:34.505370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:34.475817    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb" podUID="d27e6feb-5948-4fa9-b295-b73779b7291f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:47:35.475284 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:35.475234    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:47:35.476056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:35.475600    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb" podUID="d27e6feb-5948-4fa9-b295-b73779b7291f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:47:36.474279 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:36.473769    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 01:47:36.474279 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:36.473827    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb" podUID="d27e6feb-5948-4fa9-b295-b73779b7291f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 01:47:37.388212 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.388175    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7002-310580" event={"ID":"ca541bc3-73f5-4255-95fd-78556757bc51","Type":"ContainerStarted","Data":"f3b837421c676c6afae01a90aefd045ebeb9289f3b737aaf8594b82f5db5a9bc"}
Nov 03 01:47:37.427415 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.427377    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"]
Nov 03 01:47:37.431996 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.431972    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.439449 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.439408    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-token\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.439609 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.439528    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-podinfo\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.439609 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.439570    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872fb22d-056c-4e0c-9555-e285d73cd87f-istiod-ca-cert\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.439761 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.439675    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-envoy\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.439761 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.439743    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-data\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.440155 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.439868    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pz7z\" (UniqueName: \"kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-kube-api-access-6pz7z\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.440155 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.440046    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-certs\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.440155 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.440111    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-credential-socket\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.440305 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.440170    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-socket\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.486435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.482663    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7bcbbb-openshift-default-dockercfg-nkzfb\""
Nov 03 01:47:37.568386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.548251    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-token\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.568386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.548369    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-podinfo\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.568386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.548399    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872fb22d-056c-4e0c-9555-e285d73cd87f-istiod-ca-cert\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.568386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.548436    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-envoy\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.568386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.548538    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-data\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.568386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.548623    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pz7z\" (UniqueName: \"kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-kube-api-access-6pz7z\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.568386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.548725    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-certs\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.568386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.548768    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-credential-socket\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.568386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.548835    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-socket\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.568386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.549573    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-socket\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.568386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.566610    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"]
Nov 03 01:47:37.568386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.566756    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:37.568386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.568090    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:37.569686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.568636    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-data\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.569686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.569301    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872fb22d-056c-4e0c-9555-e285d73cd87f-istiod-ca-cert\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.683756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.604653    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:47:37.683756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.613073    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-certs\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.683756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.613396    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-credential-socket\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.683756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.667636    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7002-310580" podStartSLOduration=2.518697637 podStartE2EDuration="7.667613802s" totalImagesPullingTime="5.148916165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:37.544834877 +0000 UTC m=+1736.970441361" watchObservedRunningTime="2026-11-03 01:47:37.667613802 +0000 UTC m=+1737.093220267"
Nov 03 01:47:37.909361 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.908773    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-podinfo\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.909361 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.909266    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-token\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.912907 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.912876    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-envoy\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:37.915108 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:37.915079    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pz7z\" (UniqueName: \"kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-kube-api-access-6pz7z\") pod \"e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") " pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:38.063097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:38.062642    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:38.895377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:38.890385    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"]
Nov 03 01:47:38.944371 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:47:38.937429    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872fb22d_056c_4e0c_9555_e285d73cd87f.slice/crio-d91a3287e92023bde9b85f522da51e9bf26c41c20c89ea7637faff09d42b233b WatchSource:0}: Error finding container d91a3287e92023bde9b85f522da51e9bf26c41c20c89ea7637faff09d42b233b: Status 404 returned error can't find the container with id d91a3287e92023bde9b85f522da51e9bf26c41c20c89ea7637faff09d42b233b
Nov 03 01:47:38.944371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:38.940790    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:38.944371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:38.940884    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:38.944371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:38.940923    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:39.510693 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:39.510596    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" event={"ID":"872fb22d-056c-4e0c-9555-e285d73cd87f","Type":"ContainerStarted","Data":"d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0"}
Nov 03 01:47:39.510986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:39.510943    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" event={"ID":"872fb22d-056c-4e0c-9555-e285d73cd87f","Type":"ContainerStarted","Data":"d91a3287e92023bde9b85f522da51e9bf26c41c20c89ea7637faff09d42b233b"}
Nov 03 01:47:39.698463 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:39.698428    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7002-310580"]
Nov 03 01:47:39.712218 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:39.698832    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7002-310580" podUID="ca541bc3-73f5-4255-95fd-78556757bc51" containerName="test-curl-7002-310580" containerID="cri-o://f3b837421c676c6afae01a90aefd045ebeb9289f3b737aaf8594b82f5db5a9bc" gracePeriod=2
Nov 03 01:47:39.716839 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:39.716689    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" podStartSLOduration=2.7166701619999998 podStartE2EDuration="2.716670162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:39.676896029 +0000 UTC m=+1739.102502494" watchObservedRunningTime="2026-11-03 01:47:39.716670162 +0000 UTC m=+1739.142276626"
Nov 03 01:47:39.724923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:39.724883    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7002-310580"]
Nov 03 01:47:39.769432 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:39.761931    5508 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7002-310580\" is forbidden: User \"system:node:ip-10-0-140-255.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-255.us-west-2.compute.internal' and this object" podUID="ca541bc3-73f5-4255-95fd-78556757bc51" pod="opendatahub/test-curl-7002-310580"
Nov 03 01:47:40.076383 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:40.076275    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:40.088677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:40.088643    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:40.136651 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:40.136602    5508 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7002-310580\" is forbidden: User \"system:node:ip-10-0-140-255.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-255.us-west-2.compute.internal' and this object" podUID="ca541bc3-73f5-4255-95fd-78556757bc51" pod="opendatahub/test-curl-7002-310580"
Nov 03 01:47:40.518148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:40.518118    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:40.519050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:40.519030    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:47:40.571709 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:40.571663    5508 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7002-310580\" is forbidden: User \"system:node:ip-10-0-140-255.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-255.us-west-2.compute.internal' and this object" podUID="ca541bc3-73f5-4255-95fd-78556757bc51" pod="opendatahub/test-curl-7002-310580"
Nov 03 01:47:40.742652 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:40.742519    5508 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7002-310580\" is forbidden: User \"system:node:ip-10-0-140-255.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-255.us-west-2.compute.internal' and this object" podUID="ca541bc3-73f5-4255-95fd-78556757bc51" pod="opendatahub/test-curl-7002-310580"
Nov 03 01:47:41.294595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:41.294564    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"
Nov 03 01:47:41.327370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:41.323468    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"
Nov 03 01:47:42.034269 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:42.034240    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-310580"
Nov 03 01:47:42.213008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:42.212972    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca541bc3-73f5-4255-95fd-78556757bc51-kube-api-access-dp4q6\" (UniqueName: \"kubernetes.io/projected/ca541bc3-73f5-4255-95fd-78556757bc51-kube-api-access-dp4q6\") pod \"ca541bc3-73f5-4255-95fd-78556757bc51\" (UID: \"ca541bc3-73f5-4255-95fd-78556757bc51\") "
Nov 03 01:47:42.215315 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:42.215289    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca541bc3-73f5-4255-95fd-78556757bc51-kube-api-access-dp4q6" pod "ca541bc3-73f5-4255-95fd-78556757bc51" (UID: "ca541bc3-73f5-4255-95fd-78556757bc51"). InnerVolumeSpecName "kube-api-access-dp4q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:47:42.315306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:42.315222    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp4q6\" (UniqueName: \"kubernetes.io/projected/ca541bc3-73f5-4255-95fd-78556757bc51-kube-api-access-dp4q6\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:42.542963 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:42.542930    5508 generic.go:356] "Generic (PLEG): container finished" podID="ca541bc3-73f5-4255-95fd-78556757bc51" containerID="f3b837421c676c6afae01a90aefd045ebeb9289f3b737aaf8594b82f5db5a9bc" exitCode=137
Nov 03 01:47:42.543152 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:42.542976    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7002-310580"
Nov 03 01:47:42.543152 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:42.543029    5508 scope.go:122] "RemoveContainer" containerID="f3b837421c676c6afae01a90aefd045ebeb9289f3b737aaf8594b82f5db5a9bc"
Nov 03 01:47:42.585449 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:42.585426    5508 scope.go:122] "RemoveContainer" containerID="f3b837421c676c6afae01a90aefd045ebeb9289f3b737aaf8594b82f5db5a9bc"
Nov 03 01:47:42.585840 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:42.585806    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b837421c676c6afae01a90aefd045ebeb9289f3b737aaf8594b82f5db5a9bc\": container with ID starting with f3b837421c676c6afae01a90aefd045ebeb9289f3b737aaf8594b82f5db5a9bc not found: ID does not exist" containerID="f3b837421c676c6afae01a90aefd045ebeb9289f3b737aaf8594b82f5db5a9bc"
Nov 03 01:47:42.585916 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:42.585852    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b837421c676c6afae01a90aefd045ebeb9289f3b737aaf8594b82f5db5a9bc"} err="failed to get container status \"f3b837421c676c6afae01a90aefd045ebeb9289f3b737aaf8594b82f5db5a9bc\": rpc error: code = NotFound desc = could not find container \"f3b837421c676c6afae01a90aefd045ebeb9289f3b737aaf8594b82f5db5a9bc\": container with ID starting with f3b837421c676c6afae01a90aefd045ebeb9289f3b737aaf8594b82f5db5a9bc not found: ID does not exist"
Nov 03 01:47:42.635687 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:42.635649    5508 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7002-310580\" is forbidden: User \"system:node:ip-10-0-140-255.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-255.us-west-2.compute.internal' and this object" podUID="ca541bc3-73f5-4255-95fd-78556757bc51" pod="opendatahub/test-curl-7002-310580"
Nov 03 01:47:42.706372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:42.696649    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca541bc3-73f5-4255-95fd-78556757bc51" path="/var/lib/kubelet/pods/ca541bc3-73f5-4255-95fd-78556757bc51/volumes"
Nov 03 01:47:43.814063 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:43.814024    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:47:44.046107 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:44.046069    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"]
Nov 03 01:47:44.066942 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:44.066851    5508 scope.go:122] "RemoveContainer" containerID="8686b1361ed9cabb9e736730ecda84e3f6696a6a23915ab331edf5741967ad21"
Nov 03 01:47:44.094067 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:44.094033    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573493fe-hqvn5"]
Nov 03 01:47:44.695626 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:44.695591    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5" path="/var/lib/kubelet/pods/211dc2ea-ab3c-4f44-9b46-b4d5daa1fcb5/volumes"
Nov 03 01:47:46.269380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.268518    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"]
Nov 03 01:47:46.269900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.269507    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ca541bc3-73f5-4255-95fd-78556757bc51" containerName="test-curl-7002-310580"
Nov 03 01:47:46.269900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.269527    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca541bc3-73f5-4255-95fd-78556757bc51" containerName="test-curl-7002-310580"
Nov 03 01:47:46.269900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.269724    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca541bc3-73f5-4255-95fd-78556757bc51" containerName="test-curl-7002-310580"
Nov 03 01:47:46.309371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.308624    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"
Nov 03 01:47:46.429371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.400690    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7bcbbb-dockercfg-7pwgd\""
Nov 03 01:47:46.429371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.400775    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7bcbbb\""
Nov 03 01:47:46.429371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.409951    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r\" (UID: \"8a2934d5-bbcc-4988-9566-3a3d0b16ec35\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"
Nov 03 01:47:46.429371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.410156    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjt9s\" (UniqueName: \"kubernetes.io/projected/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-kube-api-access-gjt9s\") pod \"payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r\" (UID: \"8a2934d5-bbcc-4988-9566-3a3d0b16ec35\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"
Nov 03 01:47:46.533424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.506885    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"]
Nov 03 01:47:46.558369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.517234    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjt9s\" (UniqueName: \"kubernetes.io/projected/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-kube-api-access-gjt9s\") pod \"payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r\" (UID: \"8a2934d5-bbcc-4988-9566-3a3d0b16ec35\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"
Nov 03 01:47:46.583428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.558766    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r\" (UID: \"8a2934d5-bbcc-4988-9566-3a3d0b16ec35\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"
Nov 03 01:47:46.615617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.565468    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r\" (UID: \"8a2934d5-bbcc-4988-9566-3a3d0b16ec35\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"
Nov 03 01:47:46.660752 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.605062    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjt9s\" (UniqueName: \"kubernetes.io/projected/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-kube-api-access-gjt9s\") pod \"payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r\" (UID: \"8a2934d5-bbcc-4988-9566-3a3d0b16ec35\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"
Nov 03 01:47:46.667972 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.667944    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"]
Nov 03 01:47:46.685384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.684159    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"]
Nov 03 01:47:46.689404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.689379    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"
Nov 03 01:47:46.717968 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.717933    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4hkk\" (UniqueName: \"kubernetes.io/projected/24931ff3-bf48-42d9-93d3-52456caf73b4-kube-api-access-v4hkk\") pod \"payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx\" (UID: \"24931ff3-bf48-42d9-93d3-52456caf73b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"
Nov 03 01:47:46.718415 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.718328    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24931ff3-bf48-42d9-93d3-52456caf73b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx\" (UID: \"24931ff3-bf48-42d9-93d3-52456caf73b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"
Nov 03 01:47:46.819486 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.819394    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4hkk\" (UniqueName: \"kubernetes.io/projected/24931ff3-bf48-42d9-93d3-52456caf73b4-kube-api-access-v4hkk\") pod \"payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx\" (UID: \"24931ff3-bf48-42d9-93d3-52456caf73b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"
Nov 03 01:47:46.819741 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.819519    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24931ff3-bf48-42d9-93d3-52456caf73b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx\" (UID: \"24931ff3-bf48-42d9-93d3-52456caf73b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"
Nov 03 01:47:46.820344 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.820321    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24931ff3-bf48-42d9-93d3-52456caf73b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx\" (UID: \"24931ff3-bf48-42d9-93d3-52456caf73b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"
Nov 03 01:47:46.838842 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.838798    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"]
Nov 03 01:47:46.877087 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.877047    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4hkk\" (UniqueName: \"kubernetes.io/projected/24931ff3-bf48-42d9-93d3-52456caf73b4-kube-api-access-v4hkk\") pod \"payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx\" (UID: \"24931ff3-bf48-42d9-93d3-52456caf73b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"
Nov 03 01:47:46.886826 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:46.886796    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"
Nov 03 01:47:47.027047 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:47.027003    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"
Nov 03 01:47:47.337069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:47.337026    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"]
Nov 03 01:47:47.348808 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:47:47.348764    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2934d5_bbcc_4988_9566_3a3d0b16ec35.slice/crio-7785830ff65f99cf1fa604773d0d87e6f3b0f8fed8872faac03f8cd5aec42cdb WatchSource:0}: Error finding container 7785830ff65f99cf1fa604773d0d87e6f3b0f8fed8872faac03f8cd5aec42cdb: Status 404 returned error can't find the container with id 7785830ff65f99cf1fa604773d0d87e6f3b0f8fed8872faac03f8cd5aec42cdb
Nov 03 01:47:47.523597 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:47.522874    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"]
Nov 03 01:47:47.654247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:47.654211    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r" event={"ID":"8a2934d5-bbcc-4988-9566-3a3d0b16ec35","Type":"ContainerStarted","Data":"499965c6a49610b4ba49b599d8f38cb70c12ff048392c38c4f063386ef0519f9"}
Nov 03 01:47:47.655800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:47.654276    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r" event={"ID":"8a2934d5-bbcc-4988-9566-3a3d0b16ec35","Type":"ContainerStarted","Data":"7785830ff65f99cf1fa604773d0d87e6f3b0f8fed8872faac03f8cd5aec42cdb"}
Nov 03 01:47:47.655800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:47.654304    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"
Nov 03 01:47:47.656100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:47.656072    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx" event={"ID":"24931ff3-bf48-42d9-93d3-52456caf73b4","Type":"ContainerStarted","Data":"1a12d5ad8898df6cfa576c6de073a4acbf76b58d230612819866a580c5651b47"}
Nov 03 01:47:47.766322 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:47.766260    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r" podStartSLOduration=1.766242789 podStartE2EDuration="1.766242789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:47.763325977 +0000 UTC m=+1747.188932460" watchObservedRunningTime="2026-11-03 01:47:47.766242789 +0000 UTC m=+1747.191849254"
Nov 03 01:47:48.664667 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:48.664633    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx" event={"ID":"24931ff3-bf48-42d9-93d3-52456caf73b4","Type":"ContainerStarted","Data":"84bf770ac94cbb58d4b596245e4a028722a8e5796f56c8b3a660e405a94c759a"}
Nov 03 01:47:48.665140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:48.664714    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"
Nov 03 01:47:48.666508 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:48.666472    5508 generic.go:356] "Generic (PLEG): container finished" podID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerID="bb0bcf5c0f6c82e59ce87cd1f457690c80d9e800c602b4af418edacea49e6d3e" exitCode=0
Nov 03 01:47:48.666624 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:48.666538    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" event={"ID":"d7e96865-e9ca-4040-bfe1-27d0c745ca85","Type":"ContainerDied","Data":"bb0bcf5c0f6c82e59ce87cd1f457690c80d9e800c602b4af418edacea49e6d3e"}
Nov 03 01:47:48.667078 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:48.667061    5508 scope.go:122] "RemoveContainer" containerID="bb0bcf5c0f6c82e59ce87cd1f457690c80d9e800c602b4af418edacea49e6d3e"
Nov 03 01:47:48.668441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:48.668414    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:48.668539 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:48.668492    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:48.668606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:48.668542    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:47:48.773894 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:48.773844    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx" podStartSLOduration=2.773826103 podStartE2EDuration="2.773826103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:48.772661718 +0000 UTC m=+1748.198268181" watchObservedRunningTime="2026-11-03 01:47:48.773826103 +0000 UTC m=+1748.199432568"
Nov 03 01:47:49.682835 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:49.682795    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" event={"ID":"d7e96865-e9ca-4040-bfe1-27d0c745ca85","Type":"ContainerStarted","Data":"ac7bfcb8128b94c85b3feb988a544148a4749acf4d76d5ead6904fb5c27e4c83"}
Nov 03 01:47:50.408597 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:50.408565    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:47:50.410085 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:50.410056    5508 patch_prober.go:29] interesting pod/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 01:47:50.410218 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:50.410120    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 01:47:51.408970 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:51.408926    5508 patch_prober.go:29] interesting pod/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 01:47:51.409466 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:51.408994    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 01:47:51.840050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:51.839935    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn" podUID="03b5ab3a-3ae0-4313-9619-42fdc2ca9682" containerName="payload-processing" containerID="cri-o://d3de9abd26031a7bfa91db2e061dfe9e005d40d229a216766699a9640c2c9065" gracePeriod=25
Nov 03 01:47:52.297969 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.297941    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"
Nov 03 01:47:52.409770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.409730    5508 patch_prober.go:29] interesting pod/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 01:47:52.410173 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.409801    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 01:47:52.472373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.448656    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-plugins-config-volume\") pod \"03b5ab3a-3ae0-4313-9619-42fdc2ca9682\" (UID: \"03b5ab3a-3ae0-4313-9619-42fdc2ca9682\") "
Nov 03 01:47:52.472373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.448865    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-kube-api-access-7xh4k\" (UniqueName: \"kubernetes.io/projected/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-kube-api-access-7xh4k\") pod \"03b5ab3a-3ae0-4313-9619-42fdc2ca9682\" (UID: \"03b5ab3a-3ae0-4313-9619-42fdc2ca9682\") "
Nov 03 01:47:52.472373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.456782    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-plugins-config-volume" pod "03b5ab3a-3ae0-4313-9619-42fdc2ca9682" (UID: "03b5ab3a-3ae0-4313-9619-42fdc2ca9682"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:47:52.475947 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.475907    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-kube-api-access-7xh4k" pod "03b5ab3a-3ae0-4313-9619-42fdc2ca9682" (UID: "03b5ab3a-3ae0-4313-9619-42fdc2ca9682"). InnerVolumeSpecName "kube-api-access-7xh4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:47:52.572744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.572607    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:52.572744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.572650    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xh4k\" (UniqueName: \"kubernetes.io/projected/03b5ab3a-3ae0-4313-9619-42fdc2ca9682-kube-api-access-7xh4k\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:47:52.618123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.618090    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:47:52.789994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.789952    5508 generic.go:356] "Generic (PLEG): container finished" podID="03b5ab3a-3ae0-4313-9619-42fdc2ca9682" containerID="d3de9abd26031a7bfa91db2e061dfe9e005d40d229a216766699a9640c2c9065" exitCode=0
Nov 03 01:47:52.790183 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.790016    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"
Nov 03 01:47:52.790183 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.790048    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn" event={"ID":"03b5ab3a-3ae0-4313-9619-42fdc2ca9682","Type":"ContainerDied","Data":"d3de9abd26031a7bfa91db2e061dfe9e005d40d229a216766699a9640c2c9065"}
Nov 03 01:47:52.790183 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.790116    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn" event={"ID":"03b5ab3a-3ae0-4313-9619-42fdc2ca9682","Type":"ContainerDied","Data":"743b8267a7b405076958c1e6c43cd5465be2431e2b05e0bf423012949064d2db"}
Nov 03 01:47:52.790183 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.790147    5508 scope.go:122] "RemoveContainer" containerID="d3de9abd26031a7bfa91db2e061dfe9e005d40d229a216766699a9640c2c9065"
Nov 03 01:47:52.802706 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.802682    5508 scope.go:122] "RemoveContainer" containerID="d3de9abd26031a7bfa91db2e061dfe9e005d40d229a216766699a9640c2c9065"
Nov 03 01:47:52.803052 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:47:52.803032    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3de9abd26031a7bfa91db2e061dfe9e005d40d229a216766699a9640c2c9065\": container with ID starting with d3de9abd26031a7bfa91db2e061dfe9e005d40d229a216766699a9640c2c9065 not found: ID does not exist" containerID="d3de9abd26031a7bfa91db2e061dfe9e005d40d229a216766699a9640c2c9065"
Nov 03 01:47:52.803111 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.803066    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3de9abd26031a7bfa91db2e061dfe9e005d40d229a216766699a9640c2c9065"} err="failed to get container status \"d3de9abd26031a7bfa91db2e061dfe9e005d40d229a216766699a9640c2c9065\": rpc error: code = NotFound desc = could not find container \"d3de9abd26031a7bfa91db2e061dfe9e005d40d229a216766699a9640c2c9065\": container with ID starting with d3de9abd26031a7bfa91db2e061dfe9e005d40d229a216766699a9640c2c9065 not found: ID does not exist"
Nov 03 01:47:52.980805 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.980765    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"]
Nov 03 01:47:52.984267 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:52.984240    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-118857-858d9f85-x57bn"]
Nov 03 01:47:53.409343 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:53.409307    5508 patch_prober.go:29] interesting pod/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 01:47:53.409611 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:53.409408    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 01:47:54.408949 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:54.408908    5508 patch_prober.go:29] interesting pod/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 01:47:54.409434 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:54.408967    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 01:47:54.694471 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:54.694387    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b5ab3a-3ae0-4313-9619-42fdc2ca9682" path="/var/lib/kubelet/pods/03b5ab3a-3ae0-4313-9619-42fdc2ca9682/volumes"
Nov 03 01:47:55.414025 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:55.413986    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:47:55.415141 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:55.415118    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:47:58.671002 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:58.670968    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"
Nov 03 01:47:59.686972 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:47:59.686942    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"
Nov 03 01:48:00.048123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:00.048026    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"]
Nov 03 01:48:00.150774 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:00.150734    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"]
Nov 03 01:48:05.048944 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.048891    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz" podUID="65d323ac-8a9a-422a-b97c-26c7e9b096ca" containerName="payload-processing" containerID="cri-o://31a4f447fef68dc35ee9d8e4c9a8603e5ad29d7dc191a83b76ead173a3b6a124" gracePeriod=25
Nov 03 01:48:05.151259 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.151205    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7" podUID="01dc3983-c36e-4e40-a7ea-a61d27f24d4a" containerName="payload-pre-processing" containerID="cri-o://f2fb342704c8e272fe2d28da8bf9e2b06a24599450f4cb66de2f72165360e9d1" gracePeriod=25
Nov 03 01:48:05.346279 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.346244    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"
Nov 03 01:48:05.460815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.460580    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"
Nov 03 01:48:05.482606 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.482575    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-kube-api-access-g75fl\" (UniqueName: \"kubernetes.io/projected/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-kube-api-access-g75fl\") pod \"01dc3983-c36e-4e40-a7ea-a61d27f24d4a\" (UID: \"01dc3983-c36e-4e40-a7ea-a61d27f24d4a\") "
Nov 03 01:48:05.482790 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.482695    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-plugins-config-volume\") pod \"01dc3983-c36e-4e40-a7ea-a61d27f24d4a\" (UID: \"01dc3983-c36e-4e40-a7ea-a61d27f24d4a\") "
Nov 03 01:48:05.482790 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.482745    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d323ac-8a9a-422a-b97c-26c7e9b096ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d323ac-8a9a-422a-b97c-26c7e9b096ca-plugins-config-volume\") pod \"65d323ac-8a9a-422a-b97c-26c7e9b096ca\" (UID: \"65d323ac-8a9a-422a-b97c-26c7e9b096ca\") "
Nov 03 01:48:05.482790 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.482775    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d323ac-8a9a-422a-b97c-26c7e9b096ca-kube-api-access-wrg2t\" (UniqueName: \"kubernetes.io/projected/65d323ac-8a9a-422a-b97c-26c7e9b096ca-kube-api-access-wrg2t\") pod \"65d323ac-8a9a-422a-b97c-26c7e9b096ca\" (UID: \"65d323ac-8a9a-422a-b97c-26c7e9b096ca\") "
Nov 03 01:48:05.483591 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.483556    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d323ac-8a9a-422a-b97c-26c7e9b096ca-plugins-config-volume" pod "65d323ac-8a9a-422a-b97c-26c7e9b096ca" (UID: "65d323ac-8a9a-422a-b97c-26c7e9b096ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:48:05.483794 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.483768    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-plugins-config-volume" pod "01dc3983-c36e-4e40-a7ea-a61d27f24d4a" (UID: "01dc3983-c36e-4e40-a7ea-a61d27f24d4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:48:05.485021 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.484998    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d323ac-8a9a-422a-b97c-26c7e9b096ca-kube-api-access-wrg2t" pod "65d323ac-8a9a-422a-b97c-26c7e9b096ca" (UID: "65d323ac-8a9a-422a-b97c-26c7e9b096ca"). InnerVolumeSpecName "kube-api-access-wrg2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:48:05.485148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.485035    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-kube-api-access-g75fl" pod "01dc3983-c36e-4e40-a7ea-a61d27f24d4a" (UID: "01dc3983-c36e-4e40-a7ea-a61d27f24d4a"). InnerVolumeSpecName "kube-api-access-g75fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:48:05.584521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.584427    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:05.584521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.584462    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d323ac-8a9a-422a-b97c-26c7e9b096ca-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:05.584521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.584474    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrg2t\" (UniqueName: \"kubernetes.io/projected/65d323ac-8a9a-422a-b97c-26c7e9b096ca-kube-api-access-wrg2t\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:05.584521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.584487    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g75fl\" (UniqueName: \"kubernetes.io/projected/01dc3983-c36e-4e40-a7ea-a61d27f24d4a-kube-api-access-g75fl\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:05.899465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.899430    5508 generic.go:356] "Generic (PLEG): container finished" podID="01dc3983-c36e-4e40-a7ea-a61d27f24d4a" containerID="f2fb342704c8e272fe2d28da8bf9e2b06a24599450f4cb66de2f72165360e9d1" exitCode=0
Nov 03 01:48:05.899665 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.899481    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"
Nov 03 01:48:05.899665 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.899514    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7" event={"ID":"01dc3983-c36e-4e40-a7ea-a61d27f24d4a","Type":"ContainerDied","Data":"f2fb342704c8e272fe2d28da8bf9e2b06a24599450f4cb66de2f72165360e9d1"}
Nov 03 01:48:05.899665 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.899582    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7" event={"ID":"01dc3983-c36e-4e40-a7ea-a61d27f24d4a","Type":"ContainerDied","Data":"cb33339271788f94493af15deaf22ca5425dd3bfadca720ce5200ace7a8261b2"}
Nov 03 01:48:05.899665 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.899615    5508 scope.go:122] "RemoveContainer" containerID="f2fb342704c8e272fe2d28da8bf9e2b06a24599450f4cb66de2f72165360e9d1"
Nov 03 01:48:05.901508 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.901476    5508 generic.go:356] "Generic (PLEG): container finished" podID="65d323ac-8a9a-422a-b97c-26c7e9b096ca" containerID="31a4f447fef68dc35ee9d8e4c9a8603e5ad29d7dc191a83b76ead173a3b6a124" exitCode=0
Nov 03 01:48:05.901617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.901525    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"
Nov 03 01:48:05.901617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.901557    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz" event={"ID":"65d323ac-8a9a-422a-b97c-26c7e9b096ca","Type":"ContainerDied","Data":"31a4f447fef68dc35ee9d8e4c9a8603e5ad29d7dc191a83b76ead173a3b6a124"}
Nov 03 01:48:05.901734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.901619    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz" event={"ID":"65d323ac-8a9a-422a-b97c-26c7e9b096ca","Type":"ContainerDied","Data":"82c5e7175ac9813b090be0772d486596f00bcc21be124417250dd6e300af1a70"}
Nov 03 01:48:05.911006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.910928    5508 scope.go:122] "RemoveContainer" containerID="f2fb342704c8e272fe2d28da8bf9e2b06a24599450f4cb66de2f72165360e9d1"
Nov 03 01:48:05.911391 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:48:05.911330    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fb342704c8e272fe2d28da8bf9e2b06a24599450f4cb66de2f72165360e9d1\": container with ID starting with f2fb342704c8e272fe2d28da8bf9e2b06a24599450f4cb66de2f72165360e9d1 not found: ID does not exist" containerID="f2fb342704c8e272fe2d28da8bf9e2b06a24599450f4cb66de2f72165360e9d1"
Nov 03 01:48:05.911477 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.911403    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fb342704c8e272fe2d28da8bf9e2b06a24599450f4cb66de2f72165360e9d1"} err="failed to get container status \"f2fb342704c8e272fe2d28da8bf9e2b06a24599450f4cb66de2f72165360e9d1\": rpc error: code = NotFound desc = could not find container \"f2fb342704c8e272fe2d28da8bf9e2b06a24599450f4cb66de2f72165360e9d1\": container with ID starting with f2fb342704c8e272fe2d28da8bf9e2b06a24599450f4cb66de2f72165360e9d1 not found: ID does not exist"
Nov 03 01:48:05.911477 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.911427    5508 scope.go:122] "RemoveContainer" containerID="31a4f447fef68dc35ee9d8e4c9a8603e5ad29d7dc191a83b76ead173a3b6a124"
Nov 03 01:48:05.922239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.922123    5508 scope.go:122] "RemoveContainer" containerID="31a4f447fef68dc35ee9d8e4c9a8603e5ad29d7dc191a83b76ead173a3b6a124"
Nov 03 01:48:05.922568 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:48:05.922544    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a4f447fef68dc35ee9d8e4c9a8603e5ad29d7dc191a83b76ead173a3b6a124\": container with ID starting with 31a4f447fef68dc35ee9d8e4c9a8603e5ad29d7dc191a83b76ead173a3b6a124 not found: ID does not exist" containerID="31a4f447fef68dc35ee9d8e4c9a8603e5ad29d7dc191a83b76ead173a3b6a124"
Nov 03 01:48:05.922645 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:05.922584    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a4f447fef68dc35ee9d8e4c9a8603e5ad29d7dc191a83b76ead173a3b6a124"} err="failed to get container status \"31a4f447fef68dc35ee9d8e4c9a8603e5ad29d7dc191a83b76ead173a3b6a124\": rpc error: code = NotFound desc = could not find container \"31a4f447fef68dc35ee9d8e4c9a8603e5ad29d7dc191a83b76ead173a3b6a124\": container with ID starting with 31a4f447fef68dc35ee9d8e4c9a8603e5ad29d7dc191a83b76ead173a3b6a124 not found: ID does not exist"
Nov 03 01:48:06.009888 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:06.009849    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:48:06.057487 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:06.057454    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"]
Nov 03 01:48:06.062956 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:06.062929    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4e704-56ccf598bc-p92dz"]
Nov 03 01:48:06.163564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:06.163474    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"]
Nov 03 01:48:06.168972 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:06.168940    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f4e704-75d769b6ff-mhrg7"]
Nov 03 01:48:06.714492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:06.714459    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01dc3983-c36e-4e40-a7ea-a61d27f24d4a" path="/var/lib/kubelet/pods/01dc3983-c36e-4e40-a7ea-a61d27f24d4a/volumes"
Nov 03 01:48:06.715843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:06.715813    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d323ac-8a9a-422a-b97c-26c7e9b096ca" path="/var/lib/kubelet/pods/65d323ac-8a9a-422a-b97c-26c7e9b096ca/volumes"
Nov 03 01:48:11.434708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:11.434668    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"]
Nov 03 01:48:11.541981 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:11.541944    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"]
Nov 03 01:48:13.816262 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:48:13.816220    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:48:16.435422 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.435319    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb" podUID="c17e7840-e240-40eb-b0b0-8ba0a790dd8f" containerName="payload-processing" containerID="cri-o://ff74c3d8827fba4726c600a2a43b5529aed388e76bb1bea3b188ff401d20d27b" gracePeriod=25
Nov 03 01:48:16.470112 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.470073    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"]
Nov 03 01:48:16.470734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.470721    5508 cpu_manager.go:414] "containerMap: removing container" podUID="03b5ab3a-3ae0-4313-9619-42fdc2ca9682" containerName="payload-processing"
Nov 03 01:48:16.470783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.470735    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b5ab3a-3ae0-4313-9619-42fdc2ca9682" containerName="payload-processing"
Nov 03 01:48:16.470783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.470767    5508 cpu_manager.go:414] "containerMap: removing container" podUID="65d323ac-8a9a-422a-b97c-26c7e9b096ca" containerName="payload-processing"
Nov 03 01:48:16.470783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.470773    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d323ac-8a9a-422a-b97c-26c7e9b096ca" containerName="payload-processing"
Nov 03 01:48:16.470880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.470784    5508 cpu_manager.go:414] "containerMap: removing container" podUID="01dc3983-c36e-4e40-a7ea-a61d27f24d4a" containerName="payload-pre-processing"
Nov 03 01:48:16.470880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.470789    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01dc3983-c36e-4e40-a7ea-a61d27f24d4a" containerName="payload-pre-processing"
Nov 03 01:48:16.470880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.470869    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b5ab3a-3ae0-4313-9619-42fdc2ca9682" containerName="payload-processing"
Nov 03 01:48:16.470969 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.470882    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d323ac-8a9a-422a-b97c-26c7e9b096ca" containerName="payload-processing"
Nov 03 01:48:16.470969 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.470890    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="01dc3983-c36e-4e40-a7ea-a61d27f24d4a" containerName="payload-pre-processing"
Nov 03 01:48:16.474554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.474535    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:16.525151 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.525106    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-23e673-serving-cert\""
Nov 03 01:48:16.542934 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.542884    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j" podUID="424066ff-90ea-4e4b-a556-9830b0835d26" containerName="payload-pre-processing" containerID="cri-o://82799efa8bbeed7572e5449a15cd1656a296530ec5fb6a2d89af122a0a4fd76a" gracePeriod=25
Nov 03 01:48:16.588144 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.588095    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"]
Nov 03 01:48:16.608412 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.608381    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-maas-api-tls\") pod \"maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:16.608622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.608499    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e472801-3966-49f6-979d-3c2943800101-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:16.608622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.608588    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-metrics-certs\") pod \"maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:16.608741 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.608622    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmpt4\" (UniqueName: \"kubernetes.io/projected/5e472801-3966-49f6-979d-3c2943800101-kube-api-access-xmpt4\") pod \"maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:16.710573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.709831    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-maas-api-tls\") pod \"maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:16.710573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.709942    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e472801-3966-49f6-979d-3c2943800101-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:16.710573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.710033    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-metrics-certs\") pod \"maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:16.710573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.710077    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmpt4\" (UniqueName: \"kubernetes.io/projected/5e472801-3966-49f6-979d-3c2943800101-kube-api-access-xmpt4\") pod \"maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:16.710573 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:48:16.710198    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-serving-cert: secret "maas-api-e2e-ipp-a-23e673-serving-cert" not found
Nov 03 01:48:16.710573 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:48:16.710265    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-maas-api-tls podName:5e472801-3966-49f6-979d-3c2943800101 nodeName:}" failed. No retries permitted until 2026-11-03 01:48:17.210248192 +0000 UTC m=+1776.635854639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-maas-api-tls") pod "maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz" (UID: "5e472801-3966-49f6-979d-3c2943800101") : secret "maas-api-e2e-ipp-a-23e673-serving-cert" not found
Nov 03 01:48:16.711126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.711101    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e472801-3966-49f6-979d-3c2943800101-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:16.713013 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.712989    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-metrics-certs\") pod \"maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:16.808298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.802606    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmpt4\" (UniqueName: \"kubernetes.io/projected/5e472801-3966-49f6-979d-3c2943800101-kube-api-access-xmpt4\") pod \"maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:16.947630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.947006    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"
Nov 03 01:48:16.982422 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:16.979661    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"
Nov 03 01:48:17.012846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.012809    5508 generic.go:356] "Generic (PLEG): container finished" podID="424066ff-90ea-4e4b-a556-9830b0835d26" containerID="82799efa8bbeed7572e5449a15cd1656a296530ec5fb6a2d89af122a0a4fd76a" exitCode=0
Nov 03 01:48:17.013050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.012862    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"
Nov 03 01:48:17.013050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.012892    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j" event={"ID":"424066ff-90ea-4e4b-a556-9830b0835d26","Type":"ContainerDied","Data":"82799efa8bbeed7572e5449a15cd1656a296530ec5fb6a2d89af122a0a4fd76a"}
Nov 03 01:48:17.013050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.012951    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j" event={"ID":"424066ff-90ea-4e4b-a556-9830b0835d26","Type":"ContainerDied","Data":"2e2e03e0f531e3fea92828133b361ca564aceaab38fbc72de8731f33fb7ee2e3"}
Nov 03 01:48:17.013050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.012981    5508 scope.go:122] "RemoveContainer" containerID="82799efa8bbeed7572e5449a15cd1656a296530ec5fb6a2d89af122a0a4fd76a"
Nov 03 01:48:17.014434 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.014413    5508 generic.go:356] "Generic (PLEG): container finished" podID="c17e7840-e240-40eb-b0b0-8ba0a790dd8f" containerID="ff74c3d8827fba4726c600a2a43b5529aed388e76bb1bea3b188ff401d20d27b" exitCode=0
Nov 03 01:48:17.014544 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.014465    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"
Nov 03 01:48:17.014544 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.014512    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb" event={"ID":"c17e7840-e240-40eb-b0b0-8ba0a790dd8f","Type":"ContainerDied","Data":"ff74c3d8827fba4726c600a2a43b5529aed388e76bb1bea3b188ff401d20d27b"}
Nov 03 01:48:17.014625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.014572    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb" event={"ID":"c17e7840-e240-40eb-b0b0-8ba0a790dd8f","Type":"ContainerDied","Data":"68ec6a0ad5b4654b2e42740ef89e8cd379fd8c5a2ba4461aa2eb33a7f96d6d58"}
Nov 03 01:48:17.023137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.023121    5508 scope.go:122] "RemoveContainer" containerID="82799efa8bbeed7572e5449a15cd1656a296530ec5fb6a2d89af122a0a4fd76a"
Nov 03 01:48:17.023452 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:48:17.023426    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82799efa8bbeed7572e5449a15cd1656a296530ec5fb6a2d89af122a0a4fd76a\": container with ID starting with 82799efa8bbeed7572e5449a15cd1656a296530ec5fb6a2d89af122a0a4fd76a not found: ID does not exist" containerID="82799efa8bbeed7572e5449a15cd1656a296530ec5fb6a2d89af122a0a4fd76a"
Nov 03 01:48:17.023514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.023450    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82799efa8bbeed7572e5449a15cd1656a296530ec5fb6a2d89af122a0a4fd76a"} err="failed to get container status \"82799efa8bbeed7572e5449a15cd1656a296530ec5fb6a2d89af122a0a4fd76a\": rpc error: code = NotFound desc = could not find container \"82799efa8bbeed7572e5449a15cd1656a296530ec5fb6a2d89af122a0a4fd76a\": container with ID starting with 82799efa8bbeed7572e5449a15cd1656a296530ec5fb6a2d89af122a0a4fd76a not found: ID does not exist"
Nov 03 01:48:17.023514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.023468    5508 scope.go:122] "RemoveContainer" containerID="ff74c3d8827fba4726c600a2a43b5529aed388e76bb1bea3b188ff401d20d27b"
Nov 03 01:48:17.032901 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.032883    5508 scope.go:122] "RemoveContainer" containerID="ff74c3d8827fba4726c600a2a43b5529aed388e76bb1bea3b188ff401d20d27b"
Nov 03 01:48:17.033189 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:48:17.033165    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff74c3d8827fba4726c600a2a43b5529aed388e76bb1bea3b188ff401d20d27b\": container with ID starting with ff74c3d8827fba4726c600a2a43b5529aed388e76bb1bea3b188ff401d20d27b not found: ID does not exist" containerID="ff74c3d8827fba4726c600a2a43b5529aed388e76bb1bea3b188ff401d20d27b"
Nov 03 01:48:17.033241 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.033196    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff74c3d8827fba4726c600a2a43b5529aed388e76bb1bea3b188ff401d20d27b"} err="failed to get container status \"ff74c3d8827fba4726c600a2a43b5529aed388e76bb1bea3b188ff401d20d27b\": rpc error: code = NotFound desc = could not find container \"ff74c3d8827fba4726c600a2a43b5529aed388e76bb1bea3b188ff401d20d27b\": container with ID starting with ff74c3d8827fba4726c600a2a43b5529aed388e76bb1bea3b188ff401d20d27b not found: ID does not exist"
Nov 03 01:48:17.040809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.040782    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424066ff-90ea-4e4b-a556-9830b0835d26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424066ff-90ea-4e4b-a556-9830b0835d26-plugins-config-volume\") pod \"424066ff-90ea-4e4b-a556-9830b0835d26\" (UID: \"424066ff-90ea-4e4b-a556-9830b0835d26\") "
Nov 03 01:48:17.040910 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.040828    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-plugins-config-volume\") pod \"c17e7840-e240-40eb-b0b0-8ba0a790dd8f\" (UID: \"c17e7840-e240-40eb-b0b0-8ba0a790dd8f\") "
Nov 03 01:48:17.040910 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.040878    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-kube-api-access-5rjwv\" (UniqueName: \"kubernetes.io/projected/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-kube-api-access-5rjwv\") pod \"c17e7840-e240-40eb-b0b0-8ba0a790dd8f\" (UID: \"c17e7840-e240-40eb-b0b0-8ba0a790dd8f\") "
Nov 03 01:48:17.040993 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.040940    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424066ff-90ea-4e4b-a556-9830b0835d26-kube-api-access-v2t5c\" (UniqueName: \"kubernetes.io/projected/424066ff-90ea-4e4b-a556-9830b0835d26-kube-api-access-v2t5c\") pod \"424066ff-90ea-4e4b-a556-9830b0835d26\" (UID: \"424066ff-90ea-4e4b-a556-9830b0835d26\") "
Nov 03 01:48:17.041243 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.041212    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424066ff-90ea-4e4b-a556-9830b0835d26-plugins-config-volume" pod "424066ff-90ea-4e4b-a556-9830b0835d26" (UID: "424066ff-90ea-4e4b-a556-9830b0835d26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:48:17.041318 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.041213    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-plugins-config-volume" pod "c17e7840-e240-40eb-b0b0-8ba0a790dd8f" (UID: "c17e7840-e240-40eb-b0b0-8ba0a790dd8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:48:17.041539 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.041523    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/424066ff-90ea-4e4b-a556-9830b0835d26-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:17.041598 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.041542    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:17.043073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.043052    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-kube-api-access-5rjwv" pod "c17e7840-e240-40eb-b0b0-8ba0a790dd8f" (UID: "c17e7840-e240-40eb-b0b0-8ba0a790dd8f"). InnerVolumeSpecName "kube-api-access-5rjwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:48:17.043141 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.043084    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424066ff-90ea-4e4b-a556-9830b0835d26-kube-api-access-v2t5c" pod "424066ff-90ea-4e4b-a556-9830b0835d26" (UID: "424066ff-90ea-4e4b-a556-9830b0835d26"). InnerVolumeSpecName "kube-api-access-v2t5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:48:17.142899 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.142856    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rjwv\" (UniqueName: \"kubernetes.io/projected/c17e7840-e240-40eb-b0b0-8ba0a790dd8f-kube-api-access-5rjwv\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:17.142899 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.142893    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2t5c\" (UniqueName: \"kubernetes.io/projected/424066ff-90ea-4e4b-a556-9830b0835d26-kube-api-access-v2t5c\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:17.244616 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.244524    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-maas-api-tls\") pod \"maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:17.246974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.246954    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-maas-api-tls\") pod \"maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:17.431767 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.431732    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"]
Nov 03 01:48:17.437744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.437710    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2b9f22c1-866c44f89c-ft7nb"]
Nov 03 01:48:17.438164 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.437895    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:17.544420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.544384    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"]
Nov 03 01:48:17.564333 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.561427    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2b9f22c1-7cd5c774dc-52h7j"]
Nov 03 01:48:17.824738 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:17.824653    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"]
Nov 03 01:48:17.830885 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:48:17.830850    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e472801_3966_49f6_979d_3c2943800101.slice/crio-7cdde1073e70bb06169d098b716715ffb76b2add1840d328473815b40a6beda8 WatchSource:0}: Error finding container 7cdde1073e70bb06169d098b716715ffb76b2add1840d328473815b40a6beda8: Status 404 returned error can't find the container with id 7cdde1073e70bb06169d098b716715ffb76b2add1840d328473815b40a6beda8
Nov 03 01:48:18.024878 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:18.024844    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz" event={"ID":"5e472801-3966-49f6-979d-3c2943800101","Type":"ContainerStarted","Data":"7cdde1073e70bb06169d098b716715ffb76b2add1840d328473815b40a6beda8"}
Nov 03 01:48:18.694187 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:18.694154    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424066ff-90ea-4e4b-a556-9830b0835d26" path="/var/lib/kubelet/pods/424066ff-90ea-4e4b-a556-9830b0835d26/volumes"
Nov 03 01:48:18.694694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:18.694678    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c17e7840-e240-40eb-b0b0-8ba0a790dd8f" path="/var/lib/kubelet/pods/c17e7840-e240-40eb-b0b0-8ba0a790dd8f/volumes"
Nov 03 01:48:19.033956 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:19.033918    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz" event={"ID":"5e472801-3966-49f6-979d-3c2943800101","Type":"ContainerStarted","Data":"4d10e741076b0f8ce81783a4d9ea532a62e1204b4fd71de03acd76462e7e100b"}
Nov 03 01:48:19.034150 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:19.034012    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:19.156104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:19.156038    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz" podStartSLOduration=2.111926267 podStartE2EDuration="3.156020187s" totalImagesPullingTime="1.04409392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:48:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:48:19.152601061 +0000 UTC m=+1778.578207538" watchObservedRunningTime="2026-11-03 01:48:19.156020187 +0000 UTC m=+1778.581626653"
Nov 03 01:48:25.049400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:25.049370    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:48:27.274649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.274609    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"]
Nov 03 01:48:27.275581 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.275562    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c17e7840-e240-40eb-b0b0-8ba0a790dd8f" containerName="payload-processing"
Nov 03 01:48:27.275638 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.275584    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17e7840-e240-40eb-b0b0-8ba0a790dd8f" containerName="payload-processing"
Nov 03 01:48:27.275638 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.275620    5508 cpu_manager.go:414] "containerMap: removing container" podUID="424066ff-90ea-4e4b-a556-9830b0835d26" containerName="payload-pre-processing"
Nov 03 01:48:27.275638 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.275628    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424066ff-90ea-4e4b-a556-9830b0835d26" containerName="payload-pre-processing"
Nov 03 01:48:27.275763 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.275752    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="424066ff-90ea-4e4b-a556-9830b0835d26" containerName="payload-pre-processing"
Nov 03 01:48:27.275797 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.275776    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17e7840-e240-40eb-b0b0-8ba0a790dd8f" containerName="payload-processing"
Nov 03 01:48:27.279869 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.279847    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:27.332120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.332087    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"]
Nov 03 01:48:27.371965 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.371920    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-metrics-certs\") pod \"maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:27.372172 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.372049    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:27.372172 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.372101    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl4tw\" (UniqueName: \"kubernetes.io/projected/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-kube-api-access-xl4tw\") pod \"maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:27.372172 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.372145    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:27.373488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.373458    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7bcbbb-serving-cert\""
Nov 03 01:48:27.473833 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.473798    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-metrics-certs\") pod \"maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:27.474018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.473894    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:27.474018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.473949    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl4tw\" (UniqueName: \"kubernetes.io/projected/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-kube-api-access-xl4tw\") pod \"maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:27.474112 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.474015    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:27.474216 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:48:27.474195    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-serving-cert: secret "maas-api-e2e-ipp-a-7bcbbb-serving-cert" not found
Nov 03 01:48:27.474300 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:48:27.474288    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-maas-api-tls podName:8a092c54-5ed5-4bb9-b65b-e80a1a5122cb nodeName:}" failed. No retries permitted until 2026-11-03 01:48:27.974266312 +0000 UTC m=+1787.399872759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-maas-api-tls") pod "maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d" (UID: "8a092c54-5ed5-4bb9-b65b-e80a1a5122cb") : secret "maas-api-e2e-ipp-a-7bcbbb-serving-cert" not found
Nov 03 01:48:27.474645 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.474625    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:27.476436 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.476414    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-metrics-certs\") pod \"maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:27.571421 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.570523    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl4tw\" (UniqueName: \"kubernetes.io/projected/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-kube-api-access-xl4tw\") pod \"maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:27.990382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.990320    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:27.993993 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:27.993965    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:28.194759 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:28.194710    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:28.579493 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:28.579412    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"]
Nov 03 01:48:28.585290 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:48:28.585242    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a092c54_5ed5_4bb9_b65b_e80a1a5122cb.slice/crio-57e84b00381bd033e70e248ba0d2009ccb20cf98b9d7ab171752998021cf46b1 WatchSource:0}: Error finding container 57e84b00381bd033e70e248ba0d2009ccb20cf98b9d7ab171752998021cf46b1: Status 404 returned error can't find the container with id 57e84b00381bd033e70e248ba0d2009ccb20cf98b9d7ab171752998021cf46b1
Nov 03 01:48:29.114794 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:29.114758    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d" event={"ID":"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb","Type":"ContainerStarted","Data":"57e84b00381bd033e70e248ba0d2009ccb20cf98b9d7ab171752998021cf46b1"}
Nov 03 01:48:30.124470 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:30.124438    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d" event={"ID":"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb","Type":"ContainerStarted","Data":"ccf0659b9a9b7611d0d4bed59fb260d2868cdbc504048e24ecebc79f56444ae2"}
Nov 03 01:48:30.124895 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:30.124598    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:30.241370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:30.237461    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d" podStartSLOduration=2.292170404 podStartE2EDuration="3.237442488s" totalImagesPullingTime="945.272084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:48:30.236795925 +0000 UTC m=+1789.662402415" watchObservedRunningTime="2026-11-03 01:48:30.237442488 +0000 UTC m=+1789.663048957"
Nov 03 01:48:31.975887 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:31.975837    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"]
Nov 03 01:48:31.976363 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:31.976096    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g" podUID="b14432b6-e3db-4a33-aaa8-1dd9c3e96305" containerName="istio-proxy" containerID="cri-o://eb71cfa854eff391feb46bca6128de54aa7c5015491e3dec5fe954cfcc1f1d92" gracePeriod=30
Nov 03 01:48:31.992224 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:31.992188    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"]
Nov 03 01:48:31.992438 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:31.992410    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy" containerID="cri-o://6fc0ee606e99d3142d49459afdc65404baa6682d46e026e0818ddc66a3505d5c" gracePeriod=30
Nov 03 01:48:35.576900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:35.576859    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:48:35.577329 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:35.576933    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g" podUID="b14432b6-e3db-4a33-aaa8-1dd9c3e96305" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:48:36.087169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:36.087121    5508 patch_prober.go:29] interesting pod/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:48:36.087340 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:36.087208    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:48:36.165191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:36.165162    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:48:37.231476 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.231444    5508 generic.go:356] "Generic (PLEG): container finished" podID="b14432b6-e3db-4a33-aaa8-1dd9c3e96305" containerID="eb71cfa854eff391feb46bca6128de54aa7c5015491e3dec5fe954cfcc1f1d92" exitCode=0
Nov 03 01:48:37.232516 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.231500    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g" event={"ID":"b14432b6-e3db-4a33-aaa8-1dd9c3e96305","Type":"ContainerDied","Data":"eb71cfa854eff391feb46bca6128de54aa7c5015491e3dec5fe954cfcc1f1d92"}
Nov 03 01:48:37.235860 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.235831    5508 generic.go:356] "Generic (PLEG): container finished" podID="c55a9f34-b745-4306-83d9-5460d87832e5" containerID="6fc0ee606e99d3142d49459afdc65404baa6682d46e026e0818ddc66a3505d5c" exitCode=0
Nov 03 01:48:37.236078 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.236058    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" event={"ID":"c55a9f34-b745-4306-83d9-5460d87832e5","Type":"ContainerDied","Data":"6fc0ee606e99d3142d49459afdc65404baa6682d46e026e0818ddc66a3505d5c"}
Nov 03 01:48:37.236154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.236119    5508 scope.go:122] "RemoveContainer" containerID="40be624633397e27b1ce74e2571f33917b777b75e8b286edcfa494655021de62"
Nov 03 01:48:37.444402 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.444368    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:48:37.446537 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.446513    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:48:37.516783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.516692    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-socket\") pod \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") "
Nov 03 01:48:37.516783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.516742    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-certs\") pod \"c55a9f34-b745-4306-83d9-5460d87832e5\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") "
Nov 03 01:48:37.516783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.516773    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c55a9f34-b745-4306-83d9-5460d87832e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55a9f34-b745-4306-83d9-5460d87832e5-istio-podinfo\") pod \"c55a9f34-b745-4306-83d9-5460d87832e5\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") "
Nov 03 01:48:37.517071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.516805    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-socket\") pod \"c55a9f34-b745-4306-83d9-5460d87832e5\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") "
Nov 03 01:48:37.517071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.516834    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-istio-token\" (UniqueName: \"kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-istio-token\") pod \"c55a9f34-b745-4306-83d9-5460d87832e5\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") "
Nov 03 01:48:37.517071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.516853    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-envoy\") pod \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") "
Nov 03 01:48:37.517071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.516873    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-credential-socket\") pod \"c55a9f34-b745-4306-83d9-5460d87832e5\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") "
Nov 03 01:48:37.517071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.516899    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-envoy\") pod \"c55a9f34-b745-4306-83d9-5460d87832e5\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") "
Nov 03 01:48:37.517071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.516930    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-data\") pod \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") "
Nov 03 01:48:37.517071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.516965    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-podinfo\") pod \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") "
Nov 03 01:48:37.517071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.517012    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-kube-api-access-phzpj\" (UniqueName: \"kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-kube-api-access-phzpj\") pod \"c55a9f34-b745-4306-83d9-5460d87832e5\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") "
Nov 03 01:48:37.517071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.517060    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-certs\") pod \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") "
Nov 03 01:48:37.517574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.517076    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-certs" pod "c55a9f34-b745-4306-83d9-5460d87832e5" (UID: "c55a9f34-b745-4306-83d9-5460d87832e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:37.517574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.517082    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-socket" pod "b14432b6-e3db-4a33-aaa8-1dd9c3e96305" (UID: "b14432b6-e3db-4a33-aaa8-1dd9c3e96305"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:37.517574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.517111    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istiod-ca-cert\") pod \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") "
Nov 03 01:48:37.517574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.517448    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-credential-socket" pod "c55a9f34-b745-4306-83d9-5460d87832e5" (UID: "c55a9f34-b745-4306-83d9-5460d87832e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:37.517814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.517639    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-socket" pod "c55a9f34-b745-4306-83d9-5460d87832e5" (UID: "c55a9f34-b745-4306-83d9-5460d87832e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:37.517814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.517773    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-certs" pod "b14432b6-e3db-4a33-aaa8-1dd9c3e96305" (UID: "b14432b6-e3db-4a33-aaa8-1dd9c3e96305"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:37.517929 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.517880    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istiod-ca-cert" pod "b14432b6-e3db-4a33-aaa8-1dd9c3e96305" (UID: "b14432b6-e3db-4a33-aaa8-1dd9c3e96305"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:48:37.518117 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.518075    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-data" pod "b14432b6-e3db-4a33-aaa8-1dd9c3e96305" (UID: "b14432b6-e3db-4a33-aaa8-1dd9c3e96305"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:37.518217 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.518181    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-token\" (UniqueName: \"kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-token\") pod \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") "
Nov 03 01:48:37.518869 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.518270    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-data\") pod \"c55a9f34-b745-4306-83d9-5460d87832e5\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") "
Nov 03 01:48:37.518869 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.518313    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-credential-socket\") pod \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") "
Nov 03 01:48:37.518869 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.518373    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55a9f34-b745-4306-83d9-5460d87832e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55a9f34-b745-4306-83d9-5460d87832e5-istiod-ca-cert\") pod \"c55a9f34-b745-4306-83d9-5460d87832e5\" (UID: \"c55a9f34-b745-4306-83d9-5460d87832e5\") "
Nov 03 01:48:37.518869 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.518419    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-kube-api-access-hmdp5\" (UniqueName: \"kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-kube-api-access-hmdp5\") pod \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\" (UID: \"b14432b6-e3db-4a33-aaa8-1dd9c3e96305\") "
Nov 03 01:48:37.518869 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.518529    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-data" pod "c55a9f34-b745-4306-83d9-5460d87832e5" (UID: "c55a9f34-b745-4306-83d9-5460d87832e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:37.518869 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.518606    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-credential-socket" pod "b14432b6-e3db-4a33-aaa8-1dd9c3e96305" (UID: "b14432b6-e3db-4a33-aaa8-1dd9c3e96305"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:37.518869 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.518791    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55a9f34-b745-4306-83d9-5460d87832e5-istiod-ca-cert" pod "c55a9f34-b745-4306-83d9-5460d87832e5" (UID: "c55a9f34-b745-4306-83d9-5460d87832e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:48:37.519246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.519116    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.519246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.519136    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.519246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.519149    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.519246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.519164    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.519246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.519177    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c55a9f34-b745-4306-83d9-5460d87832e5-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.519246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.519190    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.519246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.519204    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.519246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.519221    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.519246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.519235    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.519246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.519248    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.520006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.519980    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-kube-api-access-phzpj" pod "c55a9f34-b745-4306-83d9-5460d87832e5" (UID: "c55a9f34-b745-4306-83d9-5460d87832e5"). InnerVolumeSpecName "kube-api-access-phzpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:48:37.520959 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.520939    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-kube-api-access-hmdp5" pod "b14432b6-e3db-4a33-aaa8-1dd9c3e96305" (UID: "b14432b6-e3db-4a33-aaa8-1dd9c3e96305"). InnerVolumeSpecName "kube-api-access-hmdp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:48:37.648394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.647756    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phzpj\" (UniqueName: \"kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-kube-api-access-phzpj\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.648394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.647787    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmdp5\" (UniqueName: \"kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-kube-api-access-hmdp5\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.649418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.649383    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-envoy" pod "b14432b6-e3db-4a33-aaa8-1dd9c3e96305" (UID: "b14432b6-e3db-4a33-aaa8-1dd9c3e96305"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:37.649564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.649547    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c55a9f34-b745-4306-83d9-5460d87832e5-istio-podinfo" pod "c55a9f34-b745-4306-83d9-5460d87832e5" (UID: "c55a9f34-b745-4306-83d9-5460d87832e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:48:37.649695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.649669    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-istio-token" pod "c55a9f34-b745-4306-83d9-5460d87832e5" (UID: "c55a9f34-b745-4306-83d9-5460d87832e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:48:37.650260 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.650239    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-podinfo" pod "b14432b6-e3db-4a33-aaa8-1dd9c3e96305" (UID: "b14432b6-e3db-4a33-aaa8-1dd9c3e96305"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:48:37.652256 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.652223    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-token" pod "b14432b6-e3db-4a33-aaa8-1dd9c3e96305" (UID: "b14432b6-e3db-4a33-aaa8-1dd9c3e96305"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:48:37.666474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.666434    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-envoy" pod "c55a9f34-b745-4306-83d9-5460d87832e5" (UID: "c55a9f34-b745-4306-83d9-5460d87832e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:37.750092 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.750053    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.750092 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.750092    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c55a9f34-b745-4306-83d9-5460d87832e5-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.750293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.750107    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c55a9f34-b745-4306-83d9-5460d87832e5-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.750293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.750121    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.750293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.750135    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c55a9f34-b745-4306-83d9-5460d87832e5-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:37.750293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:37.750151    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b14432b6-e3db-4a33-aaa8-1dd9c3e96305-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:38.270572 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:38.270488    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"
Nov 03 01:48:38.271154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:38.271123    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g" event={"ID":"c55a9f34-b745-4306-83d9-5460d87832e5","Type":"ContainerDied","Data":"787a3db19bf3d542d642d7f74c1632991d6803174f3d0eb3977c92ea3fef8d2c"}
Nov 03 01:48:38.271567 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:38.271547    5508 scope.go:122] "RemoveContainer" containerID="6fc0ee606e99d3142d49459afdc65404baa6682d46e026e0818ddc66a3505d5c"
Nov 03 01:48:38.277496 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:38.277465    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g" event={"ID":"b14432b6-e3db-4a33-aaa8-1dd9c3e96305","Type":"ContainerDied","Data":"edbae903cd44076b2c5899349ec4a8337c27ed325e4ab412bb53f61ea0dd2d50"}
Nov 03 01:48:38.277737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:38.277721    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"
Nov 03 01:48:38.302715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:38.302696    5508 scope.go:122] "RemoveContainer" containerID="eb71cfa854eff391feb46bca6128de54aa7c5015491e3dec5fe954cfcc1f1d92"
Nov 03 01:48:38.474512 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:38.474468    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"]
Nov 03 01:48:38.521067 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:38.520991    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-twv4g"]
Nov 03 01:48:38.622769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:38.622734    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"]
Nov 03 01:48:38.626756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:38.626729    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f4e704-openshift-default-576d67d949-s2c4g"]
Nov 03 01:48:38.692631 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:38.692600    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b14432b6-e3db-4a33-aaa8-1dd9c3e96305" path="/var/lib/kubelet/pods/b14432b6-e3db-4a33-aaa8-1dd9c3e96305/volumes"
Nov 03 01:48:38.693128 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:38.693112    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" path="/var/lib/kubelet/pods/c55a9f34-b745-4306-83d9-5460d87832e5/volumes"
Nov 03 01:48:39.452446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:39.452410    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:48:40.025490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:40.025441    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"]
Nov 03 01:48:40.135714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:40.135680    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"]
Nov 03 01:48:41.028249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:41.028223    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:48:41.037473 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:41.037448    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:48:43.835767 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:48:43.835728    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:48:44.142624 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:44.142544    5508 scope.go:122] "RemoveContainer" containerID="d95369dca4cd8c680bc40497f0916ad4218de6fa2394ed314974bdb7e216284c"
Nov 03 01:48:44.195383 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:44.194315    5508 scope.go:122] "RemoveContainer" containerID="9d7d1c3a7d7ab22a27d1ee921cc579f206be6628736bb7a70848cbc6fe0e709a"
Nov 03 01:48:45.026139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:45.026100    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt" podUID="305befcb-aafc-48ca-8469-49697d33d49e" containerName="payload-processing" containerID="cri-o://cbc04ddac35bd879adfa55f4ddc01731273f7670aee4f8bd3163b714738edb76" gracePeriod=25
Nov 03 01:48:45.138588 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:45.138545    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5" podUID="f91d5bd0-8541-4a70-a6e7-06db5c88e2df" containerName="payload-pre-processing" containerID="cri-o://09409ce142a65af149a7ecf1d2cc343995ac6ddc7b915e7bbd9e90a4ef22c93c" gracePeriod=25
Nov 03 01:48:45.346171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:45.346058    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"]
Nov 03 01:48:45.346905 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:45.346834    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b" podUID="f930409c-7286-4a06-bb5f-93015858b97e" containerName="istio-proxy" containerID="cri-o://b149004c5b6a469ab40d04d78926d4ccd813ff21eaed14513b2144d8393854c5" gracePeriod=30
Nov 03 01:48:45.568380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:45.567198    5508 generic.go:356] "Generic (PLEG): container finished" podID="f91d5bd0-8541-4a70-a6e7-06db5c88e2df" containerID="09409ce142a65af149a7ecf1d2cc343995ac6ddc7b915e7bbd9e90a4ef22c93c" exitCode=0
Nov 03 01:48:45.568380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:45.567396    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5" event={"ID":"f91d5bd0-8541-4a70-a6e7-06db5c88e2df","Type":"ContainerDied","Data":"09409ce142a65af149a7ecf1d2cc343995ac6ddc7b915e7bbd9e90a4ef22c93c"}
Nov 03 01:48:45.611451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:45.598000    5508 generic.go:356] "Generic (PLEG): container finished" podID="305befcb-aafc-48ca-8469-49697d33d49e" containerID="cbc04ddac35bd879adfa55f4ddc01731273f7670aee4f8bd3163b714738edb76" exitCode=0
Nov 03 01:48:45.611451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:45.598154    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt" event={"ID":"305befcb-aafc-48ca-8469-49697d33d49e","Type":"ContainerDied","Data":"cbc04ddac35bd879adfa55f4ddc01731273f7670aee4f8bd3163b714738edb76"}
Nov 03 01:48:46.001463 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:45.975568    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"
Nov 03 01:48:46.118885 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.118848    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305befcb-aafc-48ca-8469-49697d33d49e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/305befcb-aafc-48ca-8469-49697d33d49e-plugins-config-volume\") pod \"305befcb-aafc-48ca-8469-49697d33d49e\" (UID: \"305befcb-aafc-48ca-8469-49697d33d49e\") "
Nov 03 01:48:46.119617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.119593    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305befcb-aafc-48ca-8469-49697d33d49e-kube-api-access-729b7\" (UniqueName: \"kubernetes.io/projected/305befcb-aafc-48ca-8469-49697d33d49e-kube-api-access-729b7\") pod \"305befcb-aafc-48ca-8469-49697d33d49e\" (UID: \"305befcb-aafc-48ca-8469-49697d33d49e\") "
Nov 03 01:48:46.121301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.121257    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305befcb-aafc-48ca-8469-49697d33d49e-plugins-config-volume" pod "305befcb-aafc-48ca-8469-49697d33d49e" (UID: "305befcb-aafc-48ca-8469-49697d33d49e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:48:46.124837 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.124811    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305befcb-aafc-48ca-8469-49697d33d49e-kube-api-access-729b7" pod "305befcb-aafc-48ca-8469-49697d33d49e" (UID: "305befcb-aafc-48ca-8469-49697d33d49e"). InnerVolumeSpecName "kube-api-access-729b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:48:46.221939 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.221858    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/305befcb-aafc-48ca-8469-49697d33d49e-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:46.221939 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.221887    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-729b7\" (UniqueName: \"kubernetes.io/projected/305befcb-aafc-48ca-8469-49697d33d49e-kube-api-access-729b7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:46.224633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.224608    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"
Nov 03 01:48:46.323100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.323069    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-kube-api-access-t64kg\" (UniqueName: \"kubernetes.io/projected/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-kube-api-access-t64kg\") pod \"f91d5bd0-8541-4a70-a6e7-06db5c88e2df\" (UID: \"f91d5bd0-8541-4a70-a6e7-06db5c88e2df\") "
Nov 03 01:48:46.323301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.323149    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-plugins-config-volume\") pod \"f91d5bd0-8541-4a70-a6e7-06db5c88e2df\" (UID: \"f91d5bd0-8541-4a70-a6e7-06db5c88e2df\") "
Nov 03 01:48:46.323960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.323919    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-plugins-config-volume" pod "f91d5bd0-8541-4a70-a6e7-06db5c88e2df" (UID: "f91d5bd0-8541-4a70-a6e7-06db5c88e2df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:48:46.326742 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.326475    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-kube-api-access-t64kg" pod "f91d5bd0-8541-4a70-a6e7-06db5c88e2df" (UID: "f91d5bd0-8541-4a70-a6e7-06db5c88e2df"). InnerVolumeSpecName "kube-api-access-t64kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:48:46.426982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.426945    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t64kg\" (UniqueName: \"kubernetes.io/projected/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-kube-api-access-t64kg\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:46.426982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.426983    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f91d5bd0-8541-4a70-a6e7-06db5c88e2df-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:46.606649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.606558    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"
Nov 03 01:48:46.606649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.606566    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5" event={"ID":"f91d5bd0-8541-4a70-a6e7-06db5c88e2df","Type":"ContainerDied","Data":"6843bb5e28e8dfcb36c7791caa2c01aa7ea067116b577a675548e9263d969008"}
Nov 03 01:48:46.606889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.606682    5508 scope.go:122] "RemoveContainer" containerID="09409ce142a65af149a7ecf1d2cc343995ac6ddc7b915e7bbd9e90a4ef22c93c"
Nov 03 01:48:46.608728 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.608700    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt" event={"ID":"305befcb-aafc-48ca-8469-49697d33d49e","Type":"ContainerDied","Data":"2a8033f851325ce2bf136c3d1536237a08bde74c2f9fdfbf756e9c53aad1c89a"}
Nov 03 01:48:46.608898 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.608767    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"
Nov 03 01:48:46.622186 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.622164    5508 scope.go:122] "RemoveContainer" containerID="cbc04ddac35bd879adfa55f4ddc01731273f7670aee4f8bd3163b714738edb76"
Nov 03 01:48:46.767324 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.767287    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"]
Nov 03 01:48:46.768228 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.768208    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-109234-5676c44df9-9b6mt"]
Nov 03 01:48:46.889839 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.889738    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"]
Nov 03 01:48:46.914898 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:46.914824    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-109234-9bd4c54ff-scls5"]
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.116506    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"]
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117405    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117424    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117445    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b14432b6-e3db-4a33-aaa8-1dd9c3e96305" containerName="istio-proxy"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117455    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14432b6-e3db-4a33-aaa8-1dd9c3e96305" containerName="istio-proxy"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117479    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117487    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117517    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f91d5bd0-8541-4a70-a6e7-06db5c88e2df" containerName="payload-pre-processing"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117525    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91d5bd0-8541-4a70-a6e7-06db5c88e2df" containerName="payload-pre-processing"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117548    5508 cpu_manager.go:414] "containerMap: removing container" podUID="305befcb-aafc-48ca-8469-49697d33d49e" containerName="payload-processing"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117555    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305befcb-aafc-48ca-8469-49697d33d49e" containerName="payload-processing"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117688    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91d5bd0-8541-4a70-a6e7-06db5c88e2df" containerName="payload-pre-processing"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117705    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="305befcb-aafc-48ca-8469-49697d33d49e" containerName="payload-processing"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117719    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117730    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b14432b6-e3db-4a33-aaa8-1dd9c3e96305" containerName="istio-proxy"
Nov 03 01:48:47.120417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.117742    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55a9f34-b745-4306-83d9-5460d87832e5" containerName="istio-proxy"
Nov 03 01:48:47.172800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.146181    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.210750 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.210706    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"]
Nov 03 01:48:47.218666 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.213154    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-socket\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.224990 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.224949    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-token\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.225370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.225324    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-podinfo\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.225641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.225595    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-data\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.225770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.225745    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-certs\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.225924 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.225903    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-envoy\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.225989 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.225977    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istiod-ca-cert\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.226076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.226010    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-credential-socket\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.226076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.226053    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmbn\" (UniqueName: \"kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-kube-api-access-4hmbn\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.299387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.295575    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-36c2c1-openshift-default-dockercfg-ftrs6\""
Nov 03 01:48:47.331394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.330228    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-envoy\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.331394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.330311    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istiod-ca-cert\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.331394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.330362    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-credential-socket\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.331394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.330400    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmbn\" (UniqueName: \"kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-kube-api-access-4hmbn\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.331394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.330546    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-socket\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.331394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.330593    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-token\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.331394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.330634    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-podinfo\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.331394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.330684    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-data\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.331394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.330736    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-certs\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.334877 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.333404    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istiod-ca-cert\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.334877 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.333920    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-socket\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.334877 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.334568    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-credential-socket\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.334877 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.334822    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-data\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.335278 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.335232    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-certs\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.398566 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.398533    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmbn\" (UniqueName: \"kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-kube-api-access-4hmbn\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.407052 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.406064    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-podinfo\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.407958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.407705    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-token\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.489525 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.488369    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-envoy\") pod \"e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") " pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:47.668018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:47.667977    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:48.111888 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:48.111781    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"]
Nov 03 01:48:48.117327 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:48:48.117298    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37e873e_8f3e_4e1b_a409_ca3fe4192728.slice/crio-a967c5609eafbd680e87b7e8faa14d389d95f2a2227ddedbaa580c4007185d4a WatchSource:0}: Error finding container a967c5609eafbd680e87b7e8faa14d389d95f2a2227ddedbaa580c4007185d4a: Status 404 returned error can't find the container with id a967c5609eafbd680e87b7e8faa14d389d95f2a2227ddedbaa580c4007185d4a
Nov 03 01:48:48.120725 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:48.120681    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:48:48.121104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:48.120771    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:48:48.121104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:48.120818    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:48:48.631205 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:48.631153    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp" event={"ID":"a37e873e-8f3e-4e1b-a409-ca3fe4192728","Type":"ContainerStarted","Data":"14b47e8eb3d0fd8d995c16f1a6bf4e130df23edd792531c29e8639db07f07dd8"}
Nov 03 01:48:48.631462 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:48.631219    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp" event={"ID":"a37e873e-8f3e-4e1b-a409-ca3fe4192728","Type":"ContainerStarted","Data":"a967c5609eafbd680e87b7e8faa14d389d95f2a2227ddedbaa580c4007185d4a"}
Nov 03 01:48:48.669513 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:48.669106    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:48.699368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:48.699322    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305befcb-aafc-48ca-8469-49697d33d49e" path="/var/lib/kubelet/pods/305befcb-aafc-48ca-8469-49697d33d49e/volumes"
Nov 03 01:48:48.700141 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:48.700112    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91d5bd0-8541-4a70-a6e7-06db5c88e2df" path="/var/lib/kubelet/pods/f91d5bd0-8541-4a70-a6e7-06db5c88e2df/volumes"
Nov 03 01:48:48.842922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:48.842848    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp" podStartSLOduration=2.84282325 podStartE2EDuration="2.84282325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:48:48.791157481 +0000 UTC m=+1808.216763944" watchObservedRunningTime="2026-11-03 01:48:48.84282325 +0000 UTC m=+1808.268429714"
Nov 03 01:48:49.674336 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:49.674309    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:50.649029 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:50.649001    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:50.650140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:50.650116    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:48:50.928525 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:50.928492    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:48:51.012682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.012639    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-certs\") pod \"f930409c-7286-4a06-bb5f-93015858b97e\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") "
Nov 03 01:48:51.012851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.012735    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f930409c-7286-4a06-bb5f-93015858b97e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f930409c-7286-4a06-bb5f-93015858b97e-istiod-ca-cert\") pod \"f930409c-7286-4a06-bb5f-93015858b97e\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") "
Nov 03 01:48:51.012851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.012792    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-data\") pod \"f930409c-7286-4a06-bb5f-93015858b97e\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") "
Nov 03 01:48:51.012939 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.012904    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-socket\") pod \"f930409c-7286-4a06-bb5f-93015858b97e\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") "
Nov 03 01:48:51.012976 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.012959    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-envoy\") pod \"f930409c-7286-4a06-bb5f-93015858b97e\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") "
Nov 03 01:48:51.012976 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.012956    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-certs" pod "f930409c-7286-4a06-bb5f-93015858b97e" (UID: "f930409c-7286-4a06-bb5f-93015858b97e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:51.013063 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.012992    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-kube-api-access-v9tx9\" (UniqueName: \"kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-kube-api-access-v9tx9\") pod \"f930409c-7286-4a06-bb5f-93015858b97e\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") "
Nov 03 01:48:51.013063 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.013028    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-istio-token\" (UniqueName: \"kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-istio-token\") pod \"f930409c-7286-4a06-bb5f-93015858b97e\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") "
Nov 03 01:48:51.013165 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.013081    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-credential-socket\") pod \"f930409c-7286-4a06-bb5f-93015858b97e\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") "
Nov 03 01:48:51.013165 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.013143    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f930409c-7286-4a06-bb5f-93015858b97e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f930409c-7286-4a06-bb5f-93015858b97e-istio-podinfo\") pod \"f930409c-7286-4a06-bb5f-93015858b97e\" (UID: \"f930409c-7286-4a06-bb5f-93015858b97e\") "
Nov 03 01:48:51.013165 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.013148    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-socket" pod "f930409c-7286-4a06-bb5f-93015858b97e" (UID: "f930409c-7286-4a06-bb5f-93015858b97e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:51.013617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.013430    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-data" pod "f930409c-7286-4a06-bb5f-93015858b97e" (UID: "f930409c-7286-4a06-bb5f-93015858b97e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:51.013751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.013659    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f930409c-7286-4a06-bb5f-93015858b97e-istiod-ca-cert" pod "f930409c-7286-4a06-bb5f-93015858b97e" (UID: "f930409c-7286-4a06-bb5f-93015858b97e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:48:51.013920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.013848    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:51.013920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.013873    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:51.013920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.013878    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-credential-socket" pod "f930409c-7286-4a06-bb5f-93015858b97e" (UID: "f930409c-7286-4a06-bb5f-93015858b97e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:51.014119 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.013888    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:51.014119 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.013991    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f930409c-7286-4a06-bb5f-93015858b97e-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:51.016147 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.016076    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-envoy" pod "f930409c-7286-4a06-bb5f-93015858b97e" (UID: "f930409c-7286-4a06-bb5f-93015858b97e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:48:51.016245 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.016213    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-kube-api-access-v9tx9" pod "f930409c-7286-4a06-bb5f-93015858b97e" (UID: "f930409c-7286-4a06-bb5f-93015858b97e"). InnerVolumeSpecName "kube-api-access-v9tx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:48:51.016441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.016419    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f930409c-7286-4a06-bb5f-93015858b97e-istio-podinfo" pod "f930409c-7286-4a06-bb5f-93015858b97e" (UID: "f930409c-7286-4a06-bb5f-93015858b97e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:48:51.017010 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.016985    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-istio-token" pod "f930409c-7286-4a06-bb5f-93015858b97e" (UID: "f930409c-7286-4a06-bb5f-93015858b97e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:48:51.115156 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.115123    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:51.115156 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.115153    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9tx9\" (UniqueName: \"kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-kube-api-access-v9tx9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:51.115156 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.115164    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f930409c-7286-4a06-bb5f-93015858b97e-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:51.115511 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.115174    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f930409c-7286-4a06-bb5f-93015858b97e-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:51.115511 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.115184    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f930409c-7286-4a06-bb5f-93015858b97e-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:48:51.658683 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.658650    5508 generic.go:356] "Generic (PLEG): container finished" podID="f930409c-7286-4a06-bb5f-93015858b97e" containerID="b149004c5b6a469ab40d04d78926d4ccd813ff21eaed14513b2144d8393854c5" exitCode=0
Nov 03 01:48:51.658916 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.658712    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"
Nov 03 01:48:51.658916 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.658743    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b" event={"ID":"f930409c-7286-4a06-bb5f-93015858b97e","Type":"ContainerDied","Data":"b149004c5b6a469ab40d04d78926d4ccd813ff21eaed14513b2144d8393854c5"}
Nov 03 01:48:51.658916 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.658843    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b" event={"ID":"f930409c-7286-4a06-bb5f-93015858b97e","Type":"ContainerDied","Data":"12111cd2d7307a2c485b73749d3430049206da77d3d181164746b4c16d3e8b0f"}
Nov 03 01:48:51.658916 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.658879    5508 scope.go:122] "RemoveContainer" containerID="b149004c5b6a469ab40d04d78926d4ccd813ff21eaed14513b2144d8393854c5"
Nov 03 01:48:51.672824 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.672661    5508 scope.go:122] "RemoveContainer" containerID="482a99fc36dbee9778bb283835a122cf1dd20ef208186f85d2297de03c2df9de"
Nov 03 01:48:51.684406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.684389    5508 scope.go:122] "RemoveContainer" containerID="b149004c5b6a469ab40d04d78926d4ccd813ff21eaed14513b2144d8393854c5"
Nov 03 01:48:51.684778 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:48:51.684749    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b149004c5b6a469ab40d04d78926d4ccd813ff21eaed14513b2144d8393854c5\": container with ID starting with b149004c5b6a469ab40d04d78926d4ccd813ff21eaed14513b2144d8393854c5 not found: ID does not exist" containerID="b149004c5b6a469ab40d04d78926d4ccd813ff21eaed14513b2144d8393854c5"
Nov 03 01:48:51.684851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.684788    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b149004c5b6a469ab40d04d78926d4ccd813ff21eaed14513b2144d8393854c5"} err="failed to get container status \"b149004c5b6a469ab40d04d78926d4ccd813ff21eaed14513b2144d8393854c5\": rpc error: code = NotFound desc = could not find container \"b149004c5b6a469ab40d04d78926d4ccd813ff21eaed14513b2144d8393854c5\": container with ID starting with b149004c5b6a469ab40d04d78926d4ccd813ff21eaed14513b2144d8393854c5 not found: ID does not exist"
Nov 03 01:48:51.684851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.684810    5508 scope.go:122] "RemoveContainer" containerID="482a99fc36dbee9778bb283835a122cf1dd20ef208186f85d2297de03c2df9de"
Nov 03 01:48:51.685147 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:48:51.685126    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"482a99fc36dbee9778bb283835a122cf1dd20ef208186f85d2297de03c2df9de\": container with ID starting with 482a99fc36dbee9778bb283835a122cf1dd20ef208186f85d2297de03c2df9de not found: ID does not exist" containerID="482a99fc36dbee9778bb283835a122cf1dd20ef208186f85d2297de03c2df9de"
Nov 03 01:48:51.685236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.685158    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"482a99fc36dbee9778bb283835a122cf1dd20ef208186f85d2297de03c2df9de"} err="failed to get container status \"482a99fc36dbee9778bb283835a122cf1dd20ef208186f85d2297de03c2df9de\": rpc error: code = NotFound desc = could not find container \"482a99fc36dbee9778bb283835a122cf1dd20ef208186f85d2297de03c2df9de\": container with ID starting with 482a99fc36dbee9778bb283835a122cf1dd20ef208186f85d2297de03c2df9de not found: ID does not exist"
Nov 03 01:48:51.817313 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.817261    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"]
Nov 03 01:48:51.863634 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:51.863604    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-118857-openshift-default-7dd875bc6c-48v8b"]
Nov 03 01:48:52.668721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:52.668691    5508 generic.go:356] "Generic (PLEG): container finished" podID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerID="d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0" exitCode=0
Nov 03 01:48:52.669116 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:52.668757    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" event={"ID":"872fb22d-056c-4e0c-9555-e285d73cd87f","Type":"ContainerDied","Data":"d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0"}
Nov 03 01:48:52.669426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:52.669408    5508 scope.go:122] "RemoveContainer" containerID="d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0"
Nov 03 01:48:52.670871 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:52.670843    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:48:52.670971 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:52.670910    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:48:52.670971 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:52.670944    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:48:52.688129 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:48:52.684072    5508 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872fb22d_056c_4e0c_9555_e285d73cd87f.slice/crio-conmon-d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:48:52.694825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:52.694805    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f930409c-7286-4a06-bb5f-93015858b97e" path="/var/lib/kubelet/pods/f930409c-7286-4a06-bb5f-93015858b97e/volumes"
Nov 03 01:48:53.182374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:53.182329    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 01:48:53.182582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:53.182410    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 01:48:53.681847 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:53.681809    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" event={"ID":"872fb22d-056c-4e0c-9555-e285d73cd87f","Type":"ContainerStarted","Data":"fb4544c9d3253a50e2a6dd4bacc035ab07e94b48881884a3c7d06d134edc449c"}
Nov 03 01:48:53.683713 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:53.683681    5508 generic.go:356] "Generic (PLEG): container finished" podID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerID="50fba1e546916d1e949d2ddd2234df40bb647848220f5e96ea1aefb0f2069702" exitCode=0
Nov 03 01:48:53.683845 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:53.683752    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" event={"ID":"774d3ce9-baeb-4de2-837b-62b8a559b7b3","Type":"ContainerDied","Data":"50fba1e546916d1e949d2ddd2234df40bb647848220f5e96ea1aefb0f2069702"}
Nov 03 01:48:53.684137 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:53.684123    5508 scope.go:122] "RemoveContainer" containerID="50fba1e546916d1e949d2ddd2234df40bb647848220f5e96ea1aefb0f2069702"
Nov 03 01:48:53.685614 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:53.685585    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:48:53.685715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:53.685670    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:48:53.685715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:53.685703    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:48:54.086428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:54.074663    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:48:54.086428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:54.086143    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 03 01:48:54.086428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:54.086190    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 03 01:48:54.715226 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:54.715185    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" event={"ID":"774d3ce9-baeb-4de2-837b-62b8a559b7b3","Type":"ContainerStarted","Data":"1f6c9691160fa88ac5720164ad35dfb882aeb2027d265a19153b53c030563bfc"}
Nov 03 01:48:55.114425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.105531    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 03 01:48:55.114425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.105588    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:48:55.114425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.105588    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 03 01:48:55.114425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.105685    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 01:48:55.114425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.105731    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 01:48:55.280445 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.280407    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"]
Nov 03 01:48:55.295371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.281560    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f930409c-7286-4a06-bb5f-93015858b97e" containerName="istio-proxy"
Nov 03 01:48:55.318864 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.318825    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f930409c-7286-4a06-bb5f-93015858b97e" containerName="istio-proxy"
Nov 03 01:48:55.331824 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.331796    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f930409c-7286-4a06-bb5f-93015858b97e" containerName="istio-proxy"
Nov 03 01:48:55.639370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.609398    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"]
Nov 03 01:48:55.640323 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.640297    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:48:55.640546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.640532    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"]
Nov 03 01:48:55.642566 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.634639    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"
Nov 03 01:48:55.666386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.662867    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f930409c-7286-4a06-bb5f-93015858b97e" containerName="istio-proxy"
Nov 03 01:48:55.666386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.662892    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f930409c-7286-4a06-bb5f-93015858b97e" containerName="istio-proxy"
Nov 03 01:48:55.666386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.663565    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f930409c-7286-4a06-bb5f-93015858b97e" containerName="istio-proxy"
Nov 03 01:48:55.689459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.683407    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l\" (UID: \"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"
Nov 03 01:48:55.689459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.683616    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt9bp\" (UniqueName: \"kubernetes.io/projected/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-kube-api-access-gt9bp\") pod \"payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l\" (UID: \"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"
Nov 03 01:48:55.693370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.691151    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"]
Nov 03 01:48:55.693370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.691292    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"
Nov 03 01:48:55.778620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.778589    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-36c2c1\""
Nov 03 01:48:55.779214 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.779188    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-36c2c1-dockercfg-bpxtv\""
Nov 03 01:48:55.811371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.811068    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ab21b5-0596-46ff-9309-47373a0094ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm\" (UID: \"28ab21b5-0596-46ff-9309-47373a0094ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"
Nov 03 01:48:55.811371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.811152    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blvqv\" (UniqueName: \"kubernetes.io/projected/28ab21b5-0596-46ff-9309-47373a0094ca-kube-api-access-blvqv\") pod \"payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm\" (UID: \"28ab21b5-0596-46ff-9309-47373a0094ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"
Nov 03 01:48:55.811609 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.811411    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l\" (UID: \"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"
Nov 03 01:48:55.811731 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.811710    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt9bp\" (UniqueName: \"kubernetes.io/projected/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-kube-api-access-gt9bp\") pod \"payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l\" (UID: \"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"
Nov 03 01:48:55.813432 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.813106    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l\" (UID: \"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"
Nov 03 01:48:55.879283 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.879202    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt9bp\" (UniqueName: \"kubernetes.io/projected/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-kube-api-access-gt9bp\") pod \"payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l\" (UID: \"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"
Nov 03 01:48:55.915401 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.915366    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blvqv\" (UniqueName: \"kubernetes.io/projected/28ab21b5-0596-46ff-9309-47373a0094ca-kube-api-access-blvqv\") pod \"payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm\" (UID: \"28ab21b5-0596-46ff-9309-47373a0094ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"
Nov 03 01:48:55.915954 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.915928    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ab21b5-0596-46ff-9309-47373a0094ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm\" (UID: \"28ab21b5-0596-46ff-9309-47373a0094ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"
Nov 03 01:48:55.917014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.916994    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ab21b5-0596-46ff-9309-47373a0094ca-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm\" (UID: \"28ab21b5-0596-46ff-9309-47373a0094ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"
Nov 03 01:48:55.995221 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.995183    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blvqv\" (UniqueName: \"kubernetes.io/projected/28ab21b5-0596-46ff-9309-47373a0094ca-kube-api-access-blvqv\") pod \"payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm\" (UID: \"28ab21b5-0596-46ff-9309-47373a0094ca\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"
Nov 03 01:48:55.995587 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.995547    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 01:48:55.995785 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:55.995751    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 01:48:56.040635 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.040613    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"
Nov 03 01:48:56.046740 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.046721    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"
Nov 03 01:48:56.090130 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.090100    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 03 01:48:56.090389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.090359    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 03 01:48:56.539365 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.539280    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"]
Nov 03 01:48:56.602380 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:48:56.598654    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ab21b5_0596_46ff_9309_47373a0094ca.slice/crio-9416381b1e79389e2e1a6f930ed8ce2c165faea06da089a2045430181f2e1c07 WatchSource:0}: Error finding container 9416381b1e79389e2e1a6f930ed8ce2c165faea06da089a2045430181f2e1c07: Status 404 returned error can't find the container with id 9416381b1e79389e2e1a6f930ed8ce2c165faea06da089a2045430181f2e1c07
Nov 03 01:48:56.605988 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.605966    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"]
Nov 03 01:48:56.609475 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:48:56.609450    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1b6c1a_1b87_4b0c_b8dc_2510f4522c92.slice/crio-08d7e47b33a23fe06ca64f6fc80de0bf3b97bdaf0a116b10a6dbbd7831feec1d WatchSource:0}: Error finding container 08d7e47b33a23fe06ca64f6fc80de0bf3b97bdaf0a116b10a6dbbd7831feec1d: Status 404 returned error can't find the container with id 08d7e47b33a23fe06ca64f6fc80de0bf3b97bdaf0a116b10a6dbbd7831feec1d
Nov 03 01:48:56.807441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.799883    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm" event={"ID":"28ab21b5-0596-46ff-9309-47373a0094ca","Type":"ContainerStarted","Data":"7dc60be6b553ffac7e2b5e99c1617ab49abfefae6ce8c0c9f56d8f0e21f983c1"}
Nov 03 01:48:56.807441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.799963    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm" event={"ID":"28ab21b5-0596-46ff-9309-47373a0094ca","Type":"ContainerStarted","Data":"9416381b1e79389e2e1a6f930ed8ce2c165faea06da089a2045430181f2e1c07"}
Nov 03 01:48:56.807441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.800972    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"
Nov 03 01:48:56.812453 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.812426    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l" event={"ID":"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92","Type":"ContainerStarted","Data":"7ba8b72a023ab4b9b2bcd883cc69d23434d259004a598ca6a1ff7c514e729879"}
Nov 03 01:48:56.812567 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.812477    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l" event={"ID":"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92","Type":"ContainerStarted","Data":"08d7e47b33a23fe06ca64f6fc80de0bf3b97bdaf0a116b10a6dbbd7831feec1d"}
Nov 03 01:48:56.812567 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.812514    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"
Nov 03 01:48:56.940467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.940416    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm" podStartSLOduration=1.94040119 podStartE2EDuration="1.94040119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:48:56.938748902 +0000 UTC m=+1816.364355388" watchObservedRunningTime="2026-11-03 01:48:56.94040119 +0000 UTC m=+1816.366007657"
Nov 03 01:48:56.998700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.998662    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 01:48:56.998909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:56.998720    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 01:48:57.076300 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:57.076212    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 03 01:48:57.076619 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:57.076573    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 03 01:48:57.997374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:57.996670    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 01:48:57.997374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:57.996727    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 01:48:58.095373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:58.088757    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 03 01:48:58.095373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:58.088817    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 03 01:48:58.993372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:58.992868    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 01:48:58.993372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:58.992930    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 01:48:59.080375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:59.079475    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Nov 03 01:48:59.080375 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:48:59.079537    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Nov 03 01:49:00.001387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:00.001248    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 03 01:49:00.001387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:00.001326    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 03 01:49:00.116372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:00.109679    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:49:00.127370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:00.123610    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:49:00.280710 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:00.280607    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l" podStartSLOduration=5.280591031 podStartE2EDuration="5.280591031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:48:57.075795141 +0000 UTC m=+1816.501401606" watchObservedRunningTime="2026-11-03 01:49:00.280591031 +0000 UTC m=+1819.706197498"
Nov 03 01:49:01.029371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:01.024186    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:49:01.029371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:01.025193    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:49:01.029371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:01.028229    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:49:07.821782 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:07.821755    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"
Nov 03 01:49:08.828277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:08.828249    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"
Nov 03 01:49:09.489180 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:09.489146    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"]
Nov 03 01:49:09.489522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:09.489495    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy" containerID="cri-o://601e7028cc51463414bbf875d5369d43cb482e96064708209e3a3d73c73345ab" gracePeriod=30
Nov 03 01:49:09.505863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:09.505836    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"]
Nov 03 01:49:09.506018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:09.506001    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n" podUID="fd7c9be9-74ca-43af-a8f2-0667cf1d5346" containerName="istio-proxy" containerID="cri-o://fe83c79f52a5c2d5ce3abe9a14de80419db3d4cc44bcf0a8bf74654070dc2418" gracePeriod=30
Nov 03 01:49:09.527633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:09.527612    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"]
Nov 03 01:49:09.527808 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:09.527786    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s" podUID="48a8b440-97a3-47a1-83b0-fd5627a39f8d" containerName="istio-proxy" containerID="cri-o://e45ace429a16f5cf20057de9961636dd6486cb83c6002021228e2b652a57ca2e" gracePeriod=30
Nov 03 01:49:09.532654 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:09.532630    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"]
Nov 03 01:49:09.532875 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:09.532852    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb" podUID="d27e6feb-5948-4fa9-b295-b73779b7291f" containerName="istio-proxy" containerID="cri-o://9a8478cd73f9536a8e34201562aac9b1e41d24a4d34de5b32249cf91533e2174" gracePeriod=30
Nov 03 01:49:09.824809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:09.824737    5508 patch_prober.go:29] interesting pod/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:49:09.824809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:09.824787    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:49:12.454260 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:12.454223    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"]
Nov 03 01:49:12.557907 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:12.557854    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"]
Nov 03 01:49:13.812154 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:13.812118    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:49:14.843449 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.843422    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:49:14.903435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.903411    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:49:14.906469 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.906448    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:49:14.909990 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.909968    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:49:14.955638 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.955614    5508 generic.go:356] "Generic (PLEG): container finished" podID="48a8b440-97a3-47a1-83b0-fd5627a39f8d" containerID="e45ace429a16f5cf20057de9961636dd6486cb83c6002021228e2b652a57ca2e" exitCode=0
Nov 03 01:49:14.955740 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.955656    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"
Nov 03 01:49:14.955740 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.955693    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s" event={"ID":"48a8b440-97a3-47a1-83b0-fd5627a39f8d","Type":"ContainerDied","Data":"e45ace429a16f5cf20057de9961636dd6486cb83c6002021228e2b652a57ca2e"}
Nov 03 01:49:14.955823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.955749    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s" event={"ID":"48a8b440-97a3-47a1-83b0-fd5627a39f8d","Type":"ContainerDied","Data":"2d9b2b5d9024f03a2845d03116eacb4ee4e76fe15e3280191933badd358dba87"}
Nov 03 01:49:14.955823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.955772    5508 scope.go:122] "RemoveContainer" containerID="e45ace429a16f5cf20057de9961636dd6486cb83c6002021228e2b652a57ca2e"
Nov 03 01:49:14.957212 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.957192    5508 generic.go:356] "Generic (PLEG): container finished" podID="d27e6feb-5948-4fa9-b295-b73779b7291f" containerID="9a8478cd73f9536a8e34201562aac9b1e41d24a4d34de5b32249cf91533e2174" exitCode=0
Nov 03 01:49:14.957309 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.957246    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"
Nov 03 01:49:14.957309 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.957285    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb" event={"ID":"d27e6feb-5948-4fa9-b295-b73779b7291f","Type":"ContainerDied","Data":"9a8478cd73f9536a8e34201562aac9b1e41d24a4d34de5b32249cf91533e2174"}
Nov 03 01:49:14.957429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.957335    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb" event={"ID":"d27e6feb-5948-4fa9-b295-b73779b7291f","Type":"ContainerDied","Data":"5368a612f4798abd8b4b5d9787abfbb29731f84de785281cf107409e5c02bbe2"}
Nov 03 01:49:14.959140 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.959118    5508 generic.go:356] "Generic (PLEG): container finished" podID="08801e9f-1be9-4deb-931a-508623c37c7a" containerID="601e7028cc51463414bbf875d5369d43cb482e96064708209e3a3d73c73345ab" exitCode=0
Nov 03 01:49:14.959220 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.959163    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"
Nov 03 01:49:14.959220 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.959179    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" event={"ID":"08801e9f-1be9-4deb-931a-508623c37c7a","Type":"ContainerDied","Data":"601e7028cc51463414bbf875d5369d43cb482e96064708209e3a3d73c73345ab"}
Nov 03 01:49:14.959304 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.959222    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj" event={"ID":"08801e9f-1be9-4deb-931a-508623c37c7a","Type":"ContainerDied","Data":"8d5ff2d7f3bdaad54811d0254ae66e6391452d9fb14d135d566ca6a6ad29738f"}
Nov 03 01:49:14.960587 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.960568    5508 generic.go:356] "Generic (PLEG): container finished" podID="fd7c9be9-74ca-43af-a8f2-0667cf1d5346" containerID="fe83c79f52a5c2d5ce3abe9a14de80419db3d4cc44bcf0a8bf74654070dc2418" exitCode=0
Nov 03 01:49:14.960685 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.960605    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"
Nov 03 01:49:14.960685 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.960612    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n" event={"ID":"fd7c9be9-74ca-43af-a8f2-0667cf1d5346","Type":"ContainerDied","Data":"fe83c79f52a5c2d5ce3abe9a14de80419db3d4cc44bcf0a8bf74654070dc2418"}
Nov 03 01:49:14.960685 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.960656    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n" event={"ID":"fd7c9be9-74ca-43af-a8f2-0667cf1d5346","Type":"ContainerDied","Data":"36bcb8fa31412b07e10b2b13548c9fdd99a0e704ac551d9a71cfee43c7f19623"}
Nov 03 01:49:14.966266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.966249    5508 scope.go:122] "RemoveContainer" containerID="e45ace429a16f5cf20057de9961636dd6486cb83c6002021228e2b652a57ca2e"
Nov 03 01:49:14.966544 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:14.966516    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45ace429a16f5cf20057de9961636dd6486cb83c6002021228e2b652a57ca2e\": container with ID starting with e45ace429a16f5cf20057de9961636dd6486cb83c6002021228e2b652a57ca2e not found: ID does not exist" containerID="e45ace429a16f5cf20057de9961636dd6486cb83c6002021228e2b652a57ca2e"
Nov 03 01:49:14.966642 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.966541    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45ace429a16f5cf20057de9961636dd6486cb83c6002021228e2b652a57ca2e"} err="failed to get container status \"e45ace429a16f5cf20057de9961636dd6486cb83c6002021228e2b652a57ca2e\": rpc error: code = NotFound desc = could not find container \"e45ace429a16f5cf20057de9961636dd6486cb83c6002021228e2b652a57ca2e\": container with ID starting with e45ace429a16f5cf20057de9961636dd6486cb83c6002021228e2b652a57ca2e not found: ID does not exist"
Nov 03 01:49:14.966642 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.966557    5508 scope.go:122] "RemoveContainer" containerID="9a8478cd73f9536a8e34201562aac9b1e41d24a4d34de5b32249cf91533e2174"
Nov 03 01:49:14.976308 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.976280    5508 scope.go:122] "RemoveContainer" containerID="9a8478cd73f9536a8e34201562aac9b1e41d24a4d34de5b32249cf91533e2174"
Nov 03 01:49:14.976637 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:14.976608    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a8478cd73f9536a8e34201562aac9b1e41d24a4d34de5b32249cf91533e2174\": container with ID starting with 9a8478cd73f9536a8e34201562aac9b1e41d24a4d34de5b32249cf91533e2174 not found: ID does not exist" containerID="9a8478cd73f9536a8e34201562aac9b1e41d24a4d34de5b32249cf91533e2174"
Nov 03 01:49:14.976835 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.976643    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8478cd73f9536a8e34201562aac9b1e41d24a4d34de5b32249cf91533e2174"} err="failed to get container status \"9a8478cd73f9536a8e34201562aac9b1e41d24a4d34de5b32249cf91533e2174\": rpc error: code = NotFound desc = could not find container \"9a8478cd73f9536a8e34201562aac9b1e41d24a4d34de5b32249cf91533e2174\": container with ID starting with 9a8478cd73f9536a8e34201562aac9b1e41d24a4d34de5b32249cf91533e2174 not found: ID does not exist"
Nov 03 01:49:14.976835 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.976666    5508 scope.go:122] "RemoveContainer" containerID="601e7028cc51463414bbf875d5369d43cb482e96064708209e3a3d73c73345ab"
Nov 03 01:49:14.989125 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.989105    5508 scope.go:122] "RemoveContainer" containerID="e84add4aad2b8a95c39f4b1cdc4d9675e88f46d5097b10416da0cc6a4361aa88"
Nov 03 01:49:14.992469 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.992452    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-token\" (UniqueName: \"kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-token\") pod \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") "
Nov 03 01:49:14.992534 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.992489    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-envoy\") pod \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") "
Nov 03 01:49:14.992534 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.992516    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-token\" (UniqueName: \"kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-token\") pod \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") "
Nov 03 01:49:14.992671 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.992653    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-data\") pod \"08801e9f-1be9-4deb-931a-508623c37c7a\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") "
Nov 03 01:49:14.992734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.992720    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-certs\") pod \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") "
Nov 03 01:49:14.992794 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.992756    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-kube-api-access-lqk99\" (UniqueName: \"kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-kube-api-access-lqk99\") pod \"08801e9f-1be9-4deb-931a-508623c37c7a\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") "
Nov 03 01:49:14.992851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.992789    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-credential-socket\") pod \"d27e6feb-5948-4fa9-b295-b73779b7291f\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") "
Nov 03 01:49:14.992851 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.992824    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-data\") pod \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") "
Nov 03 01:49:14.992953 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.992867    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-certs\") pod \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") "
Nov 03 01:49:14.992953 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.992915    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-data\") pod \"d27e6feb-5948-4fa9-b295-b73779b7291f\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") "
Nov 03 01:49:14.992953 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.992934    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-data" pod "08801e9f-1be9-4deb-931a-508623c37c7a" (UID: "08801e9f-1be9-4deb-931a-508623c37c7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.993100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.992964    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-envoy\") pod \"08801e9f-1be9-4deb-931a-508623c37c7a\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") "
Nov 03 01:49:14.993100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.993012    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-kube-api-access-6wlps\" (UniqueName: \"kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-kube-api-access-6wlps\") pod \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") "
Nov 03 01:49:14.993100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.993012    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-certs" pod "48a8b440-97a3-47a1-83b0-fd5627a39f8d" (UID: "48a8b440-97a3-47a1-83b0-fd5627a39f8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.993100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.993044    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-socket\") pod \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") "
Nov 03 01:49:14.993100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.993084    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-podinfo\") pod \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") "
Nov 03 01:49:14.993406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.993116    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08801e9f-1be9-4deb-931a-508623c37c7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08801e9f-1be9-4deb-931a-508623c37c7a-istiod-ca-cert\") pod \"08801e9f-1be9-4deb-931a-508623c37c7a\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") "
Nov 03 01:49:14.993406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.993148    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-data\") pod \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") "
Nov 03 01:49:14.993406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.993143    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-data" pod "48a8b440-97a3-47a1-83b0-fd5627a39f8d" (UID: "48a8b440-97a3-47a1-83b0-fd5627a39f8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.993574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.993457    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-data" pod "fd7c9be9-74ca-43af-a8f2-0667cf1d5346" (UID: "fd7c9be9-74ca-43af-a8f2-0667cf1d5346"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.993574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.993504    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-credential-socket" pod "d27e6feb-5948-4fa9-b295-b73779b7291f" (UID: "d27e6feb-5948-4fa9-b295-b73779b7291f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.994494 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.994273    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-data" pod "d27e6feb-5948-4fa9-b295-b73779b7291f" (UID: "d27e6feb-5948-4fa9-b295-b73779b7291f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.995075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.994735    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08801e9f-1be9-4deb-931a-508623c37c7a-istiod-ca-cert" pod "08801e9f-1be9-4deb-931a-508623c37c7a" (UID: "08801e9f-1be9-4deb-931a-508623c37c7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:49:14.995075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.994925    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-envoy" pod "48a8b440-97a3-47a1-83b0-fd5627a39f8d" (UID: "48a8b440-97a3-47a1-83b0-fd5627a39f8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.995075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995043    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-certs" pod "fd7c9be9-74ca-43af-a8f2-0667cf1d5346" (UID: "fd7c9be9-74ca-43af-a8f2-0667cf1d5346"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.995291 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995092    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-envoy\") pod \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") "
Nov 03 01:49:14.995291 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995127    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-token" pod "fd7c9be9-74ca-43af-a8f2-0667cf1d5346" (UID: "fd7c9be9-74ca-43af-a8f2-0667cf1d5346"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:14.995418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995374    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-certs" pod "d27e6feb-5948-4fa9-b295-b73779b7291f" (UID: "d27e6feb-5948-4fa9-b295-b73779b7291f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.995478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995457    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-socket" pod "fd7c9be9-74ca-43af-a8f2-0667cf1d5346" (UID: "fd7c9be9-74ca-43af-a8f2-0667cf1d5346"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.995478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995128    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-certs\") pod \"d27e6feb-5948-4fa9-b295-b73779b7291f\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") "
Nov 03 01:49:14.995584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995480    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-kube-api-access-lqk99" pod "08801e9f-1be9-4deb-931a-508623c37c7a" (UID: "08801e9f-1be9-4deb-931a-508623c37c7a"). InnerVolumeSpecName "kube-api-access-lqk99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:14.995584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995532    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08801e9f-1be9-4deb-931a-508623c37c7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08801e9f-1be9-4deb-931a-508623c37c7a-istio-podinfo\") pod \"08801e9f-1be9-4deb-931a-508623c37c7a\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") "
Nov 03 01:49:14.995584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995580    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istiod-ca-cert\") pod \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") "
Nov 03 01:49:14.995745 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995607    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-podinfo\") pod \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") "
Nov 03 01:49:14.995745 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995651    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-kube-api-access-w4wln\" (UniqueName: \"kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-kube-api-access-w4wln\") pod \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") "
Nov 03 01:49:14.995745 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995692    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-credential-socket\") pod \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") "
Nov 03 01:49:14.995745 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995723    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-socket\") pod \"d27e6feb-5948-4fa9-b295-b73779b7291f\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") "
Nov 03 01:49:14.995973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995756    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-istio-token\" (UniqueName: \"kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-istio-token\") pod \"08801e9f-1be9-4deb-931a-508623c37c7a\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") "
Nov 03 01:49:14.995973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995793    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-kube-api-access-kwcjz\" (UniqueName: \"kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-kube-api-access-kwcjz\") pod \"d27e6feb-5948-4fa9-b295-b73779b7291f\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") "
Nov 03 01:49:14.995973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995843    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-envoy\") pod \"d27e6feb-5948-4fa9-b295-b73779b7291f\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") "
Nov 03 01:49:14.995973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995876    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istiod-ca-cert\") pod \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") "
Nov 03 01:49:14.995973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995917    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-credential-socket\") pod \"08801e9f-1be9-4deb-931a-508623c37c7a\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") "
Nov 03 01:49:14.995973 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995949    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-socket\") pod \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\" (UID: \"48a8b440-97a3-47a1-83b0-fd5627a39f8d\") "
Nov 03 01:49:14.996266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.995987    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-socket\") pod \"08801e9f-1be9-4deb-931a-508623c37c7a\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") "
Nov 03 01:49:14.996266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.996046    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-podinfo\") pod \"d27e6feb-5948-4fa9-b295-b73779b7291f\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") "
Nov 03 01:49:14.996266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.996079    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-token\" (UniqueName: \"kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-token\") pod \"d27e6feb-5948-4fa9-b295-b73779b7291f\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") "
Nov 03 01:49:14.996266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.996122    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-certs\") pod \"08801e9f-1be9-4deb-931a-508623c37c7a\" (UID: \"08801e9f-1be9-4deb-931a-508623c37c7a\") "
Nov 03 01:49:14.996266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.996152    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-credential-socket\") pod \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\" (UID: \"fd7c9be9-74ca-43af-a8f2-0667cf1d5346\") "
Nov 03 01:49:14.996266 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.996195    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27e6feb-5948-4fa9-b295-b73779b7291f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27e6feb-5948-4fa9-b295-b73779b7291f-istiod-ca-cert\") pod \"d27e6feb-5948-4fa9-b295-b73779b7291f\" (UID: \"d27e6feb-5948-4fa9-b295-b73779b7291f\") "
Nov 03 01:49:14.996751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.996723    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-token" pod "48a8b440-97a3-47a1-83b0-fd5627a39f8d" (UID: "48a8b440-97a3-47a1-83b0-fd5627a39f8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997017    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997042    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08801e9f-1be9-4deb-931a-508623c37c7a-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997063    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997080    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997098    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997113    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997126    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997140    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997153    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997166    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqk99\" (UniqueName: \"kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-kube-api-access-lqk99\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997181    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997194    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997208    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997223    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997095    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27e6feb-5948-4fa9-b295-b73779b7291f-istiod-ca-cert" pod "d27e6feb-5948-4fa9-b295-b73779b7291f" (UID: "d27e6feb-5948-4fa9-b295-b73779b7291f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997201    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-socket" pod "08801e9f-1be9-4deb-931a-508623c37c7a" (UID: "08801e9f-1be9-4deb-931a-508623c37c7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997416    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-kube-api-access-6wlps" pod "48a8b440-97a3-47a1-83b0-fd5627a39f8d" (UID: "48a8b440-97a3-47a1-83b0-fd5627a39f8d"). InnerVolumeSpecName "kube-api-access-6wlps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997425    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-credential-socket" pod "fd7c9be9-74ca-43af-a8f2-0667cf1d5346" (UID: "fd7c9be9-74ca-43af-a8f2-0667cf1d5346"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.997503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997437    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-certs" pod "08801e9f-1be9-4deb-931a-508623c37c7a" (UID: "08801e9f-1be9-4deb-931a-508623c37c7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.998772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997636    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-credential-socket" pod "08801e9f-1be9-4deb-931a-508623c37c7a" (UID: "08801e9f-1be9-4deb-931a-508623c37c7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.998772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997662    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istiod-ca-cert" pod "fd7c9be9-74ca-43af-a8f2-0667cf1d5346" (UID: "fd7c9be9-74ca-43af-a8f2-0667cf1d5346"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:49:14.998772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997682    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-socket" pod "48a8b440-97a3-47a1-83b0-fd5627a39f8d" (UID: "48a8b440-97a3-47a1-83b0-fd5627a39f8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.998772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997828    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-credential-socket" pod "48a8b440-97a3-47a1-83b0-fd5627a39f8d" (UID: "48a8b440-97a3-47a1-83b0-fd5627a39f8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.998772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.997978    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istiod-ca-cert" pod "48a8b440-97a3-47a1-83b0-fd5627a39f8d" (UID: "48a8b440-97a3-47a1-83b0-fd5627a39f8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:49:14.998772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.998264    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-socket" pod "d27e6feb-5948-4fa9-b295-b73779b7291f" (UID: "d27e6feb-5948-4fa9-b295-b73779b7291f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.998772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.998593    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08801e9f-1be9-4deb-931a-508623c37c7a-istio-podinfo" pod "08801e9f-1be9-4deb-931a-508623c37c7a" (UID: "08801e9f-1be9-4deb-931a-508623c37c7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:49:14.999145 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.998912    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-envoy" pod "fd7c9be9-74ca-43af-a8f2-0667cf1d5346" (UID: "fd7c9be9-74ca-43af-a8f2-0667cf1d5346"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.999435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.999403    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-envoy" pod "08801e9f-1be9-4deb-931a-508623c37c7a" (UID: "08801e9f-1be9-4deb-931a-508623c37c7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:14.999435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.999424    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-envoy" pod "d27e6feb-5948-4fa9-b295-b73779b7291f" (UID: "d27e6feb-5948-4fa9-b295-b73779b7291f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:15.000017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:14.999995    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-podinfo" pod "48a8b440-97a3-47a1-83b0-fd5627a39f8d" (UID: "48a8b440-97a3-47a1-83b0-fd5627a39f8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:49:15.002801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.002782    5508 scope.go:122] "RemoveContainer" containerID="601e7028cc51463414bbf875d5369d43cb482e96064708209e3a3d73c73345ab"
Nov 03 01:49:15.002979 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.002958    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-podinfo" pod "d27e6feb-5948-4fa9-b295-b73779b7291f" (UID: "d27e6feb-5948-4fa9-b295-b73779b7291f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:49:15.003169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.002961    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-istio-token" pod "08801e9f-1be9-4deb-931a-508623c37c7a" (UID: "08801e9f-1be9-4deb-931a-508623c37c7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:15.003169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.003005    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-kube-api-access-w4wln" pod "fd7c9be9-74ca-43af-a8f2-0667cf1d5346" (UID: "fd7c9be9-74ca-43af-a8f2-0667cf1d5346"). InnerVolumeSpecName "kube-api-access-w4wln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:15.003169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.003018    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-token" pod "d27e6feb-5948-4fa9-b295-b73779b7291f" (UID: "d27e6feb-5948-4fa9-b295-b73779b7291f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:15.003169 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:15.003059    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601e7028cc51463414bbf875d5369d43cb482e96064708209e3a3d73c73345ab\": container with ID starting with 601e7028cc51463414bbf875d5369d43cb482e96064708209e3a3d73c73345ab not found: ID does not exist" containerID="601e7028cc51463414bbf875d5369d43cb482e96064708209e3a3d73c73345ab"
Nov 03 01:49:15.003169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.003087    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601e7028cc51463414bbf875d5369d43cb482e96064708209e3a3d73c73345ab"} err="failed to get container status \"601e7028cc51463414bbf875d5369d43cb482e96064708209e3a3d73c73345ab\": rpc error: code = NotFound desc = could not find container \"601e7028cc51463414bbf875d5369d43cb482e96064708209e3a3d73c73345ab\": container with ID starting with 601e7028cc51463414bbf875d5369d43cb482e96064708209e3a3d73c73345ab not found: ID does not exist"
Nov 03 01:49:15.003169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.003107    5508 scope.go:122] "RemoveContainer" containerID="e84add4aad2b8a95c39f4b1cdc4d9675e88f46d5097b10416da0cc6a4361aa88"
Nov 03 01:49:15.003717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.003489    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-podinfo" pod "fd7c9be9-74ca-43af-a8f2-0667cf1d5346" (UID: "fd7c9be9-74ca-43af-a8f2-0667cf1d5346"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:49:15.003717 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:15.003545    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e84add4aad2b8a95c39f4b1cdc4d9675e88f46d5097b10416da0cc6a4361aa88\": container with ID starting with e84add4aad2b8a95c39f4b1cdc4d9675e88f46d5097b10416da0cc6a4361aa88 not found: ID does not exist" containerID="e84add4aad2b8a95c39f4b1cdc4d9675e88f46d5097b10416da0cc6a4361aa88"
Nov 03 01:49:15.003717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.003575    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e84add4aad2b8a95c39f4b1cdc4d9675e88f46d5097b10416da0cc6a4361aa88"} err="failed to get container status \"e84add4aad2b8a95c39f4b1cdc4d9675e88f46d5097b10416da0cc6a4361aa88\": rpc error: code = NotFound desc = could not find container \"e84add4aad2b8a95c39f4b1cdc4d9675e88f46d5097b10416da0cc6a4361aa88\": container with ID starting with e84add4aad2b8a95c39f4b1cdc4d9675e88f46d5097b10416da0cc6a4361aa88 not found: ID does not exist"
Nov 03 01:49:15.003717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.003597    5508 scope.go:122] "RemoveContainer" containerID="fe83c79f52a5c2d5ce3abe9a14de80419db3d4cc44bcf0a8bf74654070dc2418"
Nov 03 01:49:15.004533 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.004459    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-kube-api-access-kwcjz" pod "d27e6feb-5948-4fa9-b295-b73779b7291f" (UID: "d27e6feb-5948-4fa9-b295-b73779b7291f"). InnerVolumeSpecName "kube-api-access-kwcjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:15.014109 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.014087    5508 scope.go:122] "RemoveContainer" containerID="fe83c79f52a5c2d5ce3abe9a14de80419db3d4cc44bcf0a8bf74654070dc2418"
Nov 03 01:49:15.014433 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:15.014411    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe83c79f52a5c2d5ce3abe9a14de80419db3d4cc44bcf0a8bf74654070dc2418\": container with ID starting with fe83c79f52a5c2d5ce3abe9a14de80419db3d4cc44bcf0a8bf74654070dc2418 not found: ID does not exist" containerID="fe83c79f52a5c2d5ce3abe9a14de80419db3d4cc44bcf0a8bf74654070dc2418"
Nov 03 01:49:15.014505 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.014439    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe83c79f52a5c2d5ce3abe9a14de80419db3d4cc44bcf0a8bf74654070dc2418"} err="failed to get container status \"fe83c79f52a5c2d5ce3abe9a14de80419db3d4cc44bcf0a8bf74654070dc2418\": rpc error: code = NotFound desc = could not find container \"fe83c79f52a5c2d5ce3abe9a14de80419db3d4cc44bcf0a8bf74654070dc2418\": container with ID starting with fe83c79f52a5c2d5ce3abe9a14de80419db3d4cc44bcf0a8bf74654070dc2418 not found: ID does not exist"
Nov 03 01:49:15.098235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098159    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwcjz\" (UniqueName: \"kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-kube-api-access-kwcjz\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098186    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098196    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098205    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098214    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098227    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098242    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098253    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d27e6feb-5948-4fa9-b295-b73779b7291f-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098261    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098269    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098277    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d27e6feb-5948-4fa9-b295-b73779b7291f-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098285    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08801e9f-1be9-4deb-931a-508623c37c7a-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098295    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wlps\" (UniqueName: \"kubernetes.io/projected/48a8b440-97a3-47a1-83b0-fd5627a39f8d-kube-api-access-6wlps\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098303    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098316    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098329    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08801e9f-1be9-4deb-931a-508623c37c7a-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098337    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a8b440-97a3-47a1-83b0-fd5627a39f8d-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098357    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098367    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4wln\" (UniqueName: \"kubernetes.io/projected/fd7c9be9-74ca-43af-a8f2-0667cf1d5346-kube-api-access-w4wln\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098376    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a8b440-97a3-47a1-83b0-fd5627a39f8d-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098384    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d27e6feb-5948-4fa9-b295-b73779b7291f-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.098558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.098400    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08801e9f-1be9-4deb-931a-508623c37c7a-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:15.503226 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.503192    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"]
Nov 03 01:49:15.506836 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.506798    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-tp8bb"]
Nov 03 01:49:15.608919 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.608892    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"]
Nov 03 01:49:15.615049 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.615024    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-hxxgj"]
Nov 03 01:49:15.716864 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.716838    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"]
Nov 03 01:49:15.720748 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.720728    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-qcl6n"]
Nov 03 01:49:15.824830 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.824748    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"]
Nov 03 01:49:15.828427 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:15.828402    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2b9f22c1-openshift-default-7f6cd6f957-ngt2s"]
Nov 03 01:49:16.695882 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:16.695845    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" path="/var/lib/kubelet/pods/08801e9f-1be9-4deb-931a-508623c37c7a/volumes"
Nov 03 01:49:16.696560 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:16.696541    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a8b440-97a3-47a1-83b0-fd5627a39f8d" path="/var/lib/kubelet/pods/48a8b440-97a3-47a1-83b0-fd5627a39f8d/volumes"
Nov 03 01:49:16.697153 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:16.697138    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27e6feb-5948-4fa9-b295-b73779b7291f" path="/var/lib/kubelet/pods/d27e6feb-5948-4fa9-b295-b73779b7291f/volumes"
Nov 03 01:49:16.698216 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:16.698198    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7c9be9-74ca-43af-a8f2-0667cf1d5346" path="/var/lib/kubelet/pods/fd7c9be9-74ca-43af-a8f2-0667cf1d5346/volumes"
Nov 03 01:49:17.454657 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.454616    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl" podUID="4982d2d4-a1e9-492e-b5aa-b599eb7ae954" containerName="payload-processing" containerID="cri-o://e68ed68def97bbbd6a95a4a156a84a7338ef476688744073e715e6496531a760" gracePeriod=25
Nov 03 01:49:17.559176 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.559129    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv" podUID="6ab1ecd5-69b4-4aec-a116-32fd0e14db26" containerName="payload-pre-processing" containerID="cri-o://980494dc2b851ad996aba109b7a0fff44e99c94eaec0116a07ee0c84302a100e" gracePeriod=25
Nov 03 01:49:17.734231 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.734201    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"
Nov 03 01:49:17.840325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.840293    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-plugins-config-volume\") pod \"4982d2d4-a1e9-492e-b5aa-b599eb7ae954\" (UID: \"4982d2d4-a1e9-492e-b5aa-b599eb7ae954\") "
Nov 03 01:49:17.840482 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.840376    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-kube-api-access-kz88w\" (UniqueName: \"kubernetes.io/projected/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-kube-api-access-kz88w\") pod \"4982d2d4-a1e9-492e-b5aa-b599eb7ae954\" (UID: \"4982d2d4-a1e9-492e-b5aa-b599eb7ae954\") "
Nov 03 01:49:17.840645 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.840623    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-plugins-config-volume" pod "4982d2d4-a1e9-492e-b5aa-b599eb7ae954" (UID: "4982d2d4-a1e9-492e-b5aa-b599eb7ae954"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:49:17.840907 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.840895    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:17.842531 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.842507    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-kube-api-access-kz88w" pod "4982d2d4-a1e9-492e-b5aa-b599eb7ae954" (UID: "4982d2d4-a1e9-492e-b5aa-b599eb7ae954"). InnerVolumeSpecName "kube-api-access-kz88w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:17.865916 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.863529    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"
Nov 03 01:49:17.942101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.942058    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz88w\" (UniqueName: \"kubernetes.io/projected/4982d2d4-a1e9-492e-b5aa-b599eb7ae954-kube-api-access-kz88w\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:17.993426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.993285    5508 generic.go:356] "Generic (PLEG): container finished" podID="4982d2d4-a1e9-492e-b5aa-b599eb7ae954" containerID="e68ed68def97bbbd6a95a4a156a84a7338ef476688744073e715e6496531a760" exitCode=0
Nov 03 01:49:17.993598 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.993459    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl" event={"ID":"4982d2d4-a1e9-492e-b5aa-b599eb7ae954","Type":"ContainerDied","Data":"e68ed68def97bbbd6a95a4a156a84a7338ef476688744073e715e6496531a760"}
Nov 03 01:49:17.993598 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.993512    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl" event={"ID":"4982d2d4-a1e9-492e-b5aa-b599eb7ae954","Type":"ContainerDied","Data":"3b19208683342b115b402f10bede404b0395429c534f85c274361fb2d1736ae2"}
Nov 03 01:49:17.993598 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.993543    5508 scope.go:122] "RemoveContainer" containerID="e68ed68def97bbbd6a95a4a156a84a7338ef476688744073e715e6496531a760"
Nov 03 01:49:17.993801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:17.993696    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"
Nov 03 01:49:18.000803 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.000774    5508 generic.go:356] "Generic (PLEG): container finished" podID="6ab1ecd5-69b4-4aec-a116-32fd0e14db26" containerID="980494dc2b851ad996aba109b7a0fff44e99c94eaec0116a07ee0c84302a100e" exitCode=0
Nov 03 01:49:18.000931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.000814    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"
Nov 03 01:49:18.000931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.000854    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv" event={"ID":"6ab1ecd5-69b4-4aec-a116-32fd0e14db26","Type":"ContainerDied","Data":"980494dc2b851ad996aba109b7a0fff44e99c94eaec0116a07ee0c84302a100e"}
Nov 03 01:49:18.000931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.000898    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv" event={"ID":"6ab1ecd5-69b4-4aec-a116-32fd0e14db26","Type":"ContainerDied","Data":"17ae558b0c9cad1b281d4ab548ca28f3f1f20646b9fe9235651b6e30e23453e1"}
Nov 03 01:49:18.010098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.010077    5508 scope.go:122] "RemoveContainer" containerID="e68ed68def97bbbd6a95a4a156a84a7338ef476688744073e715e6496531a760"
Nov 03 01:49:18.010409 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:18.010377    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68ed68def97bbbd6a95a4a156a84a7338ef476688744073e715e6496531a760\": container with ID starting with e68ed68def97bbbd6a95a4a156a84a7338ef476688744073e715e6496531a760 not found: ID does not exist" containerID="e68ed68def97bbbd6a95a4a156a84a7338ef476688744073e715e6496531a760"
Nov 03 01:49:18.010512 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.010419    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68ed68def97bbbd6a95a4a156a84a7338ef476688744073e715e6496531a760"} err="failed to get container status \"e68ed68def97bbbd6a95a4a156a84a7338ef476688744073e715e6496531a760\": rpc error: code = NotFound desc = could not find container \"e68ed68def97bbbd6a95a4a156a84a7338ef476688744073e715e6496531a760\": container with ID starting with e68ed68def97bbbd6a95a4a156a84a7338ef476688744073e715e6496531a760 not found: ID does not exist"
Nov 03 01:49:18.010512 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.010443    5508 scope.go:122] "RemoveContainer" containerID="980494dc2b851ad996aba109b7a0fff44e99c94eaec0116a07ee0c84302a100e"
Nov 03 01:49:18.020589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.020570    5508 scope.go:122] "RemoveContainer" containerID="980494dc2b851ad996aba109b7a0fff44e99c94eaec0116a07ee0c84302a100e"
Nov 03 01:49:18.020857 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:18.020833    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980494dc2b851ad996aba109b7a0fff44e99c94eaec0116a07ee0c84302a100e\": container with ID starting with 980494dc2b851ad996aba109b7a0fff44e99c94eaec0116a07ee0c84302a100e not found: ID does not exist" containerID="980494dc2b851ad996aba109b7a0fff44e99c94eaec0116a07ee0c84302a100e"
Nov 03 01:49:18.020914 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.020866    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980494dc2b851ad996aba109b7a0fff44e99c94eaec0116a07ee0c84302a100e"} err="failed to get container status \"980494dc2b851ad996aba109b7a0fff44e99c94eaec0116a07ee0c84302a100e\": rpc error: code = NotFound desc = could not find container \"980494dc2b851ad996aba109b7a0fff44e99c94eaec0116a07ee0c84302a100e\": container with ID starting with 980494dc2b851ad996aba109b7a0fff44e99c94eaec0116a07ee0c84302a100e not found: ID does not exist"
Nov 03 01:49:18.043306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.043285    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-plugins-config-volume\") pod \"6ab1ecd5-69b4-4aec-a116-32fd0e14db26\" (UID: \"6ab1ecd5-69b4-4aec-a116-32fd0e14db26\") "
Nov 03 01:49:18.043576 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.043558    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-kube-api-access-2rnqm\" (UniqueName: \"kubernetes.io/projected/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-kube-api-access-2rnqm\") pod \"6ab1ecd5-69b4-4aec-a116-32fd0e14db26\" (UID: \"6ab1ecd5-69b4-4aec-a116-32fd0e14db26\") "
Nov 03 01:49:18.043747 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.043720    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-plugins-config-volume" pod "6ab1ecd5-69b4-4aec-a116-32fd0e14db26" (UID: "6ab1ecd5-69b4-4aec-a116-32fd0e14db26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:49:18.044019 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.044001    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:18.045556 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.045537    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-kube-api-access-2rnqm" pod "6ab1ecd5-69b4-4aec-a116-32fd0e14db26" (UID: "6ab1ecd5-69b4-4aec-a116-32fd0e14db26"). InnerVolumeSpecName "kube-api-access-2rnqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:18.145291 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.145241    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rnqm\" (UniqueName: \"kubernetes.io/projected/6ab1ecd5-69b4-4aec-a116-32fd0e14db26-kube-api-access-2rnqm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:18.153078 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.153045    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"]
Nov 03 01:49:18.158796 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.158767    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-23e673-6994f4966f-hhdvl"]
Nov 03 01:49:18.417952 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.417916    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"]
Nov 03 01:49:18.424383 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.424321    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-6754f9c595-tfrvv"]
Nov 03 01:49:18.693744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.693707    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4982d2d4-a1e9-492e-b5aa-b599eb7ae954" path="/var/lib/kubelet/pods/4982d2d4-a1e9-492e-b5aa-b599eb7ae954/volumes"
Nov 03 01:49:18.694335 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:18.694316    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab1ecd5-69b4-4aec-a116-32fd0e14db26" path="/var/lib/kubelet/pods/6ab1ecd5-69b4-4aec-a116-32fd0e14db26/volumes"
Nov 03 01:49:19.362007 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.361968    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"]
Nov 03 01:49:19.362571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362556    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d27e6feb-5948-4fa9-b295-b73779b7291f" containerName="istio-proxy"
Nov 03 01:49:19.362571 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362571    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27e6feb-5948-4fa9-b295-b73779b7291f" containerName="istio-proxy"
Nov 03 01:49:19.362694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362583    5508 cpu_manager.go:414] "containerMap: removing container" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy"
Nov 03 01:49:19.362694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362588    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy"
Nov 03 01:49:19.362694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362605    5508 cpu_manager.go:414] "containerMap: removing container" podUID="fd7c9be9-74ca-43af-a8f2-0667cf1d5346" containerName="istio-proxy"
Nov 03 01:49:19.362694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362611    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7c9be9-74ca-43af-a8f2-0667cf1d5346" containerName="istio-proxy"
Nov 03 01:49:19.362694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362623    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4982d2d4-a1e9-492e-b5aa-b599eb7ae954" containerName="payload-processing"
Nov 03 01:49:19.362694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362628    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4982d2d4-a1e9-492e-b5aa-b599eb7ae954" containerName="payload-processing"
Nov 03 01:49:19.362694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362638    5508 cpu_manager.go:414] "containerMap: removing container" podUID="48a8b440-97a3-47a1-83b0-fd5627a39f8d" containerName="istio-proxy"
Nov 03 01:49:19.362694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362642    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a8b440-97a3-47a1-83b0-fd5627a39f8d" containerName="istio-proxy"
Nov 03 01:49:19.362694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362650    5508 cpu_manager.go:414] "containerMap: removing container" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy"
Nov 03 01:49:19.362694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362654    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy"
Nov 03 01:49:19.362694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362662    5508 cpu_manager.go:414] "containerMap: removing container" podUID="6ab1ecd5-69b4-4aec-a116-32fd0e14db26" containerName="payload-pre-processing"
Nov 03 01:49:19.362694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362666    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab1ecd5-69b4-4aec-a116-32fd0e14db26" containerName="payload-pre-processing"
Nov 03 01:49:19.363262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362748    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4982d2d4-a1e9-492e-b5aa-b599eb7ae954" containerName="payload-processing"
Nov 03 01:49:19.363262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362758    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a8b440-97a3-47a1-83b0-fd5627a39f8d" containerName="istio-proxy"
Nov 03 01:49:19.363262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362763    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy"
Nov 03 01:49:19.363262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362769    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab1ecd5-69b4-4aec-a116-32fd0e14db26" containerName="payload-pre-processing"
Nov 03 01:49:19.363262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362778    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27e6feb-5948-4fa9-b295-b73779b7291f" containerName="istio-proxy"
Nov 03 01:49:19.363262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.362787    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7c9be9-74ca-43af-a8f2-0667cf1d5346" containerName="istio-proxy"
Nov 03 01:49:19.367742 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.367720    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"
Nov 03 01:49:19.467316 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.467270    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mt2j\" (UniqueName: \"kubernetes.io/projected/a899ee86-26da-4460-aac2-bfdde9d2d6bf-kube-api-access-4mt2j\") pod \"payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2\" (UID: \"a899ee86-26da-4460-aac2-bfdde9d2d6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"
Nov 03 01:49:19.467916 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.467887    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a899ee86-26da-4460-aac2-bfdde9d2d6bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2\" (UID: \"a899ee86-26da-4460-aac2-bfdde9d2d6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"
Nov 03 01:49:19.474982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.474963    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e0c1fb-dockercfg-6tn94\""
Nov 03 01:49:19.479795 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.479777    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e0c1fb\""
Nov 03 01:49:19.493064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.493042    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"]
Nov 03 01:49:19.566208 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.566175    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"]
Nov 03 01:49:19.568595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.568569    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="08801e9f-1be9-4deb-931a-508623c37c7a" containerName="istio-proxy"
Nov 03 01:49:19.570081 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.570054    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mt2j\" (UniqueName: \"kubernetes.io/projected/a899ee86-26da-4460-aac2-bfdde9d2d6bf-kube-api-access-4mt2j\") pod \"payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2\" (UID: \"a899ee86-26da-4460-aac2-bfdde9d2d6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"
Nov 03 01:49:19.570589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.570568    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a899ee86-26da-4460-aac2-bfdde9d2d6bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2\" (UID: \"a899ee86-26da-4460-aac2-bfdde9d2d6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"
Nov 03 01:49:19.571704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.571683    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a899ee86-26da-4460-aac2-bfdde9d2d6bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2\" (UID: \"a899ee86-26da-4460-aac2-bfdde9d2d6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"
Nov 03 01:49:19.576940 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.576919    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"
Nov 03 01:49:19.625436 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.624244    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"]
Nov 03 01:49:19.645167 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.645131    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mt2j\" (UniqueName: \"kubernetes.io/projected/a899ee86-26da-4460-aac2-bfdde9d2d6bf-kube-api-access-4mt2j\") pod \"payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2\" (UID: \"a899ee86-26da-4460-aac2-bfdde9d2d6bf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"
Nov 03 01:49:19.671870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.671842    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b3fffd-f8f2-432e-867a-4179539374f7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd\" (UID: \"a1b3fffd-f8f2-432e-867a-4179539374f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"
Nov 03 01:49:19.672001 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.671880    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxdxl\" (UniqueName: \"kubernetes.io/projected/a1b3fffd-f8f2-432e-867a-4179539374f7-kube-api-access-fxdxl\") pod \"payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd\" (UID: \"a1b3fffd-f8f2-432e-867a-4179539374f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"
Nov 03 01:49:19.684865 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.684809    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"
Nov 03 01:49:19.774100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.772926    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b3fffd-f8f2-432e-867a-4179539374f7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd\" (UID: \"a1b3fffd-f8f2-432e-867a-4179539374f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"
Nov 03 01:49:19.774100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.772963    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxdxl\" (UniqueName: \"kubernetes.io/projected/a1b3fffd-f8f2-432e-867a-4179539374f7-kube-api-access-fxdxl\") pod \"payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd\" (UID: \"a1b3fffd-f8f2-432e-867a-4179539374f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"
Nov 03 01:49:19.774644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.774623    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b3fffd-f8f2-432e-867a-4179539374f7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd\" (UID: \"a1b3fffd-f8f2-432e-867a-4179539374f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"
Nov 03 01:49:19.831563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.831537    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxdxl\" (UniqueName: \"kubernetes.io/projected/a1b3fffd-f8f2-432e-867a-4179539374f7-kube-api-access-fxdxl\") pod \"payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd\" (UID: \"a1b3fffd-f8f2-432e-867a-4179539374f7\") " pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"
Nov 03 01:49:19.896159 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:19.896122    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"
Nov 03 01:49:20.058366 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:20.058323    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"]
Nov 03 01:49:20.064879 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:49:20.064851    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda899ee86_26da_4460_aac2_bfdde9d2d6bf.slice/crio-9cb5c5d51c7f48cb3c9d5ade5cebe34d71555c32b0775d45f06fbabb4d1d29ae WatchSource:0}: Error finding container 9cb5c5d51c7f48cb3c9d5ade5cebe34d71555c32b0775d45f06fbabb4d1d29ae: Status 404 returned error can't find the container with id 9cb5c5d51c7f48cb3c9d5ade5cebe34d71555c32b0775d45f06fbabb4d1d29ae
Nov 03 01:49:20.274395 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:20.274308    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"]
Nov 03 01:49:20.280873 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:49:20.280847    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b3fffd_f8f2_432e_867a_4179539374f7.slice/crio-ee8325cf3417e323d96d50d9a50f5b2a2d096155141883f211ce8537ee58e8d6 WatchSource:0}: Error finding container ee8325cf3417e323d96d50d9a50f5b2a2d096155141883f211ce8537ee58e8d6: Status 404 returned error can't find the container with id ee8325cf3417e323d96d50d9a50f5b2a2d096155141883f211ce8537ee58e8d6
Nov 03 01:49:21.026182 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:21.026145    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd" event={"ID":"a1b3fffd-f8f2-432e-867a-4179539374f7","Type":"ContainerStarted","Data":"de7b428d5ea82f88225e190483978dee54203edf42c8abd217ab164890d4cced"}
Nov 03 01:49:21.026678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:21.026205    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd" event={"ID":"a1b3fffd-f8f2-432e-867a-4179539374f7","Type":"ContainerStarted","Data":"ee8325cf3417e323d96d50d9a50f5b2a2d096155141883f211ce8537ee58e8d6"}
Nov 03 01:49:21.026678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:21.026381    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"
Nov 03 01:49:21.027979 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:21.027952    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2" event={"ID":"a899ee86-26da-4460-aac2-bfdde9d2d6bf","Type":"ContainerStarted","Data":"4730fe5ec1cb3330e88d313321185568c0093bb00d6a68459e07f18c4ad9208c"}
Nov 03 01:49:21.028100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:21.027996    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2" event={"ID":"a899ee86-26da-4460-aac2-bfdde9d2d6bf","Type":"ContainerStarted","Data":"9cb5c5d51c7f48cb3c9d5ade5cebe34d71555c32b0775d45f06fbabb4d1d29ae"}
Nov 03 01:49:21.028100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:21.028022    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"
Nov 03 01:49:21.134602 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:21.134555    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd" podStartSLOduration=2.134540154 podStartE2EDuration="2.134540154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:21.134376356 +0000 UTC m=+1840.559982850" watchObservedRunningTime="2026-11-03 01:49:21.134540154 +0000 UTC m=+1840.560146654"
Nov 03 01:49:21.259376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:21.257697    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2" podStartSLOduration=2.2576740490000002 podStartE2EDuration="2.257674049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:21.241683483 +0000 UTC m=+1840.667289948" watchObservedRunningTime="2026-11-03 01:49:21.257674049 +0000 UTC m=+1840.683280512"
Nov 03 01:49:22.045234 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:22.045195    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"]
Nov 03 01:49:22.090735 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:22.090709    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b95171d6-mjcdp"]
Nov 03 01:49:22.693608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:22.693575    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25245a3a-7fc5-440a-bc25-756977548b2d" path="/var/lib/kubelet/pods/25245a3a-7fc5-440a-bc25-756977548b2d/volumes"
Nov 03 01:49:25.492417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.492343    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"]
Nov 03 01:49:25.493035 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.492598    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr" podUID="48e0d034-4613-46a3-b1a9-0b6d12ef8801" containerName="maas-api" containerID="cri-o://d5c2a87af82916e212faa502a90c2d6be8b012df9b26459ec56ab2e2b8637c09" gracePeriod=30
Nov 03 01:49:25.826252 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.826228    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:49:25.901372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.897575    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e0d034-4613-46a3-b1a9-0b6d12ef8801-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e0d034-4613-46a3-b1a9-0b6d12ef8801-openshift-service-ca\") pod \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") "
Nov 03 01:49:25.901372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.897658    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e0d034-4613-46a3-b1a9-0b6d12ef8801-kube-api-access-l699h\" (UniqueName: \"kubernetes.io/projected/48e0d034-4613-46a3-b1a9-0b6d12ef8801-kube-api-access-l699h\") pod \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") "
Nov 03 01:49:25.901372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.897707    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-metrics-certs\") pod \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") "
Nov 03 01:49:25.901372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.897808    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-maas-api-tls\") pod \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\" (UID: \"48e0d034-4613-46a3-b1a9-0b6d12ef8801\") "
Nov 03 01:49:25.920951 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.920879    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e0d034-4613-46a3-b1a9-0b6d12ef8801-openshift-service-ca" pod "48e0d034-4613-46a3-b1a9-0b6d12ef8801" (UID: "48e0d034-4613-46a3-b1a9-0b6d12ef8801"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:49:25.920951 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.920895    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-maas-api-tls" pod "48e0d034-4613-46a3-b1a9-0b6d12ef8801" (UID: "48e0d034-4613-46a3-b1a9-0b6d12ef8801"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:49:25.936379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.935494    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e0d034-4613-46a3-b1a9-0b6d12ef8801-kube-api-access-l699h" pod "48e0d034-4613-46a3-b1a9-0b6d12ef8801" (UID: "48e0d034-4613-46a3-b1a9-0b6d12ef8801"). InnerVolumeSpecName "kube-api-access-l699h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:25.942615 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.940592    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-metrics-certs" pod "48e0d034-4613-46a3-b1a9-0b6d12ef8801" (UID: "48e0d034-4613-46a3-b1a9-0b6d12ef8801"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:49:25.999840 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.999806    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48e0d034-4613-46a3-b1a9-0b6d12ef8801-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:25.999840 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.999841    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l699h\" (UniqueName: \"kubernetes.io/projected/48e0d034-4613-46a3-b1a9-0b6d12ef8801-kube-api-access-l699h\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:26.000039 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.999856    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:26.000039 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:25.999871    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48e0d034-4613-46a3-b1a9-0b6d12ef8801-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:26.125429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.123422    5508 generic.go:356] "Generic (PLEG): container finished" podID="48e0d034-4613-46a3-b1a9-0b6d12ef8801" containerID="d5c2a87af82916e212faa502a90c2d6be8b012df9b26459ec56ab2e2b8637c09" exitCode=0
Nov 03 01:49:26.125429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.123545    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr" event={"ID":"48e0d034-4613-46a3-b1a9-0b6d12ef8801","Type":"ContainerDied","Data":"d5c2a87af82916e212faa502a90c2d6be8b012df9b26459ec56ab2e2b8637c09"}
Nov 03 01:49:26.125429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.123651    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr" event={"ID":"48e0d034-4613-46a3-b1a9-0b6d12ef8801","Type":"ContainerDied","Data":"690d28564667c16cdbc139b7e656d01d0735dc441e76067850c60ad567c47362"}
Nov 03 01:49:26.125429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.123694    5508 scope.go:122] "RemoveContainer" containerID="d5c2a87af82916e212faa502a90c2d6be8b012df9b26459ec56ab2e2b8637c09"
Nov 03 01:49:26.125429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.123878    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"
Nov 03 01:49:26.152181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.152155    5508 scope.go:122] "RemoveContainer" containerID="d5c2a87af82916e212faa502a90c2d6be8b012df9b26459ec56ab2e2b8637c09"
Nov 03 01:49:26.152548 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:26.152528    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c2a87af82916e212faa502a90c2d6be8b012df9b26459ec56ab2e2b8637c09\": container with ID starting with d5c2a87af82916e212faa502a90c2d6be8b012df9b26459ec56ab2e2b8637c09 not found: ID does not exist" containerID="d5c2a87af82916e212faa502a90c2d6be8b012df9b26459ec56ab2e2b8637c09"
Nov 03 01:49:26.152658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.152556    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c2a87af82916e212faa502a90c2d6be8b012df9b26459ec56ab2e2b8637c09"} err="failed to get container status \"d5c2a87af82916e212faa502a90c2d6be8b012df9b26459ec56ab2e2b8637c09\": rpc error: code = NotFound desc = could not find container \"d5c2a87af82916e212faa502a90c2d6be8b012df9b26459ec56ab2e2b8637c09\": container with ID starting with d5c2a87af82916e212faa502a90c2d6be8b012df9b26459ec56ab2e2b8637c09 not found: ID does not exist"
Nov 03 01:49:26.284135 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.284104    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"]
Nov 03 01:49:26.288807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.288779    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-109234-b64dcd57c-v8qzr"]
Nov 03 01:49:26.609320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.609260    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"]
Nov 03 01:49:26.610244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.610217    5508 cpu_manager.go:414] "containerMap: removing container" podUID="48e0d034-4613-46a3-b1a9-0b6d12ef8801" containerName="maas-api"
Nov 03 01:49:26.610244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.610244    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e0d034-4613-46a3-b1a9-0b6d12ef8801" containerName="maas-api"
Nov 03 01:49:26.610476 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.610442    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e0d034-4613-46a3-b1a9-0b6d12ef8801" containerName="maas-api"
Nov 03 01:49:26.623547 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.616463    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.650178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.650142    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"]
Nov 03 01:49:26.668122 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.668097    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"]
Nov 03 01:49:26.691063 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.691035    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-86e679-openshift-default-dockercfg-4r79v\""
Nov 03 01:49:26.693260 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.693222    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e0d034-4613-46a3-b1a9-0b6d12ef8801" path="/var/lib/kubelet/pods/48e0d034-4613-46a3-b1a9-0b6d12ef8801/volumes"
Nov 03 01:49:26.745532 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.745494    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-socket\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.745716 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.745606    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-credential-socket\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.745716 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.745634    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-istio-token\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.745716 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.745699    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-envoy\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.745842 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.745830    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-certs\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.745882 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.745854    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24d34b3-8841-4389-b20f-297008aa92a5-istio-podinfo\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.745882 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.745871    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24d34b3-8841-4389-b20f-297008aa92a5-istiod-ca-cert\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.746006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.745991    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-data\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.746050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.746031    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkg7\" (UniqueName: \"kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-kube-api-access-hnkg7\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.769786 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.769688    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"]
Nov 03 01:49:26.847846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.847809    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-certs\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.848052 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.847856    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24d34b3-8841-4389-b20f-297008aa92a5-istio-podinfo\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.848052 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.847874    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24d34b3-8841-4389-b20f-297008aa92a5-istiod-ca-cert\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.848052 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.847913    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-data\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.848052 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.847951    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkg7\" (UniqueName: \"kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-kube-api-access-hnkg7\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.848052 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.848021    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-socket\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.848336 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.848086    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-credential-socket\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.848336 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.848121    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-istio-token\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.848336 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.848159    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-envoy\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.848336 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.848281    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-certs\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.849011 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.848985    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-data\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.849332 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.849236    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-credential-socket\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.849480 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.849458    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-socket\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.849558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.849513    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24d34b3-8841-4389-b20f-297008aa92a5-istiod-ca-cert\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.857363 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.852712    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-envoy\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.857363 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.853145    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24d34b3-8841-4389-b20f-297008aa92a5-istio-podinfo\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.940407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.932265    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkg7\" (UniqueName: \"kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-kube-api-access-hnkg7\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.940407 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.937677    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-istio-token\") pod \"e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") " pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:26.975835 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:26.975578    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:27.430143 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:27.430104    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"]
Nov 03 01:49:27.461783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:27.461743    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:49:27.461943 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:27.461837    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:49:27.461943 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:27.461879    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:49:28.155982 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:28.155953    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" event={"ID":"d24d34b3-8841-4389-b20f-297008aa92a5","Type":"ContainerStarted","Data":"48682b93a6ccc6d2a688417ccb7fe4ffcf89f32dc54845b74ee24ff96ba72fd6"}
Nov 03 01:49:28.156495 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:28.156456    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" event={"ID":"d24d34b3-8841-4389-b20f-297008aa92a5","Type":"ContainerStarted","Data":"0b3dc16b2498b59f5e01e9560609127c13f0d2b875bf036fed5de7968558f3b4"}
Nov 03 01:49:28.273617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:28.273548    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" podStartSLOduration=2.273532819 podStartE2EDuration="2.273532819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:28.268418876 +0000 UTC m=+1847.694025340" watchObservedRunningTime="2026-11-03 01:49:28.273532819 +0000 UTC m=+1847.699139352"
Nov 03 01:49:28.983693 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:28.983664    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:28.989519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:28.989495    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:29.162362 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:29.162314    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:29.163162 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:29.163144    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:49:31.668656 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:31.668609    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r" podUID="8a2934d5-bbcc-4988-9566-3a3d0b16ec35" containerName="payload-processing" containerID="cri-o://499965c6a49610b4ba49b599d8f38cb70c12ff048392c38c4f063386ef0519f9" gracePeriod=25
Nov 03 01:49:31.770424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:31.770384    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx" podUID="24931ff3-bf48-42d9-93d3-52456caf73b4" containerName="payload-pre-processing" containerID="cri-o://84bf770ac94cbb58d4b596245e4a028722a8e5796f56c8b3a660e405a94c759a" gracePeriod=25
Nov 03 01:49:31.958057 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:31.958030    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"
Nov 03 01:49:32.028826 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.028787    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-plugins-config-volume\") pod \"8a2934d5-bbcc-4988-9566-3a3d0b16ec35\" (UID: \"8a2934d5-bbcc-4988-9566-3a3d0b16ec35\") "
Nov 03 01:49:32.029008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.028853    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-kube-api-access-gjt9s\" (UniqueName: \"kubernetes.io/projected/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-kube-api-access-gjt9s\") pod \"8a2934d5-bbcc-4988-9566-3a3d0b16ec35\" (UID: \"8a2934d5-bbcc-4988-9566-3a3d0b16ec35\") "
Nov 03 01:49:32.029296 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.029248    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-plugins-config-volume" pod "8a2934d5-bbcc-4988-9566-3a3d0b16ec35" (UID: "8a2934d5-bbcc-4988-9566-3a3d0b16ec35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:49:32.030032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.030006    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:32.032966 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.032942    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-kube-api-access-gjt9s" pod "8a2934d5-bbcc-4988-9566-3a3d0b16ec35" (UID: "8a2934d5-bbcc-4988-9566-3a3d0b16ec35"). InnerVolumeSpecName "kube-api-access-gjt9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:32.044183 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.044165    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"
Nov 03 01:49:32.050932 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.050908    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"
Nov 03 01:49:32.058634 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.058615    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"
Nov 03 01:49:32.130646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.130615    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24931ff3-bf48-42d9-93d3-52456caf73b4-kube-api-access-v4hkk\" (UniqueName: \"kubernetes.io/projected/24931ff3-bf48-42d9-93d3-52456caf73b4-kube-api-access-v4hkk\") pod \"24931ff3-bf48-42d9-93d3-52456caf73b4\" (UID: \"24931ff3-bf48-42d9-93d3-52456caf73b4\") "
Nov 03 01:49:32.130800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.130671    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24931ff3-bf48-42d9-93d3-52456caf73b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24931ff3-bf48-42d9-93d3-52456caf73b4-plugins-config-volume\") pod \"24931ff3-bf48-42d9-93d3-52456caf73b4\" (UID: \"24931ff3-bf48-42d9-93d3-52456caf73b4\") "
Nov 03 01:49:32.131047 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.131024    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24931ff3-bf48-42d9-93d3-52456caf73b4-plugins-config-volume" pod "24931ff3-bf48-42d9-93d3-52456caf73b4" (UID: "24931ff3-bf48-42d9-93d3-52456caf73b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:49:32.131284 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.131262    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjt9s\" (UniqueName: \"kubernetes.io/projected/8a2934d5-bbcc-4988-9566-3a3d0b16ec35-kube-api-access-gjt9s\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:32.131402 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.131289    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24931ff3-bf48-42d9-93d3-52456caf73b4-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:32.132891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.132869    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24931ff3-bf48-42d9-93d3-52456caf73b4-kube-api-access-v4hkk" pod "24931ff3-bf48-42d9-93d3-52456caf73b4" (UID: "24931ff3-bf48-42d9-93d3-52456caf73b4"). InnerVolumeSpecName "kube-api-access-v4hkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:32.185522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.185444    5508 generic.go:356] "Generic (PLEG): container finished" podID="8a2934d5-bbcc-4988-9566-3a3d0b16ec35" containerID="499965c6a49610b4ba49b599d8f38cb70c12ff048392c38c4f063386ef0519f9" exitCode=0
Nov 03 01:49:32.185522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.185514    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"
Nov 03 01:49:32.185719 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.185529    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r" event={"ID":"8a2934d5-bbcc-4988-9566-3a3d0b16ec35","Type":"ContainerDied","Data":"499965c6a49610b4ba49b599d8f38cb70c12ff048392c38c4f063386ef0519f9"}
Nov 03 01:49:32.185719 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.185597    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r" event={"ID":"8a2934d5-bbcc-4988-9566-3a3d0b16ec35","Type":"ContainerDied","Data":"7785830ff65f99cf1fa604773d0d87e6f3b0f8fed8872faac03f8cd5aec42cdb"}
Nov 03 01:49:32.185719 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.185626    5508 scope.go:122] "RemoveContainer" containerID="499965c6a49610b4ba49b599d8f38cb70c12ff048392c38c4f063386ef0519f9"
Nov 03 01:49:32.187858 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.187836    5508 generic.go:356] "Generic (PLEG): container finished" podID="24931ff3-bf48-42d9-93d3-52456caf73b4" containerID="84bf770ac94cbb58d4b596245e4a028722a8e5796f56c8b3a660e405a94c759a" exitCode=0
Nov 03 01:49:32.187959 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.187887    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx" event={"ID":"24931ff3-bf48-42d9-93d3-52456caf73b4","Type":"ContainerDied","Data":"84bf770ac94cbb58d4b596245e4a028722a8e5796f56c8b3a660e405a94c759a"}
Nov 03 01:49:32.187959 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.187924    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx" event={"ID":"24931ff3-bf48-42d9-93d3-52456caf73b4","Type":"ContainerDied","Data":"1a12d5ad8898df6cfa576c6de073a4acbf76b58d230612819866a580c5651b47"}
Nov 03 01:49:32.188073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.187987    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"
Nov 03 01:49:32.197581 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.197442    5508 scope.go:122] "RemoveContainer" containerID="499965c6a49610b4ba49b599d8f38cb70c12ff048392c38c4f063386ef0519f9"
Nov 03 01:49:32.198729 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:32.197886    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499965c6a49610b4ba49b599d8f38cb70c12ff048392c38c4f063386ef0519f9\": container with ID starting with 499965c6a49610b4ba49b599d8f38cb70c12ff048392c38c4f063386ef0519f9 not found: ID does not exist" containerID="499965c6a49610b4ba49b599d8f38cb70c12ff048392c38c4f063386ef0519f9"
Nov 03 01:49:32.198729 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.197913    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499965c6a49610b4ba49b599d8f38cb70c12ff048392c38c4f063386ef0519f9"} err="failed to get container status \"499965c6a49610b4ba49b599d8f38cb70c12ff048392c38c4f063386ef0519f9\": rpc error: code = NotFound desc = could not find container \"499965c6a49610b4ba49b599d8f38cb70c12ff048392c38c4f063386ef0519f9\": container with ID starting with 499965c6a49610b4ba49b599d8f38cb70c12ff048392c38c4f063386ef0519f9 not found: ID does not exist"
Nov 03 01:49:32.198729 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.197935    5508 scope.go:122] "RemoveContainer" containerID="84bf770ac94cbb58d4b596245e4a028722a8e5796f56c8b3a660e405a94c759a"
Nov 03 01:49:32.210209 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.210191    5508 scope.go:122] "RemoveContainer" containerID="84bf770ac94cbb58d4b596245e4a028722a8e5796f56c8b3a660e405a94c759a"
Nov 03 01:49:32.211617 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:32.211588    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84bf770ac94cbb58d4b596245e4a028722a8e5796f56c8b3a660e405a94c759a\": container with ID starting with 84bf770ac94cbb58d4b596245e4a028722a8e5796f56c8b3a660e405a94c759a not found: ID does not exist" containerID="84bf770ac94cbb58d4b596245e4a028722a8e5796f56c8b3a660e405a94c759a"
Nov 03 01:49:32.211771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.211626    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84bf770ac94cbb58d4b596245e4a028722a8e5796f56c8b3a660e405a94c759a"} err="failed to get container status \"84bf770ac94cbb58d4b596245e4a028722a8e5796f56c8b3a660e405a94c759a\": rpc error: code = NotFound desc = could not find container \"84bf770ac94cbb58d4b596245e4a028722a8e5796f56c8b3a660e405a94c759a\": container with ID starting with 84bf770ac94cbb58d4b596245e4a028722a8e5796f56c8b3a660e405a94c759a not found: ID does not exist"
Nov 03 01:49:32.232306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.232279    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4hkk\" (UniqueName: \"kubernetes.io/projected/24931ff3-bf48-42d9-93d3-52456caf73b4-kube-api-access-v4hkk\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:32.535953 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.535859    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"]
Nov 03 01:49:32.540625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.540600    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-6bcf8685c7-j769r"]
Nov 03 01:49:32.667741 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.666336    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"]
Nov 03 01:49:32.674057 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.671624    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-7d785f4c75-zv2dx"]
Nov 03 01:49:32.694472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.694447    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24931ff3-bf48-42d9-93d3-52456caf73b4" path="/var/lib/kubelet/pods/24931ff3-bf48-42d9-93d3-52456caf73b4/volumes"
Nov 03 01:49:32.694892 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:32.694880    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2934d5-bbcc-4988-9566-3a3d0b16ec35" path="/var/lib/kubelet/pods/8a2934d5-bbcc-4988-9566-3a3d0b16ec35/volumes"
Nov 03 01:49:35.438842 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.438810    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"]
Nov 03 01:49:35.439377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.439363    5508 cpu_manager.go:414] "containerMap: removing container" podUID="24931ff3-bf48-42d9-93d3-52456caf73b4" containerName="payload-pre-processing"
Nov 03 01:49:35.439433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.439378    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24931ff3-bf48-42d9-93d3-52456caf73b4" containerName="payload-pre-processing"
Nov 03 01:49:35.439433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.439399    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8a2934d5-bbcc-4988-9566-3a3d0b16ec35" containerName="payload-processing"
Nov 03 01:49:35.439433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.439404    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2934d5-bbcc-4988-9566-3a3d0b16ec35" containerName="payload-processing"
Nov 03 01:49:35.439520 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.439504    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="24931ff3-bf48-42d9-93d3-52456caf73b4" containerName="payload-pre-processing"
Nov 03 01:49:35.439520 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.439514    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2934d5-bbcc-4988-9566-3a3d0b16ec35" containerName="payload-processing"
Nov 03 01:49:35.444563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.444541    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"
Nov 03 01:49:35.497857 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.497764    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"]
Nov 03 01:49:35.543379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.542599    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-86e679\""
Nov 03 01:49:35.543379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.542782    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-86e679-dockercfg-s2czn\""
Nov 03 01:49:35.645372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.624473    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1172569a-2f6a-4327-8f03-c65d14c4a678-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9\" (UID: \"1172569a-2f6a-4327-8f03-c65d14c4a678\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"
Nov 03 01:49:35.645372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.624549    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg66k\" (UniqueName: \"kubernetes.io/projected/1172569a-2f6a-4327-8f03-c65d14c4a678-kube-api-access-tg66k\") pod \"payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9\" (UID: \"1172569a-2f6a-4327-8f03-c65d14c4a678\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"
Nov 03 01:49:35.672368 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.671053    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"]
Nov 03 01:49:35.681370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.680988    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"
Nov 03 01:49:35.707431 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.703185    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"]
Nov 03 01:49:35.745190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.745163    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7d57\" (UniqueName: \"kubernetes.io/projected/327f2e67-8150-401c-8f37-08bb6966384b-kube-api-access-j7d57\") pod \"payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm\" (UID: \"327f2e67-8150-401c-8f37-08bb6966384b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"
Nov 03 01:49:35.745382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.745250    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1172569a-2f6a-4327-8f03-c65d14c4a678-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9\" (UID: \"1172569a-2f6a-4327-8f03-c65d14c4a678\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"
Nov 03 01:49:35.745382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.745300    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg66k\" (UniqueName: \"kubernetes.io/projected/1172569a-2f6a-4327-8f03-c65d14c4a678-kube-api-access-tg66k\") pod \"payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9\" (UID: \"1172569a-2f6a-4327-8f03-c65d14c4a678\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"
Nov 03 01:49:35.745483 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.745391    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327f2e67-8150-401c-8f37-08bb6966384b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm\" (UID: \"327f2e67-8150-401c-8f37-08bb6966384b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"
Nov 03 01:49:35.746263 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.746243    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1172569a-2f6a-4327-8f03-c65d14c4a678-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9\" (UID: \"1172569a-2f6a-4327-8f03-c65d14c4a678\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"
Nov 03 01:49:35.803534 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.803505    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg66k\" (UniqueName: \"kubernetes.io/projected/1172569a-2f6a-4327-8f03-c65d14c4a678-kube-api-access-tg66k\") pod \"payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9\" (UID: \"1172569a-2f6a-4327-8f03-c65d14c4a678\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"
Nov 03 01:49:35.846311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.846287    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7d57\" (UniqueName: \"kubernetes.io/projected/327f2e67-8150-401c-8f37-08bb6966384b-kube-api-access-j7d57\") pod \"payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm\" (UID: \"327f2e67-8150-401c-8f37-08bb6966384b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"
Nov 03 01:49:35.846498 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.846448    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327f2e67-8150-401c-8f37-08bb6966384b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm\" (UID: \"327f2e67-8150-401c-8f37-08bb6966384b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"
Nov 03 01:49:35.847111 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.847085    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327f2e67-8150-401c-8f37-08bb6966384b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm\" (UID: \"327f2e67-8150-401c-8f37-08bb6966384b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"
Nov 03 01:49:35.903309 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:35.903281    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7d57\" (UniqueName: \"kubernetes.io/projected/327f2e67-8150-401c-8f37-08bb6966384b-kube-api-access-j7d57\") pod \"payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm\" (UID: \"327f2e67-8150-401c-8f37-08bb6966384b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"
Nov 03 01:49:36.019873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:36.019798    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"
Nov 03 01:49:36.059469 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:36.059430    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"
Nov 03 01:49:36.400215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:36.400186    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"]
Nov 03 01:49:36.406217 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:49:36.406195    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327f2e67_8150_401c_8f37_08bb6966384b.slice/crio-e110e03063298bdf39a47cc2bfe53c49947181497b495680c3f36e4239028dd2 WatchSource:0}: Error finding container e110e03063298bdf39a47cc2bfe53c49947181497b495680c3f36e4239028dd2: Status 404 returned error can't find the container with id e110e03063298bdf39a47cc2bfe53c49947181497b495680c3f36e4239028dd2
Nov 03 01:49:36.514733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:36.514693    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"]
Nov 03 01:49:36.520689 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:49:36.520667    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1172569a_2f6a_4327_8f03_c65d14c4a678.slice/crio-fc2f9e206a93ba665bfb1f413df97b53967574f5182691695c0e8247a93dd783 WatchSource:0}: Error finding container fc2f9e206a93ba665bfb1f413df97b53967574f5182691695c0e8247a93dd783: Status 404 returned error can't find the container with id fc2f9e206a93ba665bfb1f413df97b53967574f5182691695c0e8247a93dd783
Nov 03 01:49:37.307024 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.306986    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9" event={"ID":"1172569a-2f6a-4327-8f03-c65d14c4a678","Type":"ContainerStarted","Data":"26ee5d1c80ca9f18b5a4e5759e18704781f9286d25ea47b8aa1d2c6ff159bc8b"}
Nov 03 01:49:37.307192 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.307050    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9" event={"ID":"1172569a-2f6a-4327-8f03-c65d14c4a678","Type":"ContainerStarted","Data":"fc2f9e206a93ba665bfb1f413df97b53967574f5182691695c0e8247a93dd783"}
Nov 03 01:49:37.307192 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.307112    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"
Nov 03 01:49:37.309055 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.309027    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm" event={"ID":"327f2e67-8150-401c-8f37-08bb6966384b","Type":"ContainerStarted","Data":"dc1293fb37276f7a1e2f36ee829022f179aa4f29800833899dbb8b14fced128d"}
Nov 03 01:49:37.309181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.309076    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm" event={"ID":"327f2e67-8150-401c-8f37-08bb6966384b","Type":"ContainerStarted","Data":"e110e03063298bdf39a47cc2bfe53c49947181497b495680c3f36e4239028dd2"}
Nov 03 01:49:37.309181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.309112    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"
Nov 03 01:49:37.418155 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.418104    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9" podStartSLOduration=2.418090239 podStartE2EDuration="2.418090239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:37.414938091 +0000 UTC m=+1856.840544556" watchObservedRunningTime="2026-11-03 01:49:37.418090239 +0000 UTC m=+1856.843696704"
Nov 03 01:49:37.526239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.526180    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm" podStartSLOduration=2.526163804 podStartE2EDuration="2.526163804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:37.524743239 +0000 UTC m=+1856.950349722" watchObservedRunningTime="2026-11-03 01:49:37.526163804 +0000 UTC m=+1856.951770270"
Nov 03 01:49:37.722071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.722037    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"]
Nov 03 01:49:37.726156 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.726132    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:37.777098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.777067    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-36c2c1-serving-cert\""
Nov 03 01:49:37.778623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.778269    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-metrics-certs\") pod \"maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:37.778623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.778452    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8474ff81-60d1-4464-bd05-e24fd663f34f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:37.778623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.778508    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:37.778623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.778605    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9nr4\" (UniqueName: \"kubernetes.io/projected/8474ff81-60d1-4464-bd05-e24fd663f34f-kube-api-access-k9nr4\") pod \"maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:37.782071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.782051    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"]
Nov 03 01:49:37.879923 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.879892    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9nr4\" (UniqueName: \"kubernetes.io/projected/8474ff81-60d1-4464-bd05-e24fd663f34f-kube-api-access-k9nr4\") pod \"maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:37.880103 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.880033    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-metrics-certs\") pod \"maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:37.880909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.880182    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8474ff81-60d1-4464-bd05-e24fd663f34f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:37.880909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.880228    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:37.880909 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:37.880343    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-serving-cert: secret "maas-api-e2e-ipp-a-36c2c1-serving-cert" not found
Nov 03 01:49:37.880909 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:37.880427    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-maas-api-tls podName:8474ff81-60d1-4464-bd05-e24fd663f34f nodeName:}" failed. No retries permitted until 2026-11-03 01:49:38.380408056 +0000 UTC m=+1857.806014500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-maas-api-tls") pod "maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm" (UID: "8474ff81-60d1-4464-bd05-e24fd663f34f") : secret "maas-api-e2e-ipp-a-36c2c1-serving-cert" not found
Nov 03 01:49:37.881360 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.881324    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8474ff81-60d1-4464-bd05-e24fd663f34f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:37.884084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.884020    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-metrics-certs\") pod \"maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:37.937566 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:37.937536    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9nr4\" (UniqueName: \"kubernetes.io/projected/8474ff81-60d1-4464-bd05-e24fd663f34f-kube-api-access-k9nr4\") pod \"maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:38.401844 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:38.401810    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:38.404181 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:38.404160    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:38.640411 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:38.640380    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:39.023219 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:39.023144    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"]
Nov 03 01:49:39.030253 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:49:39.030221    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8474ff81_60d1_4464_bd05_e24fd663f34f.slice/crio-38787ccedcc22556792aa23c84ce08f3d999709ca4c4f599fedc3d59fcfdc5e0 WatchSource:0}: Error finding container 38787ccedcc22556792aa23c84ce08f3d999709ca4c4f599fedc3d59fcfdc5e0: Status 404 returned error can't find the container with id 38787ccedcc22556792aa23c84ce08f3d999709ca4c4f599fedc3d59fcfdc5e0
Nov 03 01:49:39.326374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:39.326261    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm" event={"ID":"8474ff81-60d1-4464-bd05-e24fd663f34f","Type":"ContainerStarted","Data":"38787ccedcc22556792aa23c84ce08f3d999709ca4c4f599fedc3d59fcfdc5e0"}
Nov 03 01:49:39.589420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:39.582594    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"]
Nov 03 01:49:39.589420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:39.582823    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerName="istio-proxy" containerID="cri-o://ac7bfcb8128b94c85b3feb988a544148a4749acf4d76d5ead6904fb5c27e4c83" gracePeriod=30
Nov 03 01:49:40.335074 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:40.334981    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm" event={"ID":"8474ff81-60d1-4464-bd05-e24fd663f34f","Type":"ContainerStarted","Data":"0ce64158fdfa193acd26660a1f4b26024e6e953ceb5a93f345873075826fe630"}
Nov 03 01:49:40.335479 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:40.335074    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:40.351289 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:40.351265    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:49:40.417719 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:40.417690    5508 patch_prober.go:29] interesting pod/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:49:40.417894 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:40.417746    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:49:40.445698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:40.445494    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm" podStartSLOduration=2.453993739 podStartE2EDuration="3.445476266s" totalImagesPullingTime="991.482527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:40.444467989 +0000 UTC m=+1859.870074457" watchObservedRunningTime="2026-11-03 01:49:40.445476266 +0000 UTC m=+1859.871082731"
Nov 03 01:49:43.812154 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:43.812107    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:49:44.851733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:44.851702    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:49:45.000497 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.000416    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-socket\") pod \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") "
Nov 03 01:49:45.000497 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.000478    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-credential-socket\") pod \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") "
Nov 03 01:49:45.000733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.000535    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-data\") pod \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") "
Nov 03 01:49:45.000733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.000591    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-certs\") pod \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") "
Nov 03 01:49:45.000733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.000626    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-kube-api-access-2f2kl\" (UniqueName: \"kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-kube-api-access-2f2kl\") pod \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") "
Nov 03 01:49:45.000733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.000684    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istiod-ca-cert\") pod \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") "
Nov 03 01:49:45.000733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.000711    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-token\" (UniqueName: \"kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-token\") pod \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") "
Nov 03 01:49:45.000995 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.000763    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-envoy\") pod \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") "
Nov 03 01:49:45.000995 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.000701    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-socket" pod "d7e96865-e9ca-4040-bfe1-27d0c745ca85" (UID: "d7e96865-e9ca-4040-bfe1-27d0c745ca85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:45.000995 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.000833    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-certs" pod "d7e96865-e9ca-4040-bfe1-27d0c745ca85" (UID: "d7e96865-e9ca-4040-bfe1-27d0c745ca85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:45.000995 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.000845    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-podinfo\") pod \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\" (UID: \"d7e96865-e9ca-4040-bfe1-27d0c745ca85\") "
Nov 03 01:49:45.000995 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.000847    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-data" pod "d7e96865-e9ca-4040-bfe1-27d0c745ca85" (UID: "d7e96865-e9ca-4040-bfe1-27d0c745ca85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:45.000995 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.000920    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-credential-socket" pod "d7e96865-e9ca-4040-bfe1-27d0c745ca85" (UID: "d7e96865-e9ca-4040-bfe1-27d0c745ca85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:45.001283 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.001106    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istiod-ca-cert" pod "d7e96865-e9ca-4040-bfe1-27d0c745ca85" (UID: "d7e96865-e9ca-4040-bfe1-27d0c745ca85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:49:45.001565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.001540    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:45.001704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.001568    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:45.001704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.001583    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:45.001704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.001597    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:45.001704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.001610    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:45.003096 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.003072    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-envoy" pod "d7e96865-e9ca-4040-bfe1-27d0c745ca85" (UID: "d7e96865-e9ca-4040-bfe1-27d0c745ca85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:49:45.003195 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.003098    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-podinfo" pod "d7e96865-e9ca-4040-bfe1-27d0c745ca85" (UID: "d7e96865-e9ca-4040-bfe1-27d0c745ca85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:49:45.003195 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.003107    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-kube-api-access-2f2kl" pod "d7e96865-e9ca-4040-bfe1-27d0c745ca85" (UID: "d7e96865-e9ca-4040-bfe1-27d0c745ca85"). InnerVolumeSpecName "kube-api-access-2f2kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:45.003278 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.003195    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-token" pod "d7e96865-e9ca-4040-bfe1-27d0c745ca85" (UID: "d7e96865-e9ca-4040-bfe1-27d0c745ca85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:49:45.103099 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.103063    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f2kl\" (UniqueName: \"kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-kube-api-access-2f2kl\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:45.103099 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.103097    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:45.103099 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.103108    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:45.103319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.103116    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7e96865-e9ca-4040-bfe1-27d0c745ca85-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:49:45.491814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.491337    5508 generic.go:356] "Generic (PLEG): container finished" podID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerID="ac7bfcb8128b94c85b3feb988a544148a4749acf4d76d5ead6904fb5c27e4c83" exitCode=0
Nov 03 01:49:45.491814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.491397    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" event={"ID":"d7e96865-e9ca-4040-bfe1-27d0c745ca85","Type":"ContainerDied","Data":"ac7bfcb8128b94c85b3feb988a544148a4749acf4d76d5ead6904fb5c27e4c83"}
Nov 03 01:49:45.491814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.491438    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"
Nov 03 01:49:45.491814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.491465    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c" event={"ID":"d7e96865-e9ca-4040-bfe1-27d0c745ca85","Type":"ContainerDied","Data":"5acfd0e3f03be014cc6c6569a63b21196b9d9ef71799f01ba6288f090c085068"}
Nov 03 01:49:45.491814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.491495    5508 scope.go:122] "RemoveContainer" containerID="ac7bfcb8128b94c85b3feb988a544148a4749acf4d76d5ead6904fb5c27e4c83"
Nov 03 01:49:45.503123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.503108    5508 scope.go:122] "RemoveContainer" containerID="bb0bcf5c0f6c82e59ce87cd1f457690c80d9e800c602b4af418edacea49e6d3e"
Nov 03 01:49:45.512005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.511987    5508 scope.go:122] "RemoveContainer" containerID="ac7bfcb8128b94c85b3feb988a544148a4749acf4d76d5ead6904fb5c27e4c83"
Nov 03 01:49:45.512268 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:45.512246    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7bfcb8128b94c85b3feb988a544148a4749acf4d76d5ead6904fb5c27e4c83\": container with ID starting with ac7bfcb8128b94c85b3feb988a544148a4749acf4d76d5ead6904fb5c27e4c83 not found: ID does not exist" containerID="ac7bfcb8128b94c85b3feb988a544148a4749acf4d76d5ead6904fb5c27e4c83"
Nov 03 01:49:45.512334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.512280    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7bfcb8128b94c85b3feb988a544148a4749acf4d76d5ead6904fb5c27e4c83"} err="failed to get container status \"ac7bfcb8128b94c85b3feb988a544148a4749acf4d76d5ead6904fb5c27e4c83\": rpc error: code = NotFound desc = could not find container \"ac7bfcb8128b94c85b3feb988a544148a4749acf4d76d5ead6904fb5c27e4c83\": container with ID starting with ac7bfcb8128b94c85b3feb988a544148a4749acf4d76d5ead6904fb5c27e4c83 not found: ID does not exist"
Nov 03 01:49:45.512334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.512302    5508 scope.go:122] "RemoveContainer" containerID="bb0bcf5c0f6c82e59ce87cd1f457690c80d9e800c602b4af418edacea49e6d3e"
Nov 03 01:49:45.512561 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:49:45.512543    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0bcf5c0f6c82e59ce87cd1f457690c80d9e800c602b4af418edacea49e6d3e\": container with ID starting with bb0bcf5c0f6c82e59ce87cd1f457690c80d9e800c602b4af418edacea49e6d3e not found: ID does not exist" containerID="bb0bcf5c0f6c82e59ce87cd1f457690c80d9e800c602b4af418edacea49e6d3e"
Nov 03 01:49:45.512608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.512567    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0bcf5c0f6c82e59ce87cd1f457690c80d9e800c602b4af418edacea49e6d3e"} err="failed to get container status \"bb0bcf5c0f6c82e59ce87cd1f457690c80d9e800c602b4af418edacea49e6d3e\": rpc error: code = NotFound desc = could not find container \"bb0bcf5c0f6c82e59ce87cd1f457690c80d9e800c602b4af418edacea49e6d3e\": container with ID starting with bb0bcf5c0f6c82e59ce87cd1f457690c80d9e800c602b4af418edacea49e6d3e not found: ID does not exist"
Nov 03 01:49:45.646914 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.646887    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"]
Nov 03 01:49:45.694184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:45.694157    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-109234-openshift-default-5b7f96dd5-2rh8c"]
Nov 03 01:49:46.458631 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:46.458600    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:49:46.699856 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:46.699825    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" path="/var/lib/kubelet/pods/d7e96865-e9ca-4040-bfe1-27d0c745ca85/volumes"
Nov 03 01:49:48.319816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:48.319789    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"
Nov 03 01:49:48.320210 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:48.319855    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"
Nov 03 01:49:48.877634 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:48.877602    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"]
Nov 03 01:49:48.878164 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:48.878152    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerName="istio-proxy"
Nov 03 01:49:48.878207 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:48.878166    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerName="istio-proxy"
Nov 03 01:49:48.878282 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:48.878274    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerName="istio-proxy"
Nov 03 01:49:48.878320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:48.878284    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerName="istio-proxy"
Nov 03 01:49:48.883235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:48.883219    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"
Nov 03 01:49:48.934627 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:48.934593    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"]
Nov 03 01:49:48.952397 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:48.952336    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f9264c-091c-4a9a-9f38-e5c5923ebb59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7\" (UID: \"33f9264c-091c-4a9a-9f38-e5c5923ebb59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"
Nov 03 01:49:48.952520 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:48.952428    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p68l\" (UniqueName: \"kubernetes.io/projected/33f9264c-091c-4a9a-9f38-e5c5923ebb59-kube-api-access-8p68l\") pod \"payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7\" (UID: \"33f9264c-091c-4a9a-9f38-e5c5923ebb59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"
Nov 03 01:49:49.006371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:48.999706    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b177c9-dockercfg-nlrpb\""
Nov 03 01:49:49.006371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:48.999784    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b177c9\""
Nov 03 01:49:49.062391 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.060863    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f9264c-091c-4a9a-9f38-e5c5923ebb59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7\" (UID: \"33f9264c-091c-4a9a-9f38-e5c5923ebb59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"
Nov 03 01:49:49.062391 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.060909    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p68l\" (UniqueName: \"kubernetes.io/projected/33f9264c-091c-4a9a-9f38-e5c5923ebb59-kube-api-access-8p68l\") pod \"payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7\" (UID: \"33f9264c-091c-4a9a-9f38-e5c5923ebb59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"
Nov 03 01:49:49.062391 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.062192    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f9264c-091c-4a9a-9f38-e5c5923ebb59-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7\" (UID: \"33f9264c-091c-4a9a-9f38-e5c5923ebb59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"
Nov 03 01:49:49.101174 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.101135    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"]
Nov 03 01:49:49.112456 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.112424    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerName="istio-proxy"
Nov 03 01:49:49.112667 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.112652    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7e96865-e9ca-4040-bfe1-27d0c745ca85" containerName="istio-proxy"
Nov 03 01:49:49.130859 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.129980    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"]
Nov 03 01:49:49.130859 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.130095    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"
Nov 03 01:49:49.139416 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.137995    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p68l\" (UniqueName: \"kubernetes.io/projected/33f9264c-091c-4a9a-9f38-e5c5923ebb59-kube-api-access-8p68l\") pod \"payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7\" (UID: \"33f9264c-091c-4a9a-9f38-e5c5923ebb59\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"
Nov 03 01:49:49.196190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.196158    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"
Nov 03 01:49:49.265095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.265057    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-728sh\" (UniqueName: \"kubernetes.io/projected/c53344b7-1435-47fd-9498-cca201315546-kube-api-access-728sh\") pod \"payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj\" (UID: \"c53344b7-1435-47fd-9498-cca201315546\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"
Nov 03 01:49:49.265238 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.265206    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53344b7-1435-47fd-9498-cca201315546-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj\" (UID: \"c53344b7-1435-47fd-9498-cca201315546\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"
Nov 03 01:49:49.366642 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.366605    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-728sh\" (UniqueName: \"kubernetes.io/projected/c53344b7-1435-47fd-9498-cca201315546-kube-api-access-728sh\") pod \"payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj\" (UID: \"c53344b7-1435-47fd-9498-cca201315546\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"
Nov 03 01:49:49.367032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.366721    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53344b7-1435-47fd-9498-cca201315546-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj\" (UID: \"c53344b7-1435-47fd-9498-cca201315546\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"
Nov 03 01:49:49.367434 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.367416    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53344b7-1435-47fd-9498-cca201315546-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj\" (UID: \"c53344b7-1435-47fd-9498-cca201315546\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"
Nov 03 01:49:49.423924 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.423900    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-728sh\" (UniqueName: \"kubernetes.io/projected/c53344b7-1435-47fd-9498-cca201315546-kube-api-access-728sh\") pod \"payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj\" (UID: \"c53344b7-1435-47fd-9498-cca201315546\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"
Nov 03 01:49:49.460246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.460219    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"
Nov 03 01:49:49.574746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.574691    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"]
Nov 03 01:49:49.579423 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:49:49.579395    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f9264c_091c_4a9a_9f38_e5c5923ebb59.slice/crio-fb9502462f94d0e828fef15f3c5debdb5958eb928dc34cb32f8294ca186520de WatchSource:0}: Error finding container fb9502462f94d0e828fef15f3c5debdb5958eb928dc34cb32f8294ca186520de: Status 404 returned error can't find the container with id fb9502462f94d0e828fef15f3c5debdb5958eb928dc34cb32f8294ca186520de
Nov 03 01:49:49.841366 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:49.841253    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"]
Nov 03 01:49:49.847117 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:49:49.847089    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53344b7_1435_47fd_9498_cca201315546.slice/crio-9203070a9e4b12e19ad24a4115a557b915b0abcf1718b5e98e77dc39373812ff WatchSource:0}: Error finding container 9203070a9e4b12e19ad24a4115a557b915b0abcf1718b5e98e77dc39373812ff: Status 404 returned error can't find the container with id 9203070a9e4b12e19ad24a4115a557b915b0abcf1718b5e98e77dc39373812ff
Nov 03 01:49:50.530781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:50.530739    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj" event={"ID":"c53344b7-1435-47fd-9498-cca201315546","Type":"ContainerStarted","Data":"cd5a3ace76b793c93a1dde82e46eb183b0348ddd787b01f83d7af5f4f9069210"}
Nov 03 01:49:50.531191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:50.530807    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"
Nov 03 01:49:50.531191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:50.530834    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj" event={"ID":"c53344b7-1435-47fd-9498-cca201315546","Type":"ContainerStarted","Data":"9203070a9e4b12e19ad24a4115a557b915b0abcf1718b5e98e77dc39373812ff"}
Nov 03 01:49:50.532503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:50.532476    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7" event={"ID":"33f9264c-091c-4a9a-9f38-e5c5923ebb59","Type":"ContainerStarted","Data":"1664b300c9fd35649f828272ac0b7b494d76db3b9658e395d52ce5ffa5743fcb"}
Nov 03 01:49:50.532639 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:50.532522    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7" event={"ID":"33f9264c-091c-4a9a-9f38-e5c5923ebb59","Type":"ContainerStarted","Data":"fb9502462f94d0e828fef15f3c5debdb5958eb928dc34cb32f8294ca186520de"}
Nov 03 01:49:50.532639 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:50.532567    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"
Nov 03 01:49:50.640620 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:50.640574    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj" podStartSLOduration=1.640559074 podStartE2EDuration="1.640559074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:50.639099664 +0000 UTC m=+1870.064706150" watchObservedRunningTime="2026-11-03 01:49:50.640559074 +0000 UTC m=+1870.066165538"
Nov 03 01:49:50.771522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:50.760980    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7" podStartSLOduration=2.760960559 podStartE2EDuration="2.760960559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:50.749301084 +0000 UTC m=+1870.174907577" watchObservedRunningTime="2026-11-03 01:49:50.760960559 +0000 UTC m=+1870.186567024"
Nov 03 01:49:58.945308 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:58.945256    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"]
Nov 03 01:49:58.949129 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:58.949110    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"
Nov 03 01:49:59.042136 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.042106    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm\" (UID: \"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"
Nov 03 01:49:59.042311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.042174    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v8pw\" (UniqueName: \"kubernetes.io/projected/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-kube-api-access-2v8pw\") pod \"payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm\" (UID: \"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"
Nov 03 01:49:59.045123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.045093    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-23e673\""
Nov 03 01:49:59.045123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.045109    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-23e673-dockercfg-9crbz\""
Nov 03 01:49:59.048452 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.048427    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"]
Nov 03 01:49:59.143195 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.143161    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm\" (UID: \"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"
Nov 03 01:49:59.143331 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.143243    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v8pw\" (UniqueName: \"kubernetes.io/projected/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-kube-api-access-2v8pw\") pod \"payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm\" (UID: \"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"
Nov 03 01:49:59.143887 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.143863    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm\" (UID: \"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"
Nov 03 01:49:59.200808 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.200749    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v8pw\" (UniqueName: \"kubernetes.io/projected/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-kube-api-access-2v8pw\") pod \"payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm\" (UID: \"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"
Nov 03 01:49:59.263690 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.263660    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"
Nov 03 01:49:59.655873 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.654675    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"]
Nov 03 01:49:59.659111 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:49:59.659082    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09aad0d_ebb8_4fec_898f_a1e47d1dfd95.slice/crio-722b72136ff68af48d7627ba614bc87185a3828a93fc0cbe977554f0accd0ba5 WatchSource:0}: Error finding container 722b72136ff68af48d7627ba614bc87185a3828a93fc0cbe977554f0accd0ba5: Status 404 returned error can't find the container with id 722b72136ff68af48d7627ba614bc87185a3828a93fc0cbe977554f0accd0ba5
Nov 03 01:49:59.682703 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.682677    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"]
Nov 03 01:49:59.687915 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.687890    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"
Nov 03 01:49:59.752236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.752156    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"]
Nov 03 01:49:59.857983 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.857939    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfbrm\" (UniqueName: \"kubernetes.io/projected/636dd427-8666-46d8-a788-2915dc378d5f-kube-api-access-hfbrm\") pod \"payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7\" (UID: \"636dd427-8666-46d8-a788-2915dc378d5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"
Nov 03 01:49:59.858248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.858222    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636dd427-8666-46d8-a788-2915dc378d5f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7\" (UID: \"636dd427-8666-46d8-a788-2915dc378d5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"
Nov 03 01:49:59.959433 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.959337    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfbrm\" (UniqueName: \"kubernetes.io/projected/636dd427-8666-46d8-a788-2915dc378d5f-kube-api-access-hfbrm\") pod \"payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7\" (UID: \"636dd427-8666-46d8-a788-2915dc378d5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"
Nov 03 01:49:59.959876 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.959481    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636dd427-8666-46d8-a788-2915dc378d5f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7\" (UID: \"636dd427-8666-46d8-a788-2915dc378d5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"
Nov 03 01:49:59.960269 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:49:59.960245    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636dd427-8666-46d8-a788-2915dc378d5f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7\" (UID: \"636dd427-8666-46d8-a788-2915dc378d5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"
Nov 03 01:50:00.016820 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:00.016782    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfbrm\" (UniqueName: \"kubernetes.io/projected/636dd427-8666-46d8-a788-2915dc378d5f-kube-api-access-hfbrm\") pod \"payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7\" (UID: \"636dd427-8666-46d8-a788-2915dc378d5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"
Nov 03 01:50:00.304757 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:00.304684    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"
Nov 03 01:50:00.630879 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:00.630795    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm" event={"ID":"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95","Type":"ContainerStarted","Data":"722b72136ff68af48d7627ba614bc87185a3828a93fc0cbe977554f0accd0ba5"}
Nov 03 01:50:00.738311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:00.738227    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"]
Nov 03 01:50:00.751545 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:50:00.751516    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636dd427_8666_46d8_a788_2915dc378d5f.slice/crio-e8fd656cb4b0a10c370101965765677a0bfcbf3ee333724ce359f01d65208901 WatchSource:0}: Error finding container e8fd656cb4b0a10c370101965765677a0bfcbf3ee333724ce359f01d65208901: Status 404 returned error can't find the container with id e8fd656cb4b0a10c370101965765677a0bfcbf3ee333724ce359f01d65208901
Nov 03 01:50:01.544142 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:01.544113    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"
Nov 03 01:50:01.544604 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:01.544213    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"
Nov 03 01:50:01.640650 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:01.640613    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7" event={"ID":"636dd427-8666-46d8-a788-2915dc378d5f","Type":"ContainerStarted","Data":"e8fd656cb4b0a10c370101965765677a0bfcbf3ee333724ce359f01d65208901"}
Nov 03 01:50:03.676997 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:03.676959    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7" event={"ID":"636dd427-8666-46d8-a788-2915dc378d5f","Type":"ContainerStarted","Data":"883d0ad8d3388728a51a17d74b655d1596be3add55a6a1af2bf7ecbc26cce7e9"}
Nov 03 01:50:03.677690 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:03.677664    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"
Nov 03 01:50:03.679946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:03.679913    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm" event={"ID":"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95","Type":"ContainerStarted","Data":"7e0087e5b146cfca428c2605bba7f941824b01708664a4f924eb051ed65ee8a6"}
Nov 03 01:50:03.680773 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:03.680753    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"
Nov 03 01:50:03.810558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:03.810489    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7" podStartSLOduration=2.6992533930000002 podStartE2EDuration="4.81045861s" totalImagesPullingTime="2.111205217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:03.809629513 +0000 UTC m=+1883.235235983" watchObservedRunningTime="2026-11-03 01:50:03.81045861 +0000 UTC m=+1883.236065078"
Nov 03 01:50:03.924949 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:03.924901    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm" podStartSLOduration=2.896980916 podStartE2EDuration="5.924886912s" totalImagesPullingTime="3.027905996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:03.921143591 +0000 UTC m=+1883.346750054" watchObservedRunningTime="2026-11-03 01:50:03.924886912 +0000 UTC m=+1883.350493408"
Nov 03 01:50:13.832518 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:50:13.830096    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:50:14.694566 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:14.694537    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"
Nov 03 01:50:14.824000 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:14.823959    5508 generic.go:356] "Generic (PLEG): container finished" podID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerID="14b47e8eb3d0fd8d995c16f1a6bf4e130df23edd792531c29e8639db07f07dd8" exitCode=0
Nov 03 01:50:14.824163 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:14.824072    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp" event={"ID":"a37e873e-8f3e-4e1b-a409-ca3fe4192728","Type":"ContainerDied","Data":"14b47e8eb3d0fd8d995c16f1a6bf4e130df23edd792531c29e8639db07f07dd8"}
Nov 03 01:50:14.824591 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:14.824569    5508 scope.go:122] "RemoveContainer" containerID="14b47e8eb3d0fd8d995c16f1a6bf4e130df23edd792531c29e8639db07f07dd8"
Nov 03 01:50:14.826496 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:14.826457    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:50:14.826610 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:14.826568    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:50:14.826675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:14.826620    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:50:15.239449 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.239409    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"]
Nov 03 01:50:15.243625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.243601    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"
Nov 03 01:50:15.293022 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.292989    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7bcbbb-dockercfg-bcxz5\""
Nov 03 01:50:15.293242 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.293000    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7bcbbb\""
Nov 03 01:50:15.298316 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.298290    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"]
Nov 03 01:50:15.383374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.376326    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t\" (UID: \"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"
Nov 03 01:50:15.384552 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.384523    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9rbf\" (UniqueName: \"kubernetes.io/projected/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-kube-api-access-z9rbf\") pod \"payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t\" (UID: \"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"
Nov 03 01:50:15.485391 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.485339    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t\" (UID: \"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"
Nov 03 01:50:15.485604 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.485463    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9rbf\" (UniqueName: \"kubernetes.io/projected/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-kube-api-access-z9rbf\") pod \"payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t\" (UID: \"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"
Nov 03 01:50:15.486120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.486095    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t\" (UID: \"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"
Nov 03 01:50:15.543290 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.543234    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9rbf\" (UniqueName: \"kubernetes.io/projected/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-kube-api-access-z9rbf\") pod \"payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t\" (UID: \"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"
Nov 03 01:50:15.563237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.563203    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"
Nov 03 01:50:15.699788 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.698538    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"
Nov 03 01:50:15.836715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.836640    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp" event={"ID":"a37e873e-8f3e-4e1b-a409-ca3fe4192728","Type":"ContainerStarted","Data":"fab8639e2fce1d37fff92cf0567b0e8802aaed56f9f8a71e1f705b86c23c14a1"}
Nov 03 01:50:15.948134 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:15.948062    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"]
Nov 03 01:50:15.979949 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:50:15.979922    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b7e5cf_d7cc_4ada_88d1_ba5a6a2db98d.slice/crio-a5904ec9d18c534805213860dd05fd0e5e6550255ba7fb9e39899567163d18ef WatchSource:0}: Error finding container a5904ec9d18c534805213860dd05fd0e5e6550255ba7fb9e39899567163d18ef: Status 404 returned error can't find the container with id a5904ec9d18c534805213860dd05fd0e5e6550255ba7fb9e39899567163d18ef
Nov 03 01:50:16.079860 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.079823    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"]
Nov 03 01:50:16.083944 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.083915    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"
Nov 03 01:50:16.117012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.116988    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"]
Nov 03 01:50:16.192952 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.192920    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2hxd\" (UniqueName: \"kubernetes.io/projected/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-kube-api-access-z2hxd\") pod \"payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp\" (UID: \"31af0199-a45d-4302-ac71-7e6b9e7ed8b0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"
Nov 03 01:50:16.193124 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.193092    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp\" (UID: \"31af0199-a45d-4302-ac71-7e6b9e7ed8b0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"
Nov 03 01:50:16.294789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.294751    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2hxd\" (UniqueName: \"kubernetes.io/projected/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-kube-api-access-z2hxd\") pod \"payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp\" (UID: \"31af0199-a45d-4302-ac71-7e6b9e7ed8b0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"
Nov 03 01:50:16.295204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.294934    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp\" (UID: \"31af0199-a45d-4302-ac71-7e6b9e7ed8b0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"
Nov 03 01:50:16.295828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.295806    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp\" (UID: \"31af0199-a45d-4302-ac71-7e6b9e7ed8b0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"
Nov 03 01:50:16.363414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.360761    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2hxd\" (UniqueName: \"kubernetes.io/projected/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-kube-api-access-z2hxd\") pod \"payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp\" (UID: \"31af0199-a45d-4302-ac71-7e6b9e7ed8b0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"
Nov 03 01:50:16.398290 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.398258    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"
Nov 03 01:50:16.683126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.683076    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:50:16.685384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.685336    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 03 01:50:16.685549 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.685408    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp" podUID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 03 01:50:16.815554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.815526    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"]
Nov 03 01:50:16.821447 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:50:16.821420    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31af0199_a45d_4302_ac71_7e6b9e7ed8b0.slice/crio-b8601b5419e00aadcd98bf0d8bfb67bbf9b137308983df5ac57bccac912cfbf7 WatchSource:0}: Error finding container b8601b5419e00aadcd98bf0d8bfb67bbf9b137308983df5ac57bccac912cfbf7: Status 404 returned error can't find the container with id b8601b5419e00aadcd98bf0d8bfb67bbf9b137308983df5ac57bccac912cfbf7
Nov 03 01:50:16.937522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.937443    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t" event={"ID":"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d","Type":"ContainerStarted","Data":"f9c0f29fd459ac07e442f9e570362e58b5d51f492c62e033e9cc06b258d9dcc8"}
Nov 03 01:50:16.937522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.937498    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t" event={"ID":"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d","Type":"ContainerStarted","Data":"a5904ec9d18c534805213860dd05fd0e5e6550255ba7fb9e39899567163d18ef"}
Nov 03 01:50:16.938440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.938419    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"
Nov 03 01:50:16.939533 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:16.939512    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp" event={"ID":"31af0199-a45d-4302-ac71-7e6b9e7ed8b0","Type":"ContainerStarted","Data":"b8601b5419e00aadcd98bf0d8bfb67bbf9b137308983df5ac57bccac912cfbf7"}
Nov 03 01:50:17.069589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:17.069524    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t" podStartSLOduration=2.069492107 podStartE2EDuration="2.069492107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:17.048439176 +0000 UTC m=+1896.474045640" watchObservedRunningTime="2026-11-03 01:50:17.069492107 +0000 UTC m=+1896.495098572"
Nov 03 01:50:17.669370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:17.668705    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 03 01:50:17.669370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:17.668764    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp" podUID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 03 01:50:17.948037 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:17.947941    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp" event={"ID":"31af0199-a45d-4302-ac71-7e6b9e7ed8b0","Type":"ContainerStarted","Data":"f181a77f0fce18ad8106bd4a0e34c20d96bea612158fa89bf08f0ffcd19be728"}
Nov 03 01:50:17.948191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:17.948174    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"
Nov 03 01:50:18.669370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:18.668662    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 03 01:50:18.669370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:18.668717    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp" podUID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 03 01:50:19.668771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:19.668735    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 03 01:50:19.669062 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:19.669020    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp" podUID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 03 01:50:20.649373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:20.649322    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:50:20.668861 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:20.668833    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 03 01:50:20.669036 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:20.668883    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp" podUID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 03 01:50:21.678907 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:21.678871    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:50:21.682076 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:21.682037    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:50:21.905372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:21.904535    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp" podStartSLOduration=5.904513319 podStartE2EDuration="5.904513319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:18.05960517 +0000 UTC m=+1897.485211662" watchObservedRunningTime="2026-11-03 01:50:21.904513319 +0000 UTC m=+1901.330119784"
Nov 03 01:50:24.422059 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:24.422015    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:50:27.951185 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:27.951151    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"
Nov 03 01:50:28.985883 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:28.985845    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"
Nov 03 01:50:43.812091 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:50:43.812049    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:50:44.468626 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:44.468594    5508 scope.go:122] "RemoveContainer" containerID="baf9b204e6c2fa1fd7284a0c596dd804c838a449dfc4f8cd67ace1b6d5f3b31c"
Nov 03 01:50:44.480589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:44.480566    5508 scope.go:122] "RemoveContainer" containerID="08a9faa55a47add4022d5fea90f45284afe6b55c3c0064ce50393afc78e710f2"
Nov 03 01:50:45.360271 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:45.360236    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"]
Nov 03 01:50:45.465482 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:45.465446    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"]
Nov 03 01:50:49.018972 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:49.018941    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:50:50.361321 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.361274    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm" podUID="28ab21b5-0596-46ff-9309-47373a0094ca" containerName="payload-processing" containerID="cri-o://7dc60be6b553ffac7e2b5e99c1617ab49abfefae6ce8c0c9f56d8f0e21f983c1" gracePeriod=25
Nov 03 01:50:50.466887 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.466809    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l" podUID="aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92" containerName="payload-pre-processing" containerID="cri-o://7ba8b72a023ab4b9b2bcd883cc69d23434d259004a598ca6a1ff7c514e729879" gracePeriod=25
Nov 03 01:50:50.676084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.675098    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"
Nov 03 01:50:50.747753 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.747722    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"
Nov 03 01:50:50.795888 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.795849    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-kube-api-access-gt9bp\" (UniqueName: \"kubernetes.io/projected/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-kube-api-access-gt9bp\") pod \"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92\" (UID: \"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92\") "
Nov 03 01:50:50.796053 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.795971    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ab21b5-0596-46ff-9309-47373a0094ca-kube-api-access-blvqv\" (UniqueName: \"kubernetes.io/projected/28ab21b5-0596-46ff-9309-47373a0094ca-kube-api-access-blvqv\") pod \"28ab21b5-0596-46ff-9309-47373a0094ca\" (UID: \"28ab21b5-0596-46ff-9309-47373a0094ca\") "
Nov 03 01:50:50.796053 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.796006    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ab21b5-0596-46ff-9309-47373a0094ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ab21b5-0596-46ff-9309-47373a0094ca-plugins-config-volume\") pod \"28ab21b5-0596-46ff-9309-47373a0094ca\" (UID: \"28ab21b5-0596-46ff-9309-47373a0094ca\") "
Nov 03 01:50:50.796053 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.796036    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-plugins-config-volume\") pod \"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92\" (UID: \"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92\") "
Nov 03 01:50:50.797758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.797728    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-plugins-config-volume" pod "aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92" (UID: "aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:50:50.797899 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.797844    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ab21b5-0596-46ff-9309-47373a0094ca-plugins-config-volume" pod "28ab21b5-0596-46ff-9309-47373a0094ca" (UID: "28ab21b5-0596-46ff-9309-47373a0094ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:50:50.799171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.799143    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-kube-api-access-gt9bp" pod "aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92" (UID: "aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92"). InnerVolumeSpecName "kube-api-access-gt9bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:50:50.799388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.799369    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ab21b5-0596-46ff-9309-47373a0094ca-kube-api-access-blvqv" pod "28ab21b5-0596-46ff-9309-47373a0094ca" (UID: "28ab21b5-0596-46ff-9309-47373a0094ca"). InnerVolumeSpecName "kube-api-access-blvqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:50:50.898063 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.898032    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28ab21b5-0596-46ff-9309-47373a0094ca-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:50:50.898063 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.898060    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:50:50.898310 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.898070    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt9bp\" (UniqueName: \"kubernetes.io/projected/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92-kube-api-access-gt9bp\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:50:50.898310 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:50.898080    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blvqv\" (UniqueName: \"kubernetes.io/projected/28ab21b5-0596-46ff-9309-47373a0094ca-kube-api-access-blvqv\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:50:51.282574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.282471    5508 generic.go:356] "Generic (PLEG): container finished" podID="aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92" containerID="7ba8b72a023ab4b9b2bcd883cc69d23434d259004a598ca6a1ff7c514e729879" exitCode=0
Nov 03 01:50:51.282574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.282528    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"
Nov 03 01:50:51.282818 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.282613    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l" event={"ID":"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92","Type":"ContainerDied","Data":"7ba8b72a023ab4b9b2bcd883cc69d23434d259004a598ca6a1ff7c514e729879"}
Nov 03 01:50:51.282818 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.282688    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l" event={"ID":"aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92","Type":"ContainerDied","Data":"08d7e47b33a23fe06ca64f6fc80de0bf3b97bdaf0a116b10a6dbbd7831feec1d"}
Nov 03 01:50:51.282818 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.282716    5508 scope.go:122] "RemoveContainer" containerID="7ba8b72a023ab4b9b2bcd883cc69d23434d259004a598ca6a1ff7c514e729879"
Nov 03 01:50:51.286008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.285332    5508 generic.go:356] "Generic (PLEG): container finished" podID="28ab21b5-0596-46ff-9309-47373a0094ca" containerID="7dc60be6b553ffac7e2b5e99c1617ab49abfefae6ce8c0c9f56d8f0e21f983c1" exitCode=0
Nov 03 01:50:51.286008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.285417    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"
Nov 03 01:50:51.286008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.285531    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm" event={"ID":"28ab21b5-0596-46ff-9309-47373a0094ca","Type":"ContainerDied","Data":"7dc60be6b553ffac7e2b5e99c1617ab49abfefae6ce8c0c9f56d8f0e21f983c1"}
Nov 03 01:50:51.286008 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.285573    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm" event={"ID":"28ab21b5-0596-46ff-9309-47373a0094ca","Type":"ContainerDied","Data":"9416381b1e79389e2e1a6f930ed8ce2c165faea06da089a2045430181f2e1c07"}
Nov 03 01:50:51.295867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.295850    5508 scope.go:122] "RemoveContainer" containerID="7ba8b72a023ab4b9b2bcd883cc69d23434d259004a598ca6a1ff7c514e729879"
Nov 03 01:50:51.296150 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:50:51.296122    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba8b72a023ab4b9b2bcd883cc69d23434d259004a598ca6a1ff7c514e729879\": container with ID starting with 7ba8b72a023ab4b9b2bcd883cc69d23434d259004a598ca6a1ff7c514e729879 not found: ID does not exist" containerID="7ba8b72a023ab4b9b2bcd883cc69d23434d259004a598ca6a1ff7c514e729879"
Nov 03 01:50:51.296257 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.296159    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba8b72a023ab4b9b2bcd883cc69d23434d259004a598ca6a1ff7c514e729879"} err="failed to get container status \"7ba8b72a023ab4b9b2bcd883cc69d23434d259004a598ca6a1ff7c514e729879\": rpc error: code = NotFound desc = could not find container \"7ba8b72a023ab4b9b2bcd883cc69d23434d259004a598ca6a1ff7c514e729879\": container with ID starting with 7ba8b72a023ab4b9b2bcd883cc69d23434d259004a598ca6a1ff7c514e729879 not found: ID does not exist"
Nov 03 01:50:51.296257 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.296180    5508 scope.go:122] "RemoveContainer" containerID="7dc60be6b553ffac7e2b5e99c1617ab49abfefae6ce8c0c9f56d8f0e21f983c1"
Nov 03 01:50:51.306205 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.306185    5508 scope.go:122] "RemoveContainer" containerID="7dc60be6b553ffac7e2b5e99c1617ab49abfefae6ce8c0c9f56d8f0e21f983c1"
Nov 03 01:50:51.306500 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:50:51.306481    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc60be6b553ffac7e2b5e99c1617ab49abfefae6ce8c0c9f56d8f0e21f983c1\": container with ID starting with 7dc60be6b553ffac7e2b5e99c1617ab49abfefae6ce8c0c9f56d8f0e21f983c1 not found: ID does not exist" containerID="7dc60be6b553ffac7e2b5e99c1617ab49abfefae6ce8c0c9f56d8f0e21f983c1"
Nov 03 01:50:51.306572 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.306507    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc60be6b553ffac7e2b5e99c1617ab49abfefae6ce8c0c9f56d8f0e21f983c1"} err="failed to get container status \"7dc60be6b553ffac7e2b5e99c1617ab49abfefae6ce8c0c9f56d8f0e21f983c1\": rpc error: code = NotFound desc = could not find container \"7dc60be6b553ffac7e2b5e99c1617ab49abfefae6ce8c0c9f56d8f0e21f983c1\": container with ID starting with 7dc60be6b553ffac7e2b5e99c1617ab49abfefae6ce8c0c9f56d8f0e21f983c1 not found: ID does not exist"
Nov 03 01:50:51.438977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.438944    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"]
Nov 03 01:50:51.441879 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.441858    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5f68589669-ddbhm"]
Nov 03 01:50:51.543755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.543683    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"]
Nov 03 01:50:51.575628 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:51.575591    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-6cdd6f8d55-m7w6l"]
Nov 03 01:50:52.694387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:52.694339    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ab21b5-0596-46ff-9309-47373a0094ca" path="/var/lib/kubelet/pods/28ab21b5-0596-46ff-9309-47373a0094ca/volumes"
Nov 03 01:50:52.694798 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:52.694784    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92" path="/var/lib/kubelet/pods/aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92/volumes"
Nov 03 01:50:59.612680 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:59.611929    5508 patch_prober.go:29] interesting pod/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": read tcp 10.133.0.2:40192->10.133.0.172:15021: read: connection reset by peer" start-of-body=
Nov 03 01:50:59.612680 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:50:59.612025    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": read tcp 10.133.0.2:40192->10.133.0.172:15021: read: connection reset by peer"
Nov 03 01:51:00.356975 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:00.356940    5508 generic.go:356] "Generic (PLEG): container finished" podID="d24d34b3-8841-4389-b20f-297008aa92a5" containerID="48682b93a6ccc6d2a688417ccb7fe4ffcf89f32dc54845b74ee24ff96ba72fd6" exitCode=0
Nov 03 01:51:00.357188 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:00.357002    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" event={"ID":"d24d34b3-8841-4389-b20f-297008aa92a5","Type":"ContainerDied","Data":"48682b93a6ccc6d2a688417ccb7fe4ffcf89f32dc54845b74ee24ff96ba72fd6"}
Nov 03 01:51:00.357523 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:00.357502    5508 scope.go:122] "RemoveContainer" containerID="48682b93a6ccc6d2a688417ccb7fe4ffcf89f32dc54845b74ee24ff96ba72fd6"
Nov 03 01:51:00.359249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:00.359213    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:51:00.359329 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:00.359301    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:51:00.359399 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:00.359363    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:51:00.605168 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:00.605133    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"]
Nov 03 01:51:00.605416 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:00.605391    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy" containerID="cri-o://671323f45e8d45b816114fc29f4aedcec34269068b5523edf5e4e1fe6cadc2fa" gracePeriod=30
Nov 03 01:51:01.367441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:01.367404    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" event={"ID":"d24d34b3-8841-4389-b20f-297008aa92a5","Type":"ContainerStarted","Data":"1fe97b5d9a8a06a54a19ba786088abae401b38b2aaa8eba7b4633c2e670bb0d9"}
Nov 03 01:51:01.983427 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:01.983380    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:51:01.983646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:01.983589    5508 patch_prober.go:29] interesting pod/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 01:51:01.983646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:01.983630    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 01:51:02.983728 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:02.983690    5508 patch_prober.go:29] interesting pod/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 01:51:02.984146 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:02.983753    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 01:51:03.983948 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:03.983904    5508 patch_prober.go:29] interesting pod/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 01:51:03.984474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:03.983985    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 01:51:04.983884 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:04.983848    5508 patch_prober.go:29] interesting pod/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 01:51:04.984099 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:04.983896    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 01:51:05.948156 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.948133    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:51:05.983841 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.983812    5508 patch_prober.go:29] interesting pod/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 01:51:05.983993 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.983859    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 01:51:05.996781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.995437    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-token\" (UniqueName: \"kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-token\") pod \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") "
Nov 03 01:51:05.996781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.995472    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-socket\") pod \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") "
Nov 03 01:51:05.996781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.995525    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-certs\") pod \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") "
Nov 03 01:51:05.996781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.995587    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-envoy\") pod \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") "
Nov 03 01:51:05.996781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.995653    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-credential-socket\") pod \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") "
Nov 03 01:51:05.996781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.995760    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-podinfo\") pod \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") "
Nov 03 01:51:05.996781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.995800    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-kube-api-access-2wh89\" (UniqueName: \"kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-kube-api-access-2wh89\") pod \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") "
Nov 03 01:51:05.996781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.995853    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-data\") pod \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") "
Nov 03 01:51:05.996781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.995902    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istiod-ca-cert\") pod \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\" (UID: \"c2ddd202-6ce9-48df-b027-86ac0620f0c8\") "
Nov 03 01:51:05.996781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.996729    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-data" pod "c2ddd202-6ce9-48df-b027-86ac0620f0c8" (UID: "c2ddd202-6ce9-48df-b027-86ac0620f0c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:51:05.996781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.996726    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-credential-socket" pod "c2ddd202-6ce9-48df-b027-86ac0620f0c8" (UID: "c2ddd202-6ce9-48df-b027-86ac0620f0c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:51:05.997864 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.996680    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-socket" pod "c2ddd202-6ce9-48df-b027-86ac0620f0c8" (UID: "c2ddd202-6ce9-48df-b027-86ac0620f0c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:51:05.997864 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.996910    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-certs" pod "c2ddd202-6ce9-48df-b027-86ac0620f0c8" (UID: "c2ddd202-6ce9-48df-b027-86ac0620f0c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:51:05.997864 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.997160    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istiod-ca-cert" pod "c2ddd202-6ce9-48df-b027-86ac0620f0c8" (UID: "c2ddd202-6ce9-48df-b027-86ac0620f0c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:51:05.999531 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.999505    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-envoy" pod "c2ddd202-6ce9-48df-b027-86ac0620f0c8" (UID: "c2ddd202-6ce9-48df-b027-86ac0620f0c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:51:05.999636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:05.999534    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-token" pod "c2ddd202-6ce9-48df-b027-86ac0620f0c8" (UID: "c2ddd202-6ce9-48df-b027-86ac0620f0c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:51:06.000155 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.000133    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-podinfo" pod "c2ddd202-6ce9-48df-b027-86ac0620f0c8" (UID: "c2ddd202-6ce9-48df-b027-86ac0620f0c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:51:06.003516 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.003491    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-kube-api-access-2wh89" pod "c2ddd202-6ce9-48df-b027-86ac0620f0c8" (UID: "c2ddd202-6ce9-48df-b027-86ac0620f0c8"). InnerVolumeSpecName "kube-api-access-2wh89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:51:06.099833 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.099799    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:51:06.099833 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.099828    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:51:06.099833 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.099837    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:51:06.100069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.099845    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:51:06.100069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.099854    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:51:06.100069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.099862    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:51:06.100069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.099870    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wh89\" (UniqueName: \"kubernetes.io/projected/c2ddd202-6ce9-48df-b027-86ac0620f0c8-kube-api-access-2wh89\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:51:06.100069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.099879    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:51:06.100069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.099887    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2ddd202-6ce9-48df-b027-86ac0620f0c8-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:51:06.417961 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.417925    5508 generic.go:356] "Generic (PLEG): container finished" podID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerID="671323f45e8d45b816114fc29f4aedcec34269068b5523edf5e4e1fe6cadc2fa" exitCode=0
Nov 03 01:51:06.417961 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.417966    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" event={"ID":"c2ddd202-6ce9-48df-b027-86ac0620f0c8","Type":"ContainerDied","Data":"671323f45e8d45b816114fc29f4aedcec34269068b5523edf5e4e1fe6cadc2fa"}
Nov 03 01:51:06.418189 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.417991    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"
Nov 03 01:51:06.418189 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.418004    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92" event={"ID":"c2ddd202-6ce9-48df-b027-86ac0620f0c8","Type":"ContainerDied","Data":"2a96336ad235ab1c25bca49b27008cc7ae11db660413d2015f64ff27093064ed"}
Nov 03 01:51:06.418189 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.418026    5508 scope.go:122] "RemoveContainer" containerID="671323f45e8d45b816114fc29f4aedcec34269068b5523edf5e4e1fe6cadc2fa"
Nov 03 01:51:06.433819 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.433794    5508 scope.go:122] "RemoveContainer" containerID="671323f45e8d45b816114fc29f4aedcec34269068b5523edf5e4e1fe6cadc2fa"
Nov 03 01:51:06.434450 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:51:06.434425    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671323f45e8d45b816114fc29f4aedcec34269068b5523edf5e4e1fe6cadc2fa\": container with ID starting with 671323f45e8d45b816114fc29f4aedcec34269068b5523edf5e4e1fe6cadc2fa not found: ID does not exist" containerID="671323f45e8d45b816114fc29f4aedcec34269068b5523edf5e4e1fe6cadc2fa"
Nov 03 01:51:06.434611 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.434585    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671323f45e8d45b816114fc29f4aedcec34269068b5523edf5e4e1fe6cadc2fa"} err="failed to get container status \"671323f45e8d45b816114fc29f4aedcec34269068b5523edf5e4e1fe6cadc2fa\": rpc error: code = NotFound desc = could not find container \"671323f45e8d45b816114fc29f4aedcec34269068b5523edf5e4e1fe6cadc2fa\": container with ID starting with 671323f45e8d45b816114fc29f4aedcec34269068b5523edf5e4e1fe6cadc2fa not found: ID does not exist"
Nov 03 01:51:06.595967 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.595937    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"]
Nov 03 01:51:06.642393 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.642359    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-znf92"]
Nov 03 01:51:06.695317 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:06.695226    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" path="/var/lib/kubelet/pods/c2ddd202-6ce9-48df-b027-86ac0620f0c8/volumes"
Nov 03 01:51:07.047576 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:07.047482    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:51:07.048053 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:07.047931    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:51:07.049402 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:07.049378    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:51:13.811480 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:51:13.811438    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:51:19.986665 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:19.986629    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"]
Nov 03 01:51:19.987210 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:19.987194    5508 cpu_manager.go:414] "containerMap: removing container" podUID="aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92" containerName="payload-pre-processing"
Nov 03 01:51:19.987258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:19.987211    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92" containerName="payload-pre-processing"
Nov 03 01:51:19.987258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:19.987244    5508 cpu_manager.go:414] "containerMap: removing container" podUID="28ab21b5-0596-46ff-9309-47373a0094ca" containerName="payload-processing"
Nov 03 01:51:19.987258 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:19.987250    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ab21b5-0596-46ff-9309-47373a0094ca" containerName="payload-processing"
Nov 03 01:51:19.987386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:19.987264    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy"
Nov 03 01:51:19.987386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:19.987269    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy"
Nov 03 01:51:19.987386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:19.987368    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ddd202-6ce9-48df-b027-86ac0620f0c8" containerName="istio-proxy"
Nov 03 01:51:19.987386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:19.987379    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa1b6c1a-1b87-4b0c-b8dc-2510f4522c92" containerName="payload-pre-processing"
Nov 03 01:51:19.987519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:19.987390    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ab21b5-0596-46ff-9309-47373a0094ca" containerName="payload-processing"
Nov 03 01:51:19.992475 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:19.992456    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.041792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.041761    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0e0d61-openshift-default-dockercfg-9j8xv\""
Nov 03 01:51:20.092060 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.092025    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063d1af5-8267-423a-be4b-a46a27c2e527-istio-podinfo\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.100813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.092290    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"]
Nov 03 01:51:20.100953 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.100819    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqlr5\" (UniqueName: \"kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-kube-api-access-wqlr5\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.100953 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.100857    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-istio-token\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.101034 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.100992    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-credential-socket\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.101079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.101042    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-envoy\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.101122 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.101094    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063d1af5-8267-423a-be4b-a46a27c2e527-istiod-ca-cert\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.101217 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.101199    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-certs\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.101285 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.101254    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-data\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.101480 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.101460    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-socket\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.204434 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.204398    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-socket\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.204629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.204485    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063d1af5-8267-423a-be4b-a46a27c2e527-istio-podinfo\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.204629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.204571    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqlr5\" (UniqueName: \"kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-kube-api-access-wqlr5\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.204629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.204598    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-istio-token\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.204846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.204646    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-credential-socket\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.204846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.204689    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-envoy\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.204846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.204723    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063d1af5-8267-423a-be4b-a46a27c2e527-istiod-ca-cert\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.204846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.204755    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-certs\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.204846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.204795    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-data\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.205113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.204895    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-socket\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.205221 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.205199    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-data\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.205289 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.205218    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-credential-socket\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.206000 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.205923    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-certs\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.207125 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.207100    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063d1af5-8267-423a-be4b-a46a27c2e527-istiod-ca-cert\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.208149 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.208128    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-envoy\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.209807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.209782    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063d1af5-8267-423a-be4b-a46a27c2e527-istio-podinfo\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.262398 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.262326    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-istio-token\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.262729 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.262709    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqlr5\" (UniqueName: \"kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-kube-api-access-wqlr5\") pod \"e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") " pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.305761 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.305740    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:20.742113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.742080    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"]
Nov 03 01:51:20.748520 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:51:20.748477    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063d1af5_8267_423a_be4b_a46a27c2e527.slice/crio-b087583d56b209496f08454dd6c30d20ce6ae6ade9ab76ba353664211c6dc454 WatchSource:0}: Error finding container b087583d56b209496f08454dd6c30d20ce6ae6ade9ab76ba353664211c6dc454: Status 404 returned error can't find the container with id b087583d56b209496f08454dd6c30d20ce6ae6ade9ab76ba353664211c6dc454
Nov 03 01:51:20.751513 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.751477    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:51:20.751581 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.751560    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:51:20.751629 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:20.751602    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:51:21.542735 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:21.542698    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv" event={"ID":"063d1af5-8267-423a-be4b-a46a27c2e527","Type":"ContainerStarted","Data":"87f7c728154202051c6b1aa4e67e4b70baaa270305ded414b8430ad15ae076d5"}
Nov 03 01:51:21.543124 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:21.542755    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv" event={"ID":"063d1af5-8267-423a-be4b-a46a27c2e527","Type":"ContainerStarted","Data":"b087583d56b209496f08454dd6c30d20ce6ae6ade9ab76ba353664211c6dc454"}
Nov 03 01:51:21.702052 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:21.701658    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv" podStartSLOduration=2.701641377 podStartE2EDuration="2.701641377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:21.700825776 +0000 UTC m=+1961.126432239" watchObservedRunningTime="2026-11-03 01:51:21.701641377 +0000 UTC m=+1961.127247843"
Nov 03 01:51:22.306323 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:22.306295    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:22.311944 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:22.311920    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:22.549232 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:22.549203    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:22.550082 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:22.550065    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:51:22.774170 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:22.774142    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:51:28.304205 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:28.304167    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"]
Nov 03 01:51:28.310657 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:28.310631    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"
Nov 03 01:51:28.333149 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:28.333101    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"]
Nov 03 01:51:28.373250 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:28.373211    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0e0d61-dockercfg-vvbls\""
Nov 03 01:51:28.373250 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:28.373213    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0e0d61\""
Nov 03 01:51:28.408882 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:28.408847    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9c7h\" (UniqueName: \"kubernetes.io/projected/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-kube-api-access-r9c7h\") pod \"payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w\" (UID: \"98db450f-aa09-4dc2-9166-2a9d1e8f76ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"
Nov 03 01:51:28.409120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:28.409094    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w\" (UID: \"98db450f-aa09-4dc2-9166-2a9d1e8f76ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"
Nov 03 01:51:28.511515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:28.511477    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9c7h\" (UniqueName: \"kubernetes.io/projected/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-kube-api-access-r9c7h\") pod \"payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w\" (UID: \"98db450f-aa09-4dc2-9166-2a9d1e8f76ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"
Nov 03 01:51:28.511730 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:28.511641    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w\" (UID: \"98db450f-aa09-4dc2-9166-2a9d1e8f76ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"
Nov 03 01:51:28.512853 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:28.512823    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w\" (UID: \"98db450f-aa09-4dc2-9166-2a9d1e8f76ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"
Nov 03 01:51:28.569715 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:28.569648    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9c7h\" (UniqueName: \"kubernetes.io/projected/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-kube-api-access-r9c7h\") pod \"payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w\" (UID: \"98db450f-aa09-4dc2-9166-2a9d1e8f76ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"
Nov 03 01:51:28.648663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:28.648633    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"
Nov 03 01:51:29.040015 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:29.039980    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"]
Nov 03 01:51:29.046375 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:51:29.046332    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98db450f_aa09_4dc2_9166_2a9d1e8f76ad.slice/crio-ccbd458c51bc2a8689f8d1dfce6a1c01f6fec821592e1fc8ef943cc93b81ce75 WatchSource:0}: Error finding container ccbd458c51bc2a8689f8d1dfce6a1c01f6fec821592e1fc8ef943cc93b81ce75: Status 404 returned error can't find the container with id ccbd458c51bc2a8689f8d1dfce6a1c01f6fec821592e1fc8ef943cc93b81ce75
Nov 03 01:51:29.596784 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:29.596747    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w" event={"ID":"98db450f-aa09-4dc2-9166-2a9d1e8f76ad","Type":"ContainerStarted","Data":"e4a8a46010fc8857e69f67c048783117caf90dc96d84216d4ebf44efb6a55fba"}
Nov 03 01:51:29.597202 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:29.596843    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"
Nov 03 01:51:29.597202 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:29.596870    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w" event={"ID":"98db450f-aa09-4dc2-9166-2a9d1e8f76ad","Type":"ContainerStarted","Data":"ccbd458c51bc2a8689f8d1dfce6a1c01f6fec821592e1fc8ef943cc93b81ce75"}
Nov 03 01:51:29.746759 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:29.746712    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w" podStartSLOduration=1.746697465 podStartE2EDuration="1.746697465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:29.714741602 +0000 UTC m=+1969.140348067" watchObservedRunningTime="2026-11-03 01:51:29.746697465 +0000 UTC m=+1969.172303928"
Nov 03 01:51:33.045513 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:33.045479    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"]
Nov 03 01:51:33.091113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:33.091087    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca6bf3f1-lgdr4"]
Nov 03 01:51:34.692187 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:34.692157    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="109c5dc1-dc70-4db4-9563-65f27bc8e6b7" path="/var/lib/kubelet/pods/109c5dc1-dc70-4db4-9563-65f27bc8e6b7/volumes"
Nov 03 01:51:40.607523 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:40.607490    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"
Nov 03 01:51:43.812069 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:51:43.812033    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:51:44.369898 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:44.369863    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"]
Nov 03 01:51:44.472205 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:44.472171    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"]
Nov 03 01:51:49.370520 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.370372    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm" podUID="327f2e67-8150-401c-8f37-08bb6966384b" containerName="payload-processing" containerID="cri-o://dc1293fb37276f7a1e2f36ee829022f179aa4f29800833899dbb8b14fced128d" gracePeriod=25
Nov 03 01:51:49.477703 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.477435    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9" podUID="1172569a-2f6a-4327-8f03-c65d14c4a678" containerName="payload-pre-processing" containerID="cri-o://26ee5d1c80ca9f18b5a4e5759e18704781f9286d25ea47b8aa1d2c6ff159bc8b" gracePeriod=25
Nov 03 01:51:49.839608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.839578    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"
Nov 03 01:51:49.850787 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.850754    5508 generic.go:356] "Generic (PLEG): container finished" podID="1172569a-2f6a-4327-8f03-c65d14c4a678" containerID="26ee5d1c80ca9f18b5a4e5759e18704781f9286d25ea47b8aa1d2c6ff159bc8b" exitCode=0
Nov 03 01:51:49.850977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.850866    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9" event={"ID":"1172569a-2f6a-4327-8f03-c65d14c4a678","Type":"ContainerDied","Data":"26ee5d1c80ca9f18b5a4e5759e18704781f9286d25ea47b8aa1d2c6ff159bc8b"}
Nov 03 01:51:49.853751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.852731    5508 generic.go:356] "Generic (PLEG): container finished" podID="327f2e67-8150-401c-8f37-08bb6966384b" containerID="dc1293fb37276f7a1e2f36ee829022f179aa4f29800833899dbb8b14fced128d" exitCode=0
Nov 03 01:51:49.853751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.852810    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm" event={"ID":"327f2e67-8150-401c-8f37-08bb6966384b","Type":"ContainerDied","Data":"dc1293fb37276f7a1e2f36ee829022f179aa4f29800833899dbb8b14fced128d"}
Nov 03 01:51:49.853751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.852849    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm" event={"ID":"327f2e67-8150-401c-8f37-08bb6966384b","Type":"ContainerDied","Data":"e110e03063298bdf39a47cc2bfe53c49947181497b495680c3f36e4239028dd2"}
Nov 03 01:51:49.853751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.852877    5508 scope.go:122] "RemoveContainer" containerID="dc1293fb37276f7a1e2f36ee829022f179aa4f29800833899dbb8b14fced128d"
Nov 03 01:51:49.853751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.853032    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"
Nov 03 01:51:49.870096 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.869987    5508 scope.go:122] "RemoveContainer" containerID="dc1293fb37276f7a1e2f36ee829022f179aa4f29800833899dbb8b14fced128d"
Nov 03 01:51:49.870457 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:51:49.870384    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1293fb37276f7a1e2f36ee829022f179aa4f29800833899dbb8b14fced128d\": container with ID starting with dc1293fb37276f7a1e2f36ee829022f179aa4f29800833899dbb8b14fced128d not found: ID does not exist" containerID="dc1293fb37276f7a1e2f36ee829022f179aa4f29800833899dbb8b14fced128d"
Nov 03 01:51:49.870457 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.870436    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1293fb37276f7a1e2f36ee829022f179aa4f29800833899dbb8b14fced128d"} err="failed to get container status \"dc1293fb37276f7a1e2f36ee829022f179aa4f29800833899dbb8b14fced128d\": rpc error: code = NotFound desc = could not find container \"dc1293fb37276f7a1e2f36ee829022f179aa4f29800833899dbb8b14fced128d\": container with ID starting with dc1293fb37276f7a1e2f36ee829022f179aa4f29800833899dbb8b14fced128d not found: ID does not exist"
Nov 03 01:51:49.920239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.920208    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327f2e67-8150-401c-8f37-08bb6966384b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327f2e67-8150-401c-8f37-08bb6966384b-plugins-config-volume\") pod \"327f2e67-8150-401c-8f37-08bb6966384b\" (UID: \"327f2e67-8150-401c-8f37-08bb6966384b\") "
Nov 03 01:51:49.920470 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.920295    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327f2e67-8150-401c-8f37-08bb6966384b-kube-api-access-j7d57\" (UniqueName: \"kubernetes.io/projected/327f2e67-8150-401c-8f37-08bb6966384b-kube-api-access-j7d57\") pod \"327f2e67-8150-401c-8f37-08bb6966384b\" (UID: \"327f2e67-8150-401c-8f37-08bb6966384b\") "
Nov 03 01:51:49.920706 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.920670    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327f2e67-8150-401c-8f37-08bb6966384b-plugins-config-volume" pod "327f2e67-8150-401c-8f37-08bb6966384b" (UID: "327f2e67-8150-401c-8f37-08bb6966384b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:51:49.920834 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.920804    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/327f2e67-8150-401c-8f37-08bb6966384b-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:51:49.923312 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.923286    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327f2e67-8150-401c-8f37-08bb6966384b-kube-api-access-j7d57" pod "327f2e67-8150-401c-8f37-08bb6966384b" (UID: "327f2e67-8150-401c-8f37-08bb6966384b"). InnerVolumeSpecName "kube-api-access-j7d57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:51:49.978881 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:49.978853    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"
Nov 03 01:51:50.021886 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.021847    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7d57\" (UniqueName: \"kubernetes.io/projected/327f2e67-8150-401c-8f37-08bb6966384b-kube-api-access-j7d57\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:51:50.122381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.122320    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1172569a-2f6a-4327-8f03-c65d14c4a678-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1172569a-2f6a-4327-8f03-c65d14c4a678-plugins-config-volume\") pod \"1172569a-2f6a-4327-8f03-c65d14c4a678\" (UID: \"1172569a-2f6a-4327-8f03-c65d14c4a678\") "
Nov 03 01:51:50.122579 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.122463    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1172569a-2f6a-4327-8f03-c65d14c4a678-kube-api-access-tg66k\" (UniqueName: \"kubernetes.io/projected/1172569a-2f6a-4327-8f03-c65d14c4a678-kube-api-access-tg66k\") pod \"1172569a-2f6a-4327-8f03-c65d14c4a678\" (UID: \"1172569a-2f6a-4327-8f03-c65d14c4a678\") "
Nov 03 01:51:50.122686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.122663    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1172569a-2f6a-4327-8f03-c65d14c4a678-plugins-config-volume" pod "1172569a-2f6a-4327-8f03-c65d14c4a678" (UID: "1172569a-2f6a-4327-8f03-c65d14c4a678"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:51:50.122942 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.122923    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1172569a-2f6a-4327-8f03-c65d14c4a678-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:51:50.124687 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.124660    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1172569a-2f6a-4327-8f03-c65d14c4a678-kube-api-access-tg66k" pod "1172569a-2f6a-4327-8f03-c65d14c4a678" (UID: "1172569a-2f6a-4327-8f03-c65d14c4a678"). InnerVolumeSpecName "kube-api-access-tg66k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:51:50.224275 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.224183    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg66k\" (UniqueName: \"kubernetes.io/projected/1172569a-2f6a-4327-8f03-c65d14c4a678-kube-api-access-tg66k\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:51:50.267373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.267329    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"]
Nov 03 01:51:50.274666 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.274633    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86e679-6956887d8f-8brnm"]
Nov 03 01:51:50.354274 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.354241    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"]
Nov 03 01:51:50.354855 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.354835    5508 cpu_manager.go:414] "containerMap: removing container" podUID="327f2e67-8150-401c-8f37-08bb6966384b" containerName="payload-processing"
Nov 03 01:51:50.354855 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.354851    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327f2e67-8150-401c-8f37-08bb6966384b" containerName="payload-processing"
Nov 03 01:51:50.355073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.354883    5508 cpu_manager.go:414] "containerMap: removing container" podUID="1172569a-2f6a-4327-8f03-c65d14c4a678" containerName="payload-pre-processing"
Nov 03 01:51:50.355073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.354891    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1172569a-2f6a-4327-8f03-c65d14c4a678" containerName="payload-pre-processing"
Nov 03 01:51:50.355073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.354982    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="327f2e67-8150-401c-8f37-08bb6966384b" containerName="payload-processing"
Nov 03 01:51:50.355073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.354996    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1172569a-2f6a-4327-8f03-c65d14c4a678" containerName="payload-pre-processing"
Nov 03 01:51:50.358573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.358552    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"
Nov 03 01:51:50.407371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.407298    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-36c2c1\""
Nov 03 01:51:50.408210 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.407481    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-36c2c1-dockercfg-thkzp\""
Nov 03 01:51:50.411152 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.411127    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"]
Nov 03 01:51:50.426540 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.426509    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ba26de-2eae-4bf1-a8bb-928fe164bf67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq\" (UID: \"29ba26de-2eae-4bf1-a8bb-928fe164bf67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"
Nov 03 01:51:50.426691 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.426644    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pfdf\" (UniqueName: \"kubernetes.io/projected/29ba26de-2eae-4bf1-a8bb-928fe164bf67-kube-api-access-8pfdf\") pod \"payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq\" (UID: \"29ba26de-2eae-4bf1-a8bb-928fe164bf67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"
Nov 03 01:51:50.527614 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.527541    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ba26de-2eae-4bf1-a8bb-928fe164bf67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq\" (UID: \"29ba26de-2eae-4bf1-a8bb-928fe164bf67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"
Nov 03 01:51:50.527772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.527626    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pfdf\" (UniqueName: \"kubernetes.io/projected/29ba26de-2eae-4bf1-a8bb-928fe164bf67-kube-api-access-8pfdf\") pod \"payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq\" (UID: \"29ba26de-2eae-4bf1-a8bb-928fe164bf67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"
Nov 03 01:51:50.528204 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.528181    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ba26de-2eae-4bf1-a8bb-928fe164bf67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq\" (UID: \"29ba26de-2eae-4bf1-a8bb-928fe164bf67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"
Nov 03 01:51:50.585189 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.585151    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pfdf\" (UniqueName: \"kubernetes.io/projected/29ba26de-2eae-4bf1-a8bb-928fe164bf67-kube-api-access-8pfdf\") pod \"payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq\" (UID: \"29ba26de-2eae-4bf1-a8bb-928fe164bf67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"
Nov 03 01:51:50.673191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.673162    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"
Nov 03 01:51:50.697127 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.697097    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327f2e67-8150-401c-8f37-08bb6966384b" path="/var/lib/kubelet/pods/327f2e67-8150-401c-8f37-08bb6966384b/volumes"
Nov 03 01:51:50.863388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.863298    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"
Nov 03 01:51:50.863388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.863314    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9" event={"ID":"1172569a-2f6a-4327-8f03-c65d14c4a678","Type":"ContainerDied","Data":"fc2f9e206a93ba665bfb1f413df97b53967574f5182691695c0e8247a93dd783"}
Nov 03 01:51:50.863627 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:50.863409    5508 scope.go:122] "RemoveContainer" containerID="26ee5d1c80ca9f18b5a4e5759e18704781f9286d25ea47b8aa1d2c6ff159bc8b"
Nov 03 01:51:51.017602 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.017563    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"]
Nov 03 01:51:51.023293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.023266    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7cdbb5c4b7-xrvb9"]
Nov 03 01:51:51.065812 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.065773    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"]
Nov 03 01:51:51.075952 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:51:51.075924    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ba26de_2eae_4bf1_a8bb_928fe164bf67.slice/crio-ee2e56e8d9658def264b3baa89caedce82a59da3c726d49f170d78bbce343010 WatchSource:0}: Error finding container ee2e56e8d9658def264b3baa89caedce82a59da3c726d49f170d78bbce343010: Status 404 returned error can't find the container with id ee2e56e8d9658def264b3baa89caedce82a59da3c726d49f170d78bbce343010
Nov 03 01:51:51.172114 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.172083    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"]
Nov 03 01:51:51.176523 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.176500    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"
Nov 03 01:51:51.241143 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.241110    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"]
Nov 03 01:51:51.247532 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.237287    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rdn\" (UniqueName: \"kubernetes.io/projected/b54ac1ac-b825-4335-9cff-bbd34a00ca89-kube-api-access-r7rdn\") pod \"payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw\" (UID: \"b54ac1ac-b825-4335-9cff-bbd34a00ca89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"
Nov 03 01:51:51.247810 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.247792    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54ac1ac-b825-4335-9cff-bbd34a00ca89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw\" (UID: \"b54ac1ac-b825-4335-9cff-bbd34a00ca89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"
Nov 03 01:51:51.349383 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.349332    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rdn\" (UniqueName: \"kubernetes.io/projected/b54ac1ac-b825-4335-9cff-bbd34a00ca89-kube-api-access-r7rdn\") pod \"payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw\" (UID: \"b54ac1ac-b825-4335-9cff-bbd34a00ca89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"
Nov 03 01:51:51.349590 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.349431    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54ac1ac-b825-4335-9cff-bbd34a00ca89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw\" (UID: \"b54ac1ac-b825-4335-9cff-bbd34a00ca89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"
Nov 03 01:51:51.350170 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.350145    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54ac1ac-b825-4335-9cff-bbd34a00ca89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw\" (UID: \"b54ac1ac-b825-4335-9cff-bbd34a00ca89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"
Nov 03 01:51:51.406886 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.406853    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rdn\" (UniqueName: \"kubernetes.io/projected/b54ac1ac-b825-4335-9cff-bbd34a00ca89-kube-api-access-r7rdn\") pod \"payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw\" (UID: \"b54ac1ac-b825-4335-9cff-bbd34a00ca89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"
Nov 03 01:51:51.493477 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.493441    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"
Nov 03 01:51:51.870017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.869933    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"]
Nov 03 01:51:51.876811 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.876784    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq" event={"ID":"29ba26de-2eae-4bf1-a8bb-928fe164bf67","Type":"ContainerStarted","Data":"61eb9278e16f6faca31d2236cd1d4bb1b9005a85083448eba9c2f99c3658023e"}
Nov 03 01:51:51.876933 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.876834    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq" event={"ID":"29ba26de-2eae-4bf1-a8bb-928fe164bf67","Type":"ContainerStarted","Data":"ee2e56e8d9658def264b3baa89caedce82a59da3c726d49f170d78bbce343010"}
Nov 03 01:51:51.876933 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.876887    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"
Nov 03 01:51:51.884030 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:51:51.884007    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54ac1ac_b825_4335_9cff_bbd34a00ca89.slice/crio-5e2a90dac47977d12db501968dfebe7b7102fe82021abacb1da8503b03ada519 WatchSource:0}: Error finding container 5e2a90dac47977d12db501968dfebe7b7102fe82021abacb1da8503b03ada519: Status 404 returned error can't find the container with id 5e2a90dac47977d12db501968dfebe7b7102fe82021abacb1da8503b03ada519
Nov 03 01:51:52.005377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:51.989997    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq" podStartSLOduration=1.989978073 podStartE2EDuration="1.989978073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:51.987672764 +0000 UTC m=+1991.413279245" watchObservedRunningTime="2026-11-03 01:51:51.989978073 +0000 UTC m=+1991.415584539"
Nov 03 01:51:52.696879 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:52.696842    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1172569a-2f6a-4327-8f03-c65d14c4a678" path="/var/lib/kubelet/pods/1172569a-2f6a-4327-8f03-c65d14c4a678/volumes"
Nov 03 01:51:52.887440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:52.887401    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw" event={"ID":"b54ac1ac-b825-4335-9cff-bbd34a00ca89","Type":"ContainerStarted","Data":"c74b599f17cb7a5986a3108242092436761aaa837a0f1a300ce34c5bf2413078"}
Nov 03 01:51:52.887651 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:52.887470    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw" event={"ID":"b54ac1ac-b825-4335-9cff-bbd34a00ca89","Type":"ContainerStarted","Data":"5e2a90dac47977d12db501968dfebe7b7102fe82021abacb1da8503b03ada519"}
Nov 03 01:51:52.887737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:52.887715    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"
Nov 03 01:51:52.997736 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:51:52.997631    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw" podStartSLOduration=1.997615527 podStartE2EDuration="1.997615527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:52.995999992 +0000 UTC m=+1992.421606456" watchObservedRunningTime="2026-11-03 01:51:52.997615527 +0000 UTC m=+1992.423221990"
Nov 03 01:52:02.890283 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:02.890252    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"
Nov 03 01:52:03.895618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:03.895586    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"
Nov 03 01:52:05.250279 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:05.250243    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"]
Nov 03 01:52:05.343479 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:05.343450    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"]
Nov 03 01:52:10.250809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.250767    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj" podUID="c53344b7-1435-47fd-9498-cca201315546" containerName="payload-processing" containerID="cri-o://cd5a3ace76b793c93a1dde82e46eb183b0348ddd787b01f83d7af5f4f9069210" gracePeriod=25
Nov 03 01:52:10.343719 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.343671    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7" podUID="33f9264c-091c-4a9a-9f38-e5c5923ebb59" containerName="payload-pre-processing" containerID="cri-o://1664b300c9fd35649f828272ac0b7b494d76db3b9658e395d52ce5ffa5743fcb" gracePeriod=25
Nov 03 01:52:10.498188 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.498155    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:52:10.619807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.619785    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"
Nov 03 01:52:10.646097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.646076    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"
Nov 03 01:52:10.663452 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.663422    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f9264c-091c-4a9a-9f38-e5c5923ebb59-kube-api-access-8p68l\" (UniqueName: \"kubernetes.io/projected/33f9264c-091c-4a9a-9f38-e5c5923ebb59-kube-api-access-8p68l\") pod \"33f9264c-091c-4a9a-9f38-e5c5923ebb59\" (UID: \"33f9264c-091c-4a9a-9f38-e5c5923ebb59\") "
Nov 03 01:52:10.663597 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.663460    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53344b7-1435-47fd-9498-cca201315546-kube-api-access-728sh\" (UniqueName: \"kubernetes.io/projected/c53344b7-1435-47fd-9498-cca201315546-kube-api-access-728sh\") pod \"c53344b7-1435-47fd-9498-cca201315546\" (UID: \"c53344b7-1435-47fd-9498-cca201315546\") "
Nov 03 01:52:10.663597 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.663568    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53344b7-1435-47fd-9498-cca201315546-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53344b7-1435-47fd-9498-cca201315546-plugins-config-volume\") pod \"c53344b7-1435-47fd-9498-cca201315546\" (UID: \"c53344b7-1435-47fd-9498-cca201315546\") "
Nov 03 01:52:10.663707 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.663672    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f9264c-091c-4a9a-9f38-e5c5923ebb59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f9264c-091c-4a9a-9f38-e5c5923ebb59-plugins-config-volume\") pod \"33f9264c-091c-4a9a-9f38-e5c5923ebb59\" (UID: \"33f9264c-091c-4a9a-9f38-e5c5923ebb59\") "
Nov 03 01:52:10.664017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.663994    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53344b7-1435-47fd-9498-cca201315546-plugins-config-volume" pod "c53344b7-1435-47fd-9498-cca201315546" (UID: "c53344b7-1435-47fd-9498-cca201315546"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:52:10.664119 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.664064    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f9264c-091c-4a9a-9f38-e5c5923ebb59-plugins-config-volume" pod "33f9264c-091c-4a9a-9f38-e5c5923ebb59" (UID: "33f9264c-091c-4a9a-9f38-e5c5923ebb59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:52:10.664232 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.664215    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f9264c-091c-4a9a-9f38-e5c5923ebb59-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:10.664293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.664233    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53344b7-1435-47fd-9498-cca201315546-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:10.665548 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.665528    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53344b7-1435-47fd-9498-cca201315546-kube-api-access-728sh" pod "c53344b7-1435-47fd-9498-cca201315546" (UID: "c53344b7-1435-47fd-9498-cca201315546"). InnerVolumeSpecName "kube-api-access-728sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:10.665637 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.665554    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f9264c-091c-4a9a-9f38-e5c5923ebb59-kube-api-access-8p68l" pod "33f9264c-091c-4a9a-9f38-e5c5923ebb59" (UID: "33f9264c-091c-4a9a-9f38-e5c5923ebb59"). InnerVolumeSpecName "kube-api-access-8p68l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:10.765608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.765523    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p68l\" (UniqueName: \"kubernetes.io/projected/33f9264c-091c-4a9a-9f38-e5c5923ebb59-kube-api-access-8p68l\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:10.765608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:10.765562    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-728sh\" (UniqueName: \"kubernetes.io/projected/c53344b7-1435-47fd-9498-cca201315546-kube-api-access-728sh\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:11.043990 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.043892    5508 generic.go:356] "Generic (PLEG): container finished" podID="33f9264c-091c-4a9a-9f38-e5c5923ebb59" containerID="1664b300c9fd35649f828272ac0b7b494d76db3b9658e395d52ce5ffa5743fcb" exitCode=0
Nov 03 01:52:11.043990 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.043945    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"
Nov 03 01:52:11.043990 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.043958    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7" event={"ID":"33f9264c-091c-4a9a-9f38-e5c5923ebb59","Type":"ContainerDied","Data":"1664b300c9fd35649f828272ac0b7b494d76db3b9658e395d52ce5ffa5743fcb"}
Nov 03 01:52:11.044256 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.044034    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7" event={"ID":"33f9264c-091c-4a9a-9f38-e5c5923ebb59","Type":"ContainerDied","Data":"fb9502462f94d0e828fef15f3c5debdb5958eb928dc34cb32f8294ca186520de"}
Nov 03 01:52:11.044256 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.044055    5508 scope.go:122] "RemoveContainer" containerID="1664b300c9fd35649f828272ac0b7b494d76db3b9658e395d52ce5ffa5743fcb"
Nov 03 01:52:11.045691 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.045660    5508 generic.go:356] "Generic (PLEG): container finished" podID="c53344b7-1435-47fd-9498-cca201315546" containerID="cd5a3ace76b793c93a1dde82e46eb183b0348ddd787b01f83d7af5f4f9069210" exitCode=0
Nov 03 01:52:11.045888 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.045697    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj" event={"ID":"c53344b7-1435-47fd-9498-cca201315546","Type":"ContainerDied","Data":"cd5a3ace76b793c93a1dde82e46eb183b0348ddd787b01f83d7af5f4f9069210"}
Nov 03 01:52:11.045888 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.045759    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj" event={"ID":"c53344b7-1435-47fd-9498-cca201315546","Type":"ContainerDied","Data":"9203070a9e4b12e19ad24a4115a557b915b0abcf1718b5e98e77dc39373812ff"}
Nov 03 01:52:11.045888 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.045713    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"
Nov 03 01:52:11.054272 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.054238    5508 scope.go:122] "RemoveContainer" containerID="1664b300c9fd35649f828272ac0b7b494d76db3b9658e395d52ce5ffa5743fcb"
Nov 03 01:52:11.054578 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:52:11.054557    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1664b300c9fd35649f828272ac0b7b494d76db3b9658e395d52ce5ffa5743fcb\": container with ID starting with 1664b300c9fd35649f828272ac0b7b494d76db3b9658e395d52ce5ffa5743fcb not found: ID does not exist" containerID="1664b300c9fd35649f828272ac0b7b494d76db3b9658e395d52ce5ffa5743fcb"
Nov 03 01:52:11.054675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.054583    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1664b300c9fd35649f828272ac0b7b494d76db3b9658e395d52ce5ffa5743fcb"} err="failed to get container status \"1664b300c9fd35649f828272ac0b7b494d76db3b9658e395d52ce5ffa5743fcb\": rpc error: code = NotFound desc = could not find container \"1664b300c9fd35649f828272ac0b7b494d76db3b9658e395d52ce5ffa5743fcb\": container with ID starting with 1664b300c9fd35649f828272ac0b7b494d76db3b9658e395d52ce5ffa5743fcb not found: ID does not exist"
Nov 03 01:52:11.054675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.054600    5508 scope.go:122] "RemoveContainer" containerID="cd5a3ace76b793c93a1dde82e46eb183b0348ddd787b01f83d7af5f4f9069210"
Nov 03 01:52:11.063586 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.063572    5508 scope.go:122] "RemoveContainer" containerID="cd5a3ace76b793c93a1dde82e46eb183b0348ddd787b01f83d7af5f4f9069210"
Nov 03 01:52:11.063864 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:52:11.063845    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5a3ace76b793c93a1dde82e46eb183b0348ddd787b01f83d7af5f4f9069210\": container with ID starting with cd5a3ace76b793c93a1dde82e46eb183b0348ddd787b01f83d7af5f4f9069210 not found: ID does not exist" containerID="cd5a3ace76b793c93a1dde82e46eb183b0348ddd787b01f83d7af5f4f9069210"
Nov 03 01:52:11.063934 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.063869    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5a3ace76b793c93a1dde82e46eb183b0348ddd787b01f83d7af5f4f9069210"} err="failed to get container status \"cd5a3ace76b793c93a1dde82e46eb183b0348ddd787b01f83d7af5f4f9069210\": rpc error: code = NotFound desc = could not find container \"cd5a3ace76b793c93a1dde82e46eb183b0348ddd787b01f83d7af5f4f9069210\": container with ID starting with cd5a3ace76b793c93a1dde82e46eb183b0348ddd787b01f83d7af5f4f9069210 not found: ID does not exist"
Nov 03 01:52:11.203291 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.203257    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"]
Nov 03 01:52:11.207503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.207478    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b177c9-59849cdc7f-jbjgj"]
Nov 03 01:52:11.311299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.311189    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"]
Nov 03 01:52:11.315230 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:11.315205    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-6cf687c799-rmdw7"]
Nov 03 01:52:12.692134 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:12.692102    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f9264c-091c-4a9a-9f38-e5c5923ebb59" path="/var/lib/kubelet/pods/33f9264c-091c-4a9a-9f38-e5c5923ebb59/volumes"
Nov 03 01:52:12.692608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:12.692494    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53344b7-1435-47fd-9498-cca201315546" path="/var/lib/kubelet/pods/c53344b7-1435-47fd-9498-cca201315546/volumes"
Nov 03 01:52:13.867671 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:52:13.867625    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:52:14.053956 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.053928    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"]
Nov 03 01:52:14.054543 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.054528    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c53344b7-1435-47fd-9498-cca201315546" containerName="payload-processing"
Nov 03 01:52:14.054621 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.054545    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53344b7-1435-47fd-9498-cca201315546" containerName="payload-processing"
Nov 03 01:52:14.054621 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.054574    5508 cpu_manager.go:414] "containerMap: removing container" podUID="33f9264c-091c-4a9a-9f38-e5c5923ebb59" containerName="payload-pre-processing"
Nov 03 01:52:14.054621 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.054579    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f9264c-091c-4a9a-9f38-e5c5923ebb59" containerName="payload-pre-processing"
Nov 03 01:52:14.054733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.054674    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f9264c-091c-4a9a-9f38-e5c5923ebb59" containerName="payload-pre-processing"
Nov 03 01:52:14.054733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.054693    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53344b7-1435-47fd-9498-cca201315546" containerName="payload-processing"
Nov 03 01:52:14.059774 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.059752    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"
Nov 03 01:52:14.104277 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.104251    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8zcv\" (UniqueName: \"kubernetes.io/projected/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-kube-api-access-j8zcv\") pod \"payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8\" (UID: \"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"
Nov 03 01:52:14.104422 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.104395    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8\" (UID: \"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"
Nov 03 01:52:14.110867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.110842    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"]
Nov 03 01:52:14.158474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.155697    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-738acb\""
Nov 03 01:52:14.158474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.155865    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-738acb-dockercfg-svkq2\""
Nov 03 01:52:14.226369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.225597    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8\" (UID: \"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"
Nov 03 01:52:14.226369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.225754    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8zcv\" (UniqueName: \"kubernetes.io/projected/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-kube-api-access-j8zcv\") pod \"payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8\" (UID: \"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"
Nov 03 01:52:14.251398 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.226977    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8\" (UID: \"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"
Nov 03 01:52:14.251398 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.237899    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"]
Nov 03 01:52:14.264371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.258894    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"
Nov 03 01:52:14.298931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.296590    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8zcv\" (UniqueName: \"kubernetes.io/projected/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-kube-api-access-j8zcv\") pod \"payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8\" (UID: \"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"
Nov 03 01:52:14.300779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.300752    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"]
Nov 03 01:52:14.326638 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.326601    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t\" (UID: \"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"
Nov 03 01:52:14.326827 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.326806    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ggqc\" (UniqueName: \"kubernetes.io/projected/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-kube-api-access-7ggqc\") pod \"payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t\" (UID: \"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"
Nov 03 01:52:14.373972 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.373939    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"
Nov 03 01:52:14.428483 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.428388    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t\" (UID: \"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"
Nov 03 01:52:14.428686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.428541    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ggqc\" (UniqueName: \"kubernetes.io/projected/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-kube-api-access-7ggqc\") pod \"payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t\" (UID: \"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"
Nov 03 01:52:14.429800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.429774    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t\" (UID: \"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"
Nov 03 01:52:14.485994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.485963    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ggqc\" (UniqueName: \"kubernetes.io/projected/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-kube-api-access-7ggqc\") pod \"payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t\" (UID: \"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"
Nov 03 01:52:14.599237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.599197    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"
Nov 03 01:52:14.752302 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.752227    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"]
Nov 03 01:52:14.757682 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:52:14.757653    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c8bbe6_bb8a_4096_a4f5_cc7d4e712a96.slice/crio-c1014433bc36beb50ee0d4ea9f063c8af17cb0c3335ddcb472d389239709a2a9 WatchSource:0}: Error finding container c1014433bc36beb50ee0d4ea9f063c8af17cb0c3335ddcb472d389239709a2a9: Status 404 returned error can't find the container with id c1014433bc36beb50ee0d4ea9f063c8af17cb0c3335ddcb472d389239709a2a9
Nov 03 01:52:14.974550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:14.974516    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"]
Nov 03 01:52:14.980955 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:52:14.980929    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a58d16_d2ab_46bb_bb83_9339e9f1e0b4.slice/crio-110aa445f5d36fb01cc86f754be2ed7be3286744fae4794dbce4fe9c9cb3a1ae WatchSource:0}: Error finding container 110aa445f5d36fb01cc86f754be2ed7be3286744fae4794dbce4fe9c9cb3a1ae: Status 404 returned error can't find the container with id 110aa445f5d36fb01cc86f754be2ed7be3286744fae4794dbce4fe9c9cb3a1ae
Nov 03 01:52:15.078962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:15.078932    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t" event={"ID":"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4","Type":"ContainerStarted","Data":"8bcb998798013965f4631b246bfbb2fbf023e15e7af19af17223b5bc75f2f8bf"}
Nov 03 01:52:15.079089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:15.078991    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t" event={"ID":"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4","Type":"ContainerStarted","Data":"110aa445f5d36fb01cc86f754be2ed7be3286744fae4794dbce4fe9c9cb3a1ae"}
Nov 03 01:52:15.079191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:15.079172    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"
Nov 03 01:52:15.080889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:15.080806    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8" event={"ID":"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96","Type":"ContainerStarted","Data":"384e2434d841e819ec672c5b26b42ee3c71d6285cdb93d8a7748df70f7e5b6d3"}
Nov 03 01:52:15.080889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:15.080854    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8" event={"ID":"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96","Type":"ContainerStarted","Data":"c1014433bc36beb50ee0d4ea9f063c8af17cb0c3335ddcb472d389239709a2a9"}
Nov 03 01:52:15.081066 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:15.080981    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"
Nov 03 01:52:15.190377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:15.190318    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t" podStartSLOduration=1.190304941 podStartE2EDuration="1.190304941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:15.187030098 +0000 UTC m=+2014.612636577" watchObservedRunningTime="2026-11-03 01:52:15.190304941 +0000 UTC m=+2014.615911404"
Nov 03 01:52:15.304418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:15.297437    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8" podStartSLOduration=1.297419473 podStartE2EDuration="1.297419473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:15.294464377 +0000 UTC m=+2014.720070841" watchObservedRunningTime="2026-11-03 01:52:15.297419473 +0000 UTC m=+2014.723025937"
Nov 03 01:52:23.424735 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:23.424702    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"]
Nov 03 01:52:23.425180 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:23.424892    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy" containerID="cri-o://8cd6efdb3124a34d590cbac6058aaad663954c5a1ed68edba25e8fd3e53e0a13" gracePeriod=30
Nov 03 01:52:23.521871 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:23.506453    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"]
Nov 03 01:52:23.521871 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:23.506858    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" podUID="5f420022-cf2a-44db-a72b-2934f355375e" containerName="istio-proxy" containerID="cri-o://ce85b59582911c6192c85b5cd341087b3392f46f34afec47b956b073a4276042" gracePeriod=30
Nov 03 01:52:23.521871 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:23.511677    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"]
Nov 03 01:52:23.521871 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:23.512080    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerName="istio-proxy" containerID="cri-o://d11e7b0fdcc95a5ea3eacf901838f00f4e2658808823fb60db1e7c5aa9707f03" gracePeriod=30
Nov 03 01:52:23.535442 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:23.535408    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"]
Nov 03 01:52:23.535726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:23.535695    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerName="istio-proxy" containerID="cri-o://ff1f253cf2f0176e052dc612a1a185b550457ce5cc1f822e386f13f04c459a67" gracePeriod=30
Nov 03 01:52:26.090870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:26.090841    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"
Nov 03 01:52:26.091294 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:26.090914    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"
Nov 03 01:52:28.744634 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.744603    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:52:28.826725 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.826570    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:52:28.834048 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.834023    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-credential-socket\") pod \"d05f8000-1622-4313-89df-5d55ce6b2f81\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") "
Nov 03 01:52:28.834183 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.834080    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-token\" (UniqueName: \"kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-token\") pod \"d05f8000-1622-4313-89df-5d55ce6b2f81\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") "
Nov 03 01:52:28.834183 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.834139    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-podinfo\") pod \"d05f8000-1622-4313-89df-5d55ce6b2f81\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") "
Nov 03 01:52:28.834183 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.834170    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05f8000-1622-4313-89df-5d55ce6b2f81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05f8000-1622-4313-89df-5d55ce6b2f81-istiod-ca-cert\") pod \"d05f8000-1622-4313-89df-5d55ce6b2f81\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") "
Nov 03 01:52:28.834379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.834245    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-certs\") pod \"d05f8000-1622-4313-89df-5d55ce6b2f81\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") "
Nov 03 01:52:28.834379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.834314    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-credential-socket" pod "d05f8000-1622-4313-89df-5d55ce6b2f81" (UID: "d05f8000-1622-4313-89df-5d55ce6b2f81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:28.834466 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.834428    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-data\") pod \"d05f8000-1622-4313-89df-5d55ce6b2f81\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") "
Nov 03 01:52:28.834529 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.834510    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-certs" pod "d05f8000-1622-4313-89df-5d55ce6b2f81" (UID: "d05f8000-1622-4313-89df-5d55ce6b2f81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:28.834595 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.834530    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-envoy\") pod \"d05f8000-1622-4313-89df-5d55ce6b2f81\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") "
Nov 03 01:52:28.834655 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.834622    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-kube-api-access-clhlm\" (UniqueName: \"kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-kube-api-access-clhlm\") pod \"d05f8000-1622-4313-89df-5d55ce6b2f81\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") "
Nov 03 01:52:28.834714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.834679    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-data" pod "d05f8000-1622-4313-89df-5d55ce6b2f81" (UID: "d05f8000-1622-4313-89df-5d55ce6b2f81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:28.834714 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.834705    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-socket\") pod \"d05f8000-1622-4313-89df-5d55ce6b2f81\" (UID: \"d05f8000-1622-4313-89df-5d55ce6b2f81\") "
Nov 03 01:52:28.835312 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.835209    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:28.835312 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.835233    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:28.835312 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.835248    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:28.835312 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.835243    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-socket" pod "d05f8000-1622-4313-89df-5d55ce6b2f81" (UID: "d05f8000-1622-4313-89df-5d55ce6b2f81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:28.835746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.835708    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05f8000-1622-4313-89df-5d55ce6b2f81-istiod-ca-cert" pod "d05f8000-1622-4313-89df-5d55ce6b2f81" (UID: "d05f8000-1622-4313-89df-5d55ce6b2f81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:52:28.837573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.837550    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-token" pod "d05f8000-1622-4313-89df-5d55ce6b2f81" (UID: "d05f8000-1622-4313-89df-5d55ce6b2f81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:28.838697 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.838673    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-podinfo" pod "d05f8000-1622-4313-89df-5d55ce6b2f81" (UID: "d05f8000-1622-4313-89df-5d55ce6b2f81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:52:28.838820 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.838731    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-kube-api-access-clhlm" pod "d05f8000-1622-4313-89df-5d55ce6b2f81" (UID: "d05f8000-1622-4313-89df-5d55ce6b2f81"). InnerVolumeSpecName "kube-api-access-clhlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:28.838921 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.838848    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-envoy" pod "d05f8000-1622-4313-89df-5d55ce6b2f81" (UID: "d05f8000-1622-4313-89df-5d55ce6b2f81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:28.874881 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.874856    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:52:28.890887 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.890871    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:52:28.936692 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.936665    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-podinfo\") pod \"801be76b-b7a4-468b-b72e-b6790c0bef0b\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") "
Nov 03 01:52:28.936840 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.936713    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-kube-api-access-l7zmf\" (UniqueName: \"kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-kube-api-access-l7zmf\") pod \"801be76b-b7a4-468b-b72e-b6790c0bef0b\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") "
Nov 03 01:52:28.936840 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.936755    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-certs\") pod \"801be76b-b7a4-468b-b72e-b6790c0bef0b\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") "
Nov 03 01:52:28.936840 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.936802    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-data\") pod \"801be76b-b7a4-468b-b72e-b6790c0bef0b\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") "
Nov 03 01:52:28.937010 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.936908    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-credential-socket\") pod \"801be76b-b7a4-468b-b72e-b6790c0bef0b\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") "
Nov 03 01:52:28.937010 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.936958    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-socket\") pod \"801be76b-b7a4-468b-b72e-b6790c0bef0b\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") "
Nov 03 01:52:28.937010 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.936985    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-token\" (UniqueName: \"kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-token\") pod \"801be76b-b7a4-468b-b72e-b6790c0bef0b\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") "
Nov 03 01:52:28.937172 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937052    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/801be76b-b7a4-468b-b72e-b6790c0bef0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801be76b-b7a4-468b-b72e-b6790c0bef0b-istiod-ca-cert\") pod \"801be76b-b7a4-468b-b72e-b6790c0bef0b\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") "
Nov 03 01:52:28.937172 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937082    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-certs" pod "801be76b-b7a4-468b-b72e-b6790c0bef0b" (UID: "801be76b-b7a4-468b-b72e-b6790c0bef0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:28.937172 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937163    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-envoy\") pod \"801be76b-b7a4-468b-b72e-b6790c0bef0b\" (UID: \"801be76b-b7a4-468b-b72e-b6790c0bef0b\") "
Nov 03 01:52:28.937340 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937265    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-credential-socket" pod "801be76b-b7a4-468b-b72e-b6790c0bef0b" (UID: "801be76b-b7a4-468b-b72e-b6790c0bef0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:28.937546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937482    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-socket" pod "801be76b-b7a4-468b-b72e-b6790c0bef0b" (UID: "801be76b-b7a4-468b-b72e-b6790c0bef0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:28.937546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937512    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801be76b-b7a4-468b-b72e-b6790c0bef0b-istiod-ca-cert" pod "801be76b-b7a4-468b-b72e-b6790c0bef0b" (UID: "801be76b-b7a4-468b-b72e-b6790c0bef0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:52:28.937794 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937773    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:28.937863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937801    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:28.937863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937817    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:28.937863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937830    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:28.937863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937847    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d05f8000-1622-4313-89df-5d55ce6b2f81-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:28.937863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937861    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:28.938132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937875    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:28.938132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937888    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801be76b-b7a4-468b-b72e-b6790c0bef0b-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:28.938132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937900    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d05f8000-1622-4313-89df-5d55ce6b2f81-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:28.938132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937914    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clhlm\" (UniqueName: \"kubernetes.io/projected/d05f8000-1622-4313-89df-5d55ce6b2f81-kube-api-access-clhlm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:28.938132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.937975    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-data" pod "801be76b-b7a4-468b-b72e-b6790c0bef0b" (UID: "801be76b-b7a4-468b-b72e-b6790c0bef0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:28.939538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.939490    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-token" pod "801be76b-b7a4-468b-b72e-b6790c0bef0b" (UID: "801be76b-b7a4-468b-b72e-b6790c0bef0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:28.939828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.939803    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-podinfo" pod "801be76b-b7a4-468b-b72e-b6790c0bef0b" (UID: "801be76b-b7a4-468b-b72e-b6790c0bef0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:52:28.939906 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.939872    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-kube-api-access-l7zmf" pod "801be76b-b7a4-468b-b72e-b6790c0bef0b" (UID: "801be76b-b7a4-468b-b72e-b6790c0bef0b"). InnerVolumeSpecName "kube-api-access-l7zmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:28.940101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:28.940086    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-envoy" pod "801be76b-b7a4-468b-b72e-b6790c0bef0b" (UID: "801be76b-b7a4-468b-b72e-b6790c0bef0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:29.039148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039060    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-certs\") pod \"5f420022-cf2a-44db-a72b-2934f355375e\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") "
Nov 03 01:52:29.039148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039109    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-socket\") pod \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") "
Nov 03 01:52:29.039148 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039143    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-credential-socket\") pod \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") "
Nov 03 01:52:29.039565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039188    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-socket\") pod \"5f420022-cf2a-44db-a72b-2934f355375e\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") "
Nov 03 01:52:29.039565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039230    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-kube-api-access-6nthh\" (UniqueName: \"kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-kube-api-access-6nthh\") pod \"5f420022-cf2a-44db-a72b-2934f355375e\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") "
Nov 03 01:52:29.039565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039264    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-data\") pod \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") "
Nov 03 01:52:29.039565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039317    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-credential-socket\") pod \"5f420022-cf2a-44db-a72b-2934f355375e\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") "
Nov 03 01:52:29.039565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039363    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-kube-api-access-qvbwg\" (UniqueName: \"kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-kube-api-access-qvbwg\") pod \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") "
Nov 03 01:52:29.039565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039383    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-certs" pod "5f420022-cf2a-44db-a72b-2934f355375e" (UID: "5f420022-cf2a-44db-a72b-2934f355375e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:29.039565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039395    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-socket" pod "33c8b2c7-6e14-42dd-a9e5-457550da27ff" (UID: "33c8b2c7-6e14-42dd-a9e5-457550da27ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:29.039565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039397    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f420022-cf2a-44db-a72b-2934f355375e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f420022-cf2a-44db-a72b-2934f355375e-istiod-ca-cert\") pod \"5f420022-cf2a-44db-a72b-2934f355375e\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") "
Nov 03 01:52:29.039565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039485    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f420022-cf2a-44db-a72b-2934f355375e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f420022-cf2a-44db-a72b-2934f355375e-istio-podinfo\") pod \"5f420022-cf2a-44db-a72b-2934f355375e\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") "
Nov 03 01:52:29.039565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039500    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-credential-socket" pod "33c8b2c7-6e14-42dd-a9e5-457550da27ff" (UID: "33c8b2c7-6e14-42dd-a9e5-457550da27ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:29.040097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039607    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-data" pod "33c8b2c7-6e14-42dd-a9e5-457550da27ff" (UID: "33c8b2c7-6e14-42dd-a9e5-457550da27ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:29.040097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039636    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istiod-ca-cert\") pod \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") "
Nov 03 01:52:29.040097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039668    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-podinfo\") pod \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") "
Nov 03 01:52:29.040097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039710    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-token\" (UniqueName: \"kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-token\") pod \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") "
Nov 03 01:52:29.040097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039714    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-credential-socket" pod "5f420022-cf2a-44db-a72b-2934f355375e" (UID: "5f420022-cf2a-44db-a72b-2934f355375e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:29.040097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039724    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f420022-cf2a-44db-a72b-2934f355375e-istiod-ca-cert" pod "5f420022-cf2a-44db-a72b-2934f355375e" (UID: "5f420022-cf2a-44db-a72b-2934f355375e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:52:29.040097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039746    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-data\") pod \"5f420022-cf2a-44db-a72b-2934f355375e\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") "
Nov 03 01:52:29.040097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039791    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-istio-token\" (UniqueName: \"kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-istio-token\") pod \"5f420022-cf2a-44db-a72b-2934f355375e\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") "
Nov 03 01:52:29.040097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039856    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-envoy\") pod \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") "
Nov 03 01:52:29.040097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039888    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-certs\") pod \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\" (UID: \"33c8b2c7-6e14-42dd-a9e5-457550da27ff\") "
Nov 03 01:52:29.040097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.039930    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-envoy\") pod \"5f420022-cf2a-44db-a72b-2934f355375e\" (UID: \"5f420022-cf2a-44db-a72b-2934f355375e\") "
Nov 03 01:52:29.040097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040030    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-data" pod "5f420022-cf2a-44db-a72b-2934f355375e" (UID: "5f420022-cf2a-44db-a72b-2934f355375e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:29.040097 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040070    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istiod-ca-cert" pod "33c8b2c7-6e14-42dd-a9e5-457550da27ff" (UID: "33c8b2c7-6e14-42dd-a9e5-457550da27ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:52:29.040804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040563    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.040804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040586    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.040804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040602    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.040804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040616    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.040804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040630    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.040804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040647    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.040804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040661    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f420022-cf2a-44db-a72b-2934f355375e-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.040804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040675    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.040804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040690    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.040804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040703    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.040804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040718    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.040804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040732    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7zmf\" (UniqueName: \"kubernetes.io/projected/801be76b-b7a4-468b-b72e-b6790c0bef0b-kube-api-access-l7zmf\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.040804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040745    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801be76b-b7a4-468b-b72e-b6790c0bef0b-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.041554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.040970    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-certs" pod "33c8b2c7-6e14-42dd-a9e5-457550da27ff" (UID: "33c8b2c7-6e14-42dd-a9e5-457550da27ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:29.041554 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.041045    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-socket" pod "5f420022-cf2a-44db-a72b-2934f355375e" (UID: "5f420022-cf2a-44db-a72b-2934f355375e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:29.042406 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.042335    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-kube-api-access-qvbwg" pod "33c8b2c7-6e14-42dd-a9e5-457550da27ff" (UID: "33c8b2c7-6e14-42dd-a9e5-457550da27ff"). InnerVolumeSpecName "kube-api-access-qvbwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:29.042958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.042931    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-envoy" pod "5f420022-cf2a-44db-a72b-2934f355375e" (UID: "5f420022-cf2a-44db-a72b-2934f355375e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:29.043541 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.043518    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-podinfo" pod "33c8b2c7-6e14-42dd-a9e5-457550da27ff" (UID: "33c8b2c7-6e14-42dd-a9e5-457550da27ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:52:29.043650 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.043568    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-envoy" pod "33c8b2c7-6e14-42dd-a9e5-457550da27ff" (UID: "33c8b2c7-6e14-42dd-a9e5-457550da27ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:52:29.044749 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.044723    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-istio-token" pod "5f420022-cf2a-44db-a72b-2934f355375e" (UID: "5f420022-cf2a-44db-a72b-2934f355375e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:29.044846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.044775    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-token" pod "33c8b2c7-6e14-42dd-a9e5-457550da27ff" (UID: "33c8b2c7-6e14-42dd-a9e5-457550da27ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:29.044891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.044878    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-kube-api-access-6nthh" pod "5f420022-cf2a-44db-a72b-2934f355375e" (UID: "5f420022-cf2a-44db-a72b-2934f355375e"). InnerVolumeSpecName "kube-api-access-6nthh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:52:29.045123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.045099    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f420022-cf2a-44db-a72b-2934f355375e-istio-podinfo" pod "5f420022-cf2a-44db-a72b-2934f355375e" (UID: "5f420022-cf2a-44db-a72b-2934f355375e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:52:29.142080 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.142040    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.142080 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.142080    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f420022-cf2a-44db-a72b-2934f355375e-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.142327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.142094    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nthh\" (UniqueName: \"kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-kube-api-access-6nthh\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.142327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.142108    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbwg\" (UniqueName: \"kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-kube-api-access-qvbwg\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.142327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.142122    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f420022-cf2a-44db-a72b-2934f355375e-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.142327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.142138    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.142327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.142151    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.142327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.142164    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f420022-cf2a-44db-a72b-2934f355375e-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.142327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.142179    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.142327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.142196    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33c8b2c7-6e14-42dd-a9e5-457550da27ff-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:52:29.176628 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.176561    5508 generic.go:356] "Generic (PLEG): container finished" podID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerID="d11e7b0fdcc95a5ea3eacf901838f00f4e2658808823fb60db1e7c5aa9707f03" exitCode=0
Nov 03 01:52:29.176799 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.176625    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"
Nov 03 01:52:29.176799 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.176635    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" event={"ID":"801be76b-b7a4-468b-b72e-b6790c0bef0b","Type":"ContainerDied","Data":"d11e7b0fdcc95a5ea3eacf901838f00f4e2658808823fb60db1e7c5aa9707f03"}
Nov 03 01:52:29.176799 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.176695    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46" event={"ID":"801be76b-b7a4-468b-b72e-b6790c0bef0b","Type":"ContainerDied","Data":"d8ab56a98657d0bc6a282b0474c412d17dd63be475d38ba59577e6f5539b0219"}
Nov 03 01:52:29.176799 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.176726    5508 scope.go:122] "RemoveContainer" containerID="d11e7b0fdcc95a5ea3eacf901838f00f4e2658808823fb60db1e7c5aa9707f03"
Nov 03 01:52:29.178236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.178214    5508 generic.go:356] "Generic (PLEG): container finished" podID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerID="ff1f253cf2f0176e052dc612a1a185b550457ce5cc1f822e386f13f04c459a67" exitCode=0
Nov 03 01:52:29.178443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.178273    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"
Nov 03 01:52:29.178443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.178308    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" event={"ID":"33c8b2c7-6e14-42dd-a9e5-457550da27ff","Type":"ContainerDied","Data":"ff1f253cf2f0176e052dc612a1a185b550457ce5cc1f822e386f13f04c459a67"}
Nov 03 01:52:29.178443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.178380    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx" event={"ID":"33c8b2c7-6e14-42dd-a9e5-457550da27ff","Type":"ContainerDied","Data":"6c8da091d14e110e4f2bdd573cf47d3b1c9a683709160bbdd2c21f9a52b93205"}
Nov 03 01:52:29.181421 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.181397    5508 generic.go:356] "Generic (PLEG): container finished" podID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerID="8cd6efdb3124a34d590cbac6058aaad663954c5a1ed68edba25e8fd3e53e0a13" exitCode=0
Nov 03 01:52:29.181535 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.181452    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"
Nov 03 01:52:29.181758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.181473    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" event={"ID":"d05f8000-1622-4313-89df-5d55ce6b2f81","Type":"ContainerDied","Data":"8cd6efdb3124a34d590cbac6058aaad663954c5a1ed68edba25e8fd3e53e0a13"}
Nov 03 01:52:29.181832 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.181790    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w" event={"ID":"d05f8000-1622-4313-89df-5d55ce6b2f81","Type":"ContainerDied","Data":"cefdb248c2c3eb4346144e37b8c397bdd093c7bf94d3cecc8231470cb2a29adb"}
Nov 03 01:52:29.183094 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.183071    5508 generic.go:356] "Generic (PLEG): container finished" podID="5f420022-cf2a-44db-a72b-2934f355375e" containerID="ce85b59582911c6192c85b5cd341087b3392f46f34afec47b956b073a4276042" exitCode=0
Nov 03 01:52:29.183221 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.183096    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" event={"ID":"5f420022-cf2a-44db-a72b-2934f355375e","Type":"ContainerDied","Data":"ce85b59582911c6192c85b5cd341087b3392f46f34afec47b956b073a4276042"}
Nov 03 01:52:29.183221 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.183117    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"
Nov 03 01:52:29.183221 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.183138    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656" event={"ID":"5f420022-cf2a-44db-a72b-2934f355375e","Type":"ContainerDied","Data":"c7a52e2edfb2b5976bd64273b08510afb5b0130f30ec7d708553c7ae91e01ae8"}
Nov 03 01:52:29.188409 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.188182    5508 scope.go:122] "RemoveContainer" containerID="d11e7b0fdcc95a5ea3eacf901838f00f4e2658808823fb60db1e7c5aa9707f03"
Nov 03 01:52:29.188630 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:52:29.188495    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d11e7b0fdcc95a5ea3eacf901838f00f4e2658808823fb60db1e7c5aa9707f03\": container with ID starting with d11e7b0fdcc95a5ea3eacf901838f00f4e2658808823fb60db1e7c5aa9707f03 not found: ID does not exist" containerID="d11e7b0fdcc95a5ea3eacf901838f00f4e2658808823fb60db1e7c5aa9707f03"
Nov 03 01:52:29.188630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.188528    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11e7b0fdcc95a5ea3eacf901838f00f4e2658808823fb60db1e7c5aa9707f03"} err="failed to get container status \"d11e7b0fdcc95a5ea3eacf901838f00f4e2658808823fb60db1e7c5aa9707f03\": rpc error: code = NotFound desc = could not find container \"d11e7b0fdcc95a5ea3eacf901838f00f4e2658808823fb60db1e7c5aa9707f03\": container with ID starting with d11e7b0fdcc95a5ea3eacf901838f00f4e2658808823fb60db1e7c5aa9707f03 not found: ID does not exist"
Nov 03 01:52:29.188630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.188563    5508 scope.go:122] "RemoveContainer" containerID="ff1f253cf2f0176e052dc612a1a185b550457ce5cc1f822e386f13f04c459a67"
Nov 03 01:52:29.198980 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.198964    5508 scope.go:122] "RemoveContainer" containerID="ff1f253cf2f0176e052dc612a1a185b550457ce5cc1f822e386f13f04c459a67"
Nov 03 01:52:29.199209 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:52:29.199192    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1f253cf2f0176e052dc612a1a185b550457ce5cc1f822e386f13f04c459a67\": container with ID starting with ff1f253cf2f0176e052dc612a1a185b550457ce5cc1f822e386f13f04c459a67 not found: ID does not exist" containerID="ff1f253cf2f0176e052dc612a1a185b550457ce5cc1f822e386f13f04c459a67"
Nov 03 01:52:29.199249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.199215    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1f253cf2f0176e052dc612a1a185b550457ce5cc1f822e386f13f04c459a67"} err="failed to get container status \"ff1f253cf2f0176e052dc612a1a185b550457ce5cc1f822e386f13f04c459a67\": rpc error: code = NotFound desc = could not find container \"ff1f253cf2f0176e052dc612a1a185b550457ce5cc1f822e386f13f04c459a67\": container with ID starting with ff1f253cf2f0176e052dc612a1a185b550457ce5cc1f822e386f13f04c459a67 not found: ID does not exist"
Nov 03 01:52:29.199249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.199230    5508 scope.go:122] "RemoveContainer" containerID="8cd6efdb3124a34d590cbac6058aaad663954c5a1ed68edba25e8fd3e53e0a13"
Nov 03 01:52:29.207956 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.207925    5508 scope.go:122] "RemoveContainer" containerID="ad1c888e0d2d24111a812a20b767307a08f7f2f80093c931370712b6334825d6"
Nov 03 01:52:29.216315 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.216298    5508 scope.go:122] "RemoveContainer" containerID="8cd6efdb3124a34d590cbac6058aaad663954c5a1ed68edba25e8fd3e53e0a13"
Nov 03 01:52:29.216574 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:52:29.216555    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd6efdb3124a34d590cbac6058aaad663954c5a1ed68edba25e8fd3e53e0a13\": container with ID starting with 8cd6efdb3124a34d590cbac6058aaad663954c5a1ed68edba25e8fd3e53e0a13 not found: ID does not exist" containerID="8cd6efdb3124a34d590cbac6058aaad663954c5a1ed68edba25e8fd3e53e0a13"
Nov 03 01:52:29.216621 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.216580    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd6efdb3124a34d590cbac6058aaad663954c5a1ed68edba25e8fd3e53e0a13"} err="failed to get container status \"8cd6efdb3124a34d590cbac6058aaad663954c5a1ed68edba25e8fd3e53e0a13\": rpc error: code = NotFound desc = could not find container \"8cd6efdb3124a34d590cbac6058aaad663954c5a1ed68edba25e8fd3e53e0a13\": container with ID starting with 8cd6efdb3124a34d590cbac6058aaad663954c5a1ed68edba25e8fd3e53e0a13 not found: ID does not exist"
Nov 03 01:52:29.216621 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.216597    5508 scope.go:122] "RemoveContainer" containerID="ad1c888e0d2d24111a812a20b767307a08f7f2f80093c931370712b6334825d6"
Nov 03 01:52:29.216817 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:52:29.216799    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad1c888e0d2d24111a812a20b767307a08f7f2f80093c931370712b6334825d6\": container with ID starting with ad1c888e0d2d24111a812a20b767307a08f7f2f80093c931370712b6334825d6 not found: ID does not exist" containerID="ad1c888e0d2d24111a812a20b767307a08f7f2f80093c931370712b6334825d6"
Nov 03 01:52:29.216858 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.216824    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1c888e0d2d24111a812a20b767307a08f7f2f80093c931370712b6334825d6"} err="failed to get container status \"ad1c888e0d2d24111a812a20b767307a08f7f2f80093c931370712b6334825d6\": rpc error: code = NotFound desc = could not find container \"ad1c888e0d2d24111a812a20b767307a08f7f2f80093c931370712b6334825d6\": container with ID starting with ad1c888e0d2d24111a812a20b767307a08f7f2f80093c931370712b6334825d6 not found: ID does not exist"
Nov 03 01:52:29.216858 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.216838    5508 scope.go:122] "RemoveContainer" containerID="ce85b59582911c6192c85b5cd341087b3392f46f34afec47b956b073a4276042"
Nov 03 01:52:29.224929 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.224913    5508 scope.go:122] "RemoveContainer" containerID="ce85b59582911c6192c85b5cd341087b3392f46f34afec47b956b073a4276042"
Nov 03 01:52:29.225173 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:52:29.225157    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce85b59582911c6192c85b5cd341087b3392f46f34afec47b956b073a4276042\": container with ID starting with ce85b59582911c6192c85b5cd341087b3392f46f34afec47b956b073a4276042 not found: ID does not exist" containerID="ce85b59582911c6192c85b5cd341087b3392f46f34afec47b956b073a4276042"
Nov 03 01:52:29.225223 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.225182    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce85b59582911c6192c85b5cd341087b3392f46f34afec47b956b073a4276042"} err="failed to get container status \"ce85b59582911c6192c85b5cd341087b3392f46f34afec47b956b073a4276042\": rpc error: code = NotFound desc = could not find container \"ce85b59582911c6192c85b5cd341087b3392f46f34afec47b956b073a4276042\": container with ID starting with ce85b59582911c6192c85b5cd341087b3392f46f34afec47b956b073a4276042 not found: ID does not exist"
Nov 03 01:52:29.359086 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.358853    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"]
Nov 03 01:52:29.364241 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.364214    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-s9q4w"]
Nov 03 01:52:29.461994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.461964    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"]
Nov 03 01:52:29.469272 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.469236    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-wj656"]
Nov 03 01:52:29.570679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.570630    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"]
Nov 03 01:52:29.576998 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.576968    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-kmx46"]
Nov 03 01:52:29.678288 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.678254    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"]
Nov 03 01:52:29.684230 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:29.684200    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-knxsx"]
Nov 03 01:52:30.692402 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:30.692372    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" path="/var/lib/kubelet/pods/33c8b2c7-6e14-42dd-a9e5-457550da27ff/volumes"
Nov 03 01:52:30.692890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:30.692873    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f420022-cf2a-44db-a72b-2934f355375e" path="/var/lib/kubelet/pods/5f420022-cf2a-44db-a72b-2934f355375e/volumes"
Nov 03 01:52:30.693298 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:30.693285    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" path="/var/lib/kubelet/pods/801be76b-b7a4-468b-b72e-b6790c0bef0b/volumes"
Nov 03 01:52:30.694042 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:30.694027    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" path="/var/lib/kubelet/pods/d05f8000-1622-4313-89df-5d55ce6b2f81/volumes"
Nov 03 01:52:39.089499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.089461    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"]
Nov 03 01:52:39.090286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090269    5508 cpu_manager.go:414] "containerMap: removing container" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerName="istio-proxy"
Nov 03 01:52:39.090341 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090289    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerName="istio-proxy"
Nov 03 01:52:39.090341 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090331    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5f420022-cf2a-44db-a72b-2934f355375e" containerName="istio-proxy"
Nov 03 01:52:39.090341 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090339    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f420022-cf2a-44db-a72b-2934f355375e" containerName="istio-proxy"
Nov 03 01:52:39.090478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090365    5508 cpu_manager.go:414] "containerMap: removing container" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerName="istio-proxy"
Nov 03 01:52:39.090478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090374    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerName="istio-proxy"
Nov 03 01:52:39.090478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090409    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy"
Nov 03 01:52:39.090478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090415    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy"
Nov 03 01:52:39.090478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090428    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy"
Nov 03 01:52:39.090478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090435    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy"
Nov 03 01:52:39.090663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090564    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="801be76b-b7a4-468b-b72e-b6790c0bef0b" containerName="istio-proxy"
Nov 03 01:52:39.090663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090578    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f420022-cf2a-44db-a72b-2934f355375e" containerName="istio-proxy"
Nov 03 01:52:39.090663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090588    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c8b2c7-6e14-42dd-a9e5-457550da27ff" containerName="istio-proxy"
Nov 03 01:52:39.090663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090602    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy"
Nov 03 01:52:39.090663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.090613    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05f8000-1622-4313-89df-5d55ce6b2f81" containerName="istio-proxy"
Nov 03 01:52:39.096200 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.096182    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:39.147413 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.147389    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"]
Nov 03 01:52:39.163632 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.163607    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-maas-api-tls\") pod \"maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:39.163768 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.163667    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/721dd14b-55de-4342-a676-e06e3d8ae2e3-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:39.163809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.163758    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-248lz\" (UniqueName: \"kubernetes.io/projected/721dd14b-55de-4342-a676-e06e3d8ae2e3-kube-api-access-248lz\") pod \"maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:39.163852 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.163816    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-metrics-certs\") pod \"maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:39.189691 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.189667    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0e0d61-serving-cert\""
Nov 03 01:52:39.264842 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.264810    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-maas-api-tls\") pod \"maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:39.265025 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.264860    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/721dd14b-55de-4342-a676-e06e3d8ae2e3-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:39.265025 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.264892    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-248lz\" (UniqueName: \"kubernetes.io/projected/721dd14b-55de-4342-a676-e06e3d8ae2e3-kube-api-access-248lz\") pod \"maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:39.265025 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.264924    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-metrics-certs\") pod \"maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:39.265025 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:52:39.264962    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-serving-cert: secret "maas-api-e2e-api-b-0e0d61-serving-cert" not found
Nov 03 01:52:39.265255 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:52:39.265092    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-maas-api-tls podName:721dd14b-55de-4342-a676-e06e3d8ae2e3 nodeName:}" failed. No retries permitted until 2026-11-03 01:52:39.765070237 +0000 UTC m=+2039.190676696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-maas-api-tls") pod "maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c" (UID: "721dd14b-55de-4342-a676-e06e3d8ae2e3") : secret "maas-api-e2e-api-b-0e0d61-serving-cert" not found
Nov 03 01:52:39.265667 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.265642    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/721dd14b-55de-4342-a676-e06e3d8ae2e3-openshift-service-ca\") pod \"maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:39.267256 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.267233    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-metrics-certs\") pod \"maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:39.340647 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.340567    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-248lz\" (UniqueName: \"kubernetes.io/projected/721dd14b-55de-4342-a676-e06e3d8ae2e3-kube-api-access-248lz\") pod \"maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:39.789248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.789204    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-maas-api-tls\") pod \"maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:39.791722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:39.791694    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-maas-api-tls\") pod \"maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:40.010950 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:40.010919    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:40.391454 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:40.391378    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"]
Nov 03 01:52:40.399470 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:52:40.399439    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721dd14b_55de_4342_a676_e06e3d8ae2e3.slice/crio-21aebd959be8b71601559436f0d289978053fd9971dccae2bfb9d91f05aa421a WatchSource:0}: Error finding container 21aebd959be8b71601559436f0d289978053fd9971dccae2bfb9d91f05aa421a: Status 404 returned error can't find the container with id 21aebd959be8b71601559436f0d289978053fd9971dccae2bfb9d91f05aa421a
Nov 03 01:52:40.400323 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:40.400303    5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:52:41.274687 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:41.274652    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c" event={"ID":"721dd14b-55de-4342-a676-e06e3d8ae2e3","Type":"ContainerStarted","Data":"21aebd959be8b71601559436f0d289978053fd9971dccae2bfb9d91f05aa421a"}
Nov 03 01:52:42.282691 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:42.282657    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c" event={"ID":"721dd14b-55de-4342-a676-e06e3d8ae2e3","Type":"ContainerStarted","Data":"5b8a30c2784bb2fcc7fdad1f45429855e3ba8afdb1fee6059c2b4c552178dd34"}
Nov 03 01:52:42.283115 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:42.282859    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:42.395113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:42.395053    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c" podStartSLOduration=2.441962848 podStartE2EDuration="3.395035057s" totalImagesPullingTime="953.072209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:42.393196015 +0000 UTC m=+2041.818802479" watchObservedRunningTime="2026-11-03 01:52:42.395035057 +0000 UTC m=+2041.820641521"
Nov 03 01:52:43.712356 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:43.712312    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:52:43.811010 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:52:43.810977    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:52:44.544454 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:44.544418    5508 scope.go:122] "RemoveContainer" containerID="4c6587d9f4198080c233977e6219a2605e9f813228cc126ee40647aeeddde026"
Nov 03 01:52:48.294616 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:48.294588    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:52:59.167022 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.166987    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"]
Nov 03 01:52:59.171060 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.171040    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"
Nov 03 01:52:59.240654 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.240622    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt5h5\" (UniqueName: \"kubernetes.io/projected/15bdec5e-7ba9-48c7-a3b8-2e1488567579-kube-api-access-zt5h5\") pod \"payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62\" (UID: \"15bdec5e-7ba9-48c7-a3b8-2e1488567579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"
Nov 03 01:52:59.240814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.240786    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bdec5e-7ba9-48c7-a3b8-2e1488567579-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62\" (UID: \"15bdec5e-7ba9-48c7-a3b8-2e1488567579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"
Nov 03 01:52:59.267079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.267055    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-86e679-dockercfg-bwx2j\""
Nov 03 01:52:59.267264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.267055    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-86e679\""
Nov 03 01:52:59.270984 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.270963    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"]
Nov 03 01:52:59.342241 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.342216    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt5h5\" (UniqueName: \"kubernetes.io/projected/15bdec5e-7ba9-48c7-a3b8-2e1488567579-kube-api-access-zt5h5\") pod \"payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62\" (UID: \"15bdec5e-7ba9-48c7-a3b8-2e1488567579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"
Nov 03 01:52:59.342370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.342326    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bdec5e-7ba9-48c7-a3b8-2e1488567579-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62\" (UID: \"15bdec5e-7ba9-48c7-a3b8-2e1488567579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"
Nov 03 01:52:59.342978 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.342960    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bdec5e-7ba9-48c7-a3b8-2e1488567579-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62\" (UID: \"15bdec5e-7ba9-48c7-a3b8-2e1488567579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"
Nov 03 01:52:59.399256 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.399224    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt5h5\" (UniqueName: \"kubernetes.io/projected/15bdec5e-7ba9-48c7-a3b8-2e1488567579-kube-api-access-zt5h5\") pod \"payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62\" (UID: \"15bdec5e-7ba9-48c7-a3b8-2e1488567579\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"
Nov 03 01:52:59.484467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.484392    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"
Nov 03 01:52:59.860903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.860825    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"]
Nov 03 01:52:59.866944 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:52:59.866917    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15bdec5e_7ba9_48c7_a3b8_2e1488567579.slice/crio-483cff41bf8967a7a7aa567b76dcdf8435943d98ea1917c182693dde307ce5a6 WatchSource:0}: Error finding container 483cff41bf8967a7a7aa567b76dcdf8435943d98ea1917c182693dde307ce5a6: Status 404 returned error can't find the container with id 483cff41bf8967a7a7aa567b76dcdf8435943d98ea1917c182693dde307ce5a6
Nov 03 01:52:59.984149 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.984122    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"]
Nov 03 01:52:59.988139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:52:59.988119    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"
Nov 03 01:53:00.044426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:00.044392    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"]
Nov 03 01:53:00.054367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:00.050956    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a3f669-6790-4ba7-b21d-abee6025f055-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n\" (UID: \"d5a3f669-6790-4ba7-b21d-abee6025f055\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"
Nov 03 01:53:00.054367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:00.051046    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrlch\" (UniqueName: \"kubernetes.io/projected/d5a3f669-6790-4ba7-b21d-abee6025f055-kube-api-access-nrlch\") pod \"payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n\" (UID: \"d5a3f669-6790-4ba7-b21d-abee6025f055\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"
Nov 03 01:53:00.152563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:00.152529    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a3f669-6790-4ba7-b21d-abee6025f055-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n\" (UID: \"d5a3f669-6790-4ba7-b21d-abee6025f055\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"
Nov 03 01:53:00.152753 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:00.152589    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrlch\" (UniqueName: \"kubernetes.io/projected/d5a3f669-6790-4ba7-b21d-abee6025f055-kube-api-access-nrlch\") pod \"payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n\" (UID: \"d5a3f669-6790-4ba7-b21d-abee6025f055\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"
Nov 03 01:53:00.153237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:00.153211    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a3f669-6790-4ba7-b21d-abee6025f055-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n\" (UID: \"d5a3f669-6790-4ba7-b21d-abee6025f055\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"
Nov 03 01:53:00.209823 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:00.209794    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrlch\" (UniqueName: \"kubernetes.io/projected/d5a3f669-6790-4ba7-b21d-abee6025f055-kube-api-access-nrlch\") pod \"payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n\" (UID: \"d5a3f669-6790-4ba7-b21d-abee6025f055\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"
Nov 03 01:53:00.302577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:00.302545    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"
Nov 03 01:53:00.416399 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:00.415005    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62" event={"ID":"15bdec5e-7ba9-48c7-a3b8-2e1488567579","Type":"ContainerStarted","Data":"c94b03d34ca0727b64d5700469dfea1a338ed7efb3955469292b81472fe52374"}
Nov 03 01:53:00.416399 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:00.415105    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62" event={"ID":"15bdec5e-7ba9-48c7-a3b8-2e1488567579","Type":"ContainerStarted","Data":"483cff41bf8967a7a7aa567b76dcdf8435943d98ea1917c182693dde307ce5a6"}
Nov 03 01:53:00.416399 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:00.415148    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"
Nov 03 01:53:00.525215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:00.525168    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62" podStartSLOduration=1.5251523329999999 podStartE2EDuration="1.525152333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:53:00.523364961 +0000 UTC m=+2059.948971439" watchObservedRunningTime="2026-11-03 01:53:00.525152333 +0000 UTC m=+2059.950758839"
Nov 03 01:53:00.695713 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:00.695624    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"]
Nov 03 01:53:00.701818 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:53:00.701786    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a3f669_6790_4ba7_b21d_abee6025f055.slice/crio-3c69163a1a77c5042e36eb8c79388bfb20cefdb8c6c34917ab43981c09b6aa50 WatchSource:0}: Error finding container 3c69163a1a77c5042e36eb8c79388bfb20cefdb8c6c34917ab43981c09b6aa50: Status 404 returned error can't find the container with id 3c69163a1a77c5042e36eb8c79388bfb20cefdb8c6c34917ab43981c09b6aa50
Nov 03 01:53:01.456374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:01.455693    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n" event={"ID":"d5a3f669-6790-4ba7-b21d-abee6025f055","Type":"ContainerStarted","Data":"b600c24ccaa948c5525cfddd6745bc3de57fb5f8bf9ee0d34b71bb618825ba05"}
Nov 03 01:53:01.456374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:01.455755    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n" event={"ID":"d5a3f669-6790-4ba7-b21d-abee6025f055","Type":"ContainerStarted","Data":"3c69163a1a77c5042e36eb8c79388bfb20cefdb8c6c34917ab43981c09b6aa50"}
Nov 03 01:53:01.456374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:01.455812    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"
Nov 03 01:53:01.566195 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:01.566134    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n" podStartSLOduration=2.56611983 podStartE2EDuration="2.56611983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:53:01.563869898 +0000 UTC m=+2060.989476388" watchObservedRunningTime="2026-11-03 01:53:01.56611983 +0000 UTC m=+2060.991726294"
Nov 03 01:53:11.457977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:11.457948    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"
Nov 03 01:53:12.542287 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:12.542253    5508 generic.go:356] "Generic (PLEG): container finished" podID="063d1af5-8267-423a-be4b-a46a27c2e527" containerID="87f7c728154202051c6b1aa4e67e4b70baaa270305ded414b8430ad15ae076d5" exitCode=0
Nov 03 01:53:12.542717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:12.542300    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv" event={"ID":"063d1af5-8267-423a-be4b-a46a27c2e527","Type":"ContainerDied","Data":"87f7c728154202051c6b1aa4e67e4b70baaa270305ded414b8430ad15ae076d5"}
Nov 03 01:53:12.542717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:12.542702    5508 scope.go:122] "RemoveContainer" containerID="87f7c728154202051c6b1aa4e67e4b70baaa270305ded414b8430ad15ae076d5"
Nov 03 01:53:12.544095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:12.544063    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:53:12.544210 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:12.544145    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:53:12.544274 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:12.544213    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:53:13.483001 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:13.482962    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"
Nov 03 01:53:13.582437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:13.582342    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv" event={"ID":"063d1af5-8267-423a-be4b-a46a27c2e527","Type":"ContainerStarted","Data":"5dc32cabaac82471c5a28b2d74e4cddfd8febbaf07c86cee2081d8fb29578fcc"}
Nov 03 01:53:13.813150 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:53:13.813073    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:53:14.306724 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:14.306693    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:53:14.306903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:14.306837    5508 patch_prober.go:29] interesting pod/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 01:53:14.306952 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:14.306897    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv" podUID="063d1af5-8267-423a-be4b-a46a27c2e527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 01:53:15.306512 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:15.306474    5508 patch_prober.go:29] interesting pod/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 01:53:15.306909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:15.306548    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv" podUID="063d1af5-8267-423a-be4b-a46a27c2e527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 01:53:16.306441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:16.306331    5508 patch_prober.go:29] interesting pod/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 01:53:16.306441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:16.306427    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv" podUID="063d1af5-8267-423a-be4b-a46a27c2e527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 01:53:17.306901 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:17.306866    5508 patch_prober.go:29] interesting pod/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 01:53:17.307327 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:17.306928    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv" podUID="063d1af5-8267-423a-be4b-a46a27c2e527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 01:53:18.306306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:18.306275    5508 patch_prober.go:29] interesting pod/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 03 01:53:18.306521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:18.306322    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv" podUID="063d1af5-8267-423a-be4b-a46a27c2e527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 03 01:53:19.312516 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:19.312482    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:53:19.312994 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:19.312867    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:53:19.313591 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:19.313571    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:53:19.851719 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:19.851687    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:53:21.958225 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:21.958195    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"]
Nov 03 01:53:21.963486 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:21.963462    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"
Nov 03 01:53:22.024166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.024134    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5df5j\" (UniqueName: \"kubernetes.io/projected/f93be790-6233-4f77-b8c5-5f289079cd50-kube-api-access-5df5j\") pod \"payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc\" (UID: \"f93be790-6233-4f77-b8c5-5f289079cd50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"
Nov 03 01:53:22.024296 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.024187    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93be790-6233-4f77-b8c5-5f289079cd50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc\" (UID: \"f93be790-6233-4f77-b8c5-5f289079cd50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"
Nov 03 01:53:22.058199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.058170    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b177c9\""
Nov 03 01:53:22.058329 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.058232    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b177c9-dockercfg-llkd7\""
Nov 03 01:53:22.061552 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.061530    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"]
Nov 03 01:53:22.125678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.125651    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93be790-6233-4f77-b8c5-5f289079cd50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc\" (UID: \"f93be790-6233-4f77-b8c5-5f289079cd50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"
Nov 03 01:53:22.125814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.125791    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5df5j\" (UniqueName: \"kubernetes.io/projected/f93be790-6233-4f77-b8c5-5f289079cd50-kube-api-access-5df5j\") pod \"payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc\" (UID: \"f93be790-6233-4f77-b8c5-5f289079cd50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"
Nov 03 01:53:22.126343 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.126325    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93be790-6233-4f77-b8c5-5f289079cd50-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc\" (UID: \"f93be790-6233-4f77-b8c5-5f289079cd50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"
Nov 03 01:53:22.182581 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.182559    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5df5j\" (UniqueName: \"kubernetes.io/projected/f93be790-6233-4f77-b8c5-5f289079cd50-kube-api-access-5df5j\") pod \"payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc\" (UID: \"f93be790-6233-4f77-b8c5-5f289079cd50\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"
Nov 03 01:53:22.277013 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.276954    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"
Nov 03 01:53:22.653467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.653436    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"]
Nov 03 01:53:22.660443 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:53:22.660393    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93be790_6233_4f77_b8c5_5f289079cd50.slice/crio-7fd0f77e8b8a316051a8d29ec4128e908c452f0b0f3329c9fceb5ca63cf49fab WatchSource:0}: Error finding container 7fd0f77e8b8a316051a8d29ec4128e908c452f0b0f3329c9fceb5ca63cf49fab: Status 404 returned error can't find the container with id 7fd0f77e8b8a316051a8d29ec4128e908c452f0b0f3329c9fceb5ca63cf49fab
Nov 03 01:53:22.679694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.679664    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc" event={"ID":"f93be790-6233-4f77-b8c5-5f289079cd50","Type":"ContainerStarted","Data":"7fd0f77e8b8a316051a8d29ec4128e908c452f0b0f3329c9fceb5ca63cf49fab"}
Nov 03 01:53:22.788210 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.788181    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"]
Nov 03 01:53:22.792157 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.792140    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"
Nov 03 01:53:22.859521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.857603    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"]
Nov 03 01:53:22.938485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.938403    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwntw\" (UniqueName: \"kubernetes.io/projected/f9f60d09-e74d-432e-84ef-b74af7c18842-kube-api-access-wwntw\") pod \"payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv\" (UID: \"f9f60d09-e74d-432e-84ef-b74af7c18842\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"
Nov 03 01:53:22.938485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:22.938479    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f60d09-e74d-432e-84ef-b74af7c18842-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv\" (UID: \"f9f60d09-e74d-432e-84ef-b74af7c18842\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"
Nov 03 01:53:23.039946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:23.039910    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwntw\" (UniqueName: \"kubernetes.io/projected/f9f60d09-e74d-432e-84ef-b74af7c18842-kube-api-access-wwntw\") pod \"payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv\" (UID: \"f9f60d09-e74d-432e-84ef-b74af7c18842\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"
Nov 03 01:53:23.040310 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:23.039969    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f60d09-e74d-432e-84ef-b74af7c18842-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv\" (UID: \"f9f60d09-e74d-432e-84ef-b74af7c18842\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"
Nov 03 01:53:23.040647 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:23.040627    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f60d09-e74d-432e-84ef-b74af7c18842-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv\" (UID: \"f9f60d09-e74d-432e-84ef-b74af7c18842\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"
Nov 03 01:53:23.097043 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:23.097012    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwntw\" (UniqueName: \"kubernetes.io/projected/f9f60d09-e74d-432e-84ef-b74af7c18842-kube-api-access-wwntw\") pod \"payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv\" (UID: \"f9f60d09-e74d-432e-84ef-b74af7c18842\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"
Nov 03 01:53:23.105386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:23.105360    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"
Nov 03 01:53:23.476885 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:23.476837    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"]
Nov 03 01:53:23.483228 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:53:23.483201    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f60d09_e74d_432e_84ef_b74af7c18842.slice/crio-b9482f254fe384b58aece619416e19b30f156bc80b99da533f8a739852f76c6c WatchSource:0}: Error finding container b9482f254fe384b58aece619416e19b30f156bc80b99da533f8a739852f76c6c: Status 404 returned error can't find the container with id b9482f254fe384b58aece619416e19b30f156bc80b99da533f8a739852f76c6c
Nov 03 01:53:23.688838 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:23.688805    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc" event={"ID":"f93be790-6233-4f77-b8c5-5f289079cd50","Type":"ContainerStarted","Data":"feb361101af9cce6df95c418560ab3b90fd9a87dd13925b97b3aff330d7c1b1a"}
Nov 03 01:53:23.689017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:23.688883    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"
Nov 03 01:53:23.690650 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:23.690622    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv" event={"ID":"f9f60d09-e74d-432e-84ef-b74af7c18842","Type":"ContainerStarted","Data":"e0bc29baca70abff81f4ce0127576b10bec3d5830c7b229dbe61e16fe145f351"}
Nov 03 01:53:23.690781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:23.690667    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv" event={"ID":"f9f60d09-e74d-432e-84ef-b74af7c18842","Type":"ContainerStarted","Data":"b9482f254fe384b58aece619416e19b30f156bc80b99da533f8a739852f76c6c"}
Nov 03 01:53:23.690781 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:23.690686    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"
Nov 03 01:53:23.800727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:23.800618    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc" podStartSLOduration=2.800601624 podStartE2EDuration="2.800601624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:53:23.79717 +0000 UTC m=+2083.222776464" watchObservedRunningTime="2026-11-03 01:53:23.800601624 +0000 UTC m=+2083.226208089"
Nov 03 01:53:23.907149 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:23.907093    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv" podStartSLOduration=1.907076569 podStartE2EDuration="1.907076569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:53:23.906092942 +0000 UTC m=+2083.331699419" watchObservedRunningTime="2026-11-03 01:53:23.907076569 +0000 UTC m=+2083.332683034"
Nov 03 01:53:34.699096 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:34.699054    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"
Nov 03 01:53:34.699499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:34.699343    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"
Nov 03 01:53:41.100698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:41.100671    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:53:41.150583 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:41.150554    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:53:43.813130 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:53:43.813097    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:53:53.721622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:53.721583    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:53:54.096581 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.096503    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"]
Nov 03 01:53:54.096743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.096689    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d" podUID="8a092c54-5ed5-4bb9-b65b-e80a1a5122cb" containerName="maas-api" containerID="cri-o://ccf0659b9a9b7611d0d4bed59fb260d2868cdbc504048e24ecebc79f56444ae2" gracePeriod=30
Nov 03 01:53:54.420914 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.420890    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:53:54.531329 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.531295    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"]
Nov 03 01:53:54.531575 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.531518    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp" podUID="31af0199-a45d-4302-ac71-7e6b9e7ed8b0" containerName="payload-processing" containerID="cri-o://f181a77f0fce18ad8106bd4a0e34c20d96bea612158fa89bf08f0ffcd19be728" gracePeriod=30
Nov 03 01:53:54.554460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.554437    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-maas-api-tls\") pod \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") "
Nov 03 01:53:54.554599 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.554471    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-openshift-service-ca\") pod \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") "
Nov 03 01:53:54.554599 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.554510    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-kube-api-access-xl4tw\" (UniqueName: \"kubernetes.io/projected/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-kube-api-access-xl4tw\") pod \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") "
Nov 03 01:53:54.554724 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.554657    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-metrics-certs\") pod \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\" (UID: \"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb\") "
Nov 03 01:53:54.554819 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.554795    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-openshift-service-ca" pod "8a092c54-5ed5-4bb9-b65b-e80a1a5122cb" (UID: "8a092c54-5ed5-4bb9-b65b-e80a1a5122cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:53:54.555129 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.555109    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:53:54.556953 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.556931    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-maas-api-tls" pod "8a092c54-5ed5-4bb9-b65b-e80a1a5122cb" (UID: "8a092c54-5ed5-4bb9-b65b-e80a1a5122cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:53:54.557045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.556966    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-kube-api-access-xl4tw" pod "8a092c54-5ed5-4bb9-b65b-e80a1a5122cb" (UID: "8a092c54-5ed5-4bb9-b65b-e80a1a5122cb"). InnerVolumeSpecName "kube-api-access-xl4tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:53:54.557045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.557022    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-metrics-certs" pod "8a092c54-5ed5-4bb9-b65b-e80a1a5122cb" (UID: "8a092c54-5ed5-4bb9-b65b-e80a1a5122cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:53:54.585214 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.585186    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"]
Nov 03 01:53:54.585443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.585420    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t" podUID="c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d" containerName="payload-pre-processing" containerID="cri-o://f9c0f29fd459ac07e442f9e570362e58b5d51f492c62e033e9cc06b258d9dcc8" gracePeriod=30
Nov 03 01:53:54.656217 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.656166    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:53:54.656217 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.656195    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:53:54.656217 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.656205    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl4tw\" (UniqueName: \"kubernetes.io/projected/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb-kube-api-access-xl4tw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:53:54.847674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.847642    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"
Nov 03 01:53:54.881184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.881161    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"
Nov 03 01:53:54.936521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.936449    5508 generic.go:356] "Generic (PLEG): container finished" podID="8a092c54-5ed5-4bb9-b65b-e80a1a5122cb" containerID="ccf0659b9a9b7611d0d4bed59fb260d2868cdbc504048e24ecebc79f56444ae2" exitCode=0
Nov 03 01:53:54.936521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.936498    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"
Nov 03 01:53:54.936718 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.936538    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d" event={"ID":"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb","Type":"ContainerDied","Data":"ccf0659b9a9b7611d0d4bed59fb260d2868cdbc504048e24ecebc79f56444ae2"}
Nov 03 01:53:54.936718 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.936582    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d" event={"ID":"8a092c54-5ed5-4bb9-b65b-e80a1a5122cb","Type":"ContainerDied","Data":"57e84b00381bd033e70e248ba0d2009ccb20cf98b9d7ab171752998021cf46b1"}
Nov 03 01:53:54.936718 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.936604    5508 scope.go:122] "RemoveContainer" containerID="ccf0659b9a9b7611d0d4bed59fb260d2868cdbc504048e24ecebc79f56444ae2"
Nov 03 01:53:54.937986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.937964    5508 generic.go:356] "Generic (PLEG): container finished" podID="c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d" containerID="f9c0f29fd459ac07e442f9e570362e58b5d51f492c62e033e9cc06b258d9dcc8" exitCode=0
Nov 03 01:53:54.938104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.938015    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"
Nov 03 01:53:54.938104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.938049    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t" event={"ID":"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d","Type":"ContainerDied","Data":"f9c0f29fd459ac07e442f9e570362e58b5d51f492c62e033e9cc06b258d9dcc8"}
Nov 03 01:53:54.938104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.938089    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t" event={"ID":"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d","Type":"ContainerDied","Data":"a5904ec9d18c534805213860dd05fd0e5e6550255ba7fb9e39899567163d18ef"}
Nov 03 01:53:54.939591 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.939566    5508 generic.go:356] "Generic (PLEG): container finished" podID="31af0199-a45d-4302-ac71-7e6b9e7ed8b0" containerID="f181a77f0fce18ad8106bd4a0e34c20d96bea612158fa89bf08f0ffcd19be728" exitCode=0
Nov 03 01:53:54.939676 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.939599    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"
Nov 03 01:53:54.939676 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.939631    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp" event={"ID":"31af0199-a45d-4302-ac71-7e6b9e7ed8b0","Type":"ContainerDied","Data":"f181a77f0fce18ad8106bd4a0e34c20d96bea612158fa89bf08f0ffcd19be728"}
Nov 03 01:53:54.939676 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.939667    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp" event={"ID":"31af0199-a45d-4302-ac71-7e6b9e7ed8b0","Type":"ContainerDied","Data":"b8601b5419e00aadcd98bf0d8bfb67bbf9b137308983df5ac57bccac912cfbf7"}
Nov 03 01:53:54.946273 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.946258    5508 scope.go:122] "RemoveContainer" containerID="ccf0659b9a9b7611d0d4bed59fb260d2868cdbc504048e24ecebc79f56444ae2"
Nov 03 01:53:54.946552 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:53:54.946537    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf0659b9a9b7611d0d4bed59fb260d2868cdbc504048e24ecebc79f56444ae2\": container with ID starting with ccf0659b9a9b7611d0d4bed59fb260d2868cdbc504048e24ecebc79f56444ae2 not found: ID does not exist" containerID="ccf0659b9a9b7611d0d4bed59fb260d2868cdbc504048e24ecebc79f56444ae2"
Nov 03 01:53:54.946613 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.946558    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf0659b9a9b7611d0d4bed59fb260d2868cdbc504048e24ecebc79f56444ae2"} err="failed to get container status \"ccf0659b9a9b7611d0d4bed59fb260d2868cdbc504048e24ecebc79f56444ae2\": rpc error: code = NotFound desc = could not find container \"ccf0659b9a9b7611d0d4bed59fb260d2868cdbc504048e24ecebc79f56444ae2\": container with ID starting with ccf0659b9a9b7611d0d4bed59fb260d2868cdbc504048e24ecebc79f56444ae2 not found: ID does not exist"
Nov 03 01:53:54.946613 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.946578    5508 scope.go:122] "RemoveContainer" containerID="f9c0f29fd459ac07e442f9e570362e58b5d51f492c62e033e9cc06b258d9dcc8"
Nov 03 01:53:54.955240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.955223    5508 scope.go:122] "RemoveContainer" containerID="f9c0f29fd459ac07e442f9e570362e58b5d51f492c62e033e9cc06b258d9dcc8"
Nov 03 01:53:54.955514 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:53:54.955496    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c0f29fd459ac07e442f9e570362e58b5d51f492c62e033e9cc06b258d9dcc8\": container with ID starting with f9c0f29fd459ac07e442f9e570362e58b5d51f492c62e033e9cc06b258d9dcc8 not found: ID does not exist" containerID="f9c0f29fd459ac07e442f9e570362e58b5d51f492c62e033e9cc06b258d9dcc8"
Nov 03 01:53:54.955586 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.955521    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c0f29fd459ac07e442f9e570362e58b5d51f492c62e033e9cc06b258d9dcc8"} err="failed to get container status \"f9c0f29fd459ac07e442f9e570362e58b5d51f492c62e033e9cc06b258d9dcc8\": rpc error: code = NotFound desc = could not find container \"f9c0f29fd459ac07e442f9e570362e58b5d51f492c62e033e9cc06b258d9dcc8\": container with ID starting with f9c0f29fd459ac07e442f9e570362e58b5d51f492c62e033e9cc06b258d9dcc8 not found: ID does not exist"
Nov 03 01:53:54.955586 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.955535    5508 scope.go:122] "RemoveContainer" containerID="f181a77f0fce18ad8106bd4a0e34c20d96bea612158fa89bf08f0ffcd19be728"
Nov 03 01:53:54.959808 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.959785    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-kube-api-access-z2hxd\" (UniqueName: \"kubernetes.io/projected/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-kube-api-access-z2hxd\") pod \"31af0199-a45d-4302-ac71-7e6b9e7ed8b0\" (UID: \"31af0199-a45d-4302-ac71-7e6b9e7ed8b0\") "
Nov 03 01:53:54.960478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.960455    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-plugins-config-volume\") pod \"31af0199-a45d-4302-ac71-7e6b9e7ed8b0\" (UID: \"31af0199-a45d-4302-ac71-7e6b9e7ed8b0\") "
Nov 03 01:53:54.960986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.960967    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-plugins-config-volume" pod "31af0199-a45d-4302-ac71-7e6b9e7ed8b0" (UID: "31af0199-a45d-4302-ac71-7e6b9e7ed8b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:53:54.961121 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.961096    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:53:54.962454 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.962433    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-kube-api-access-z2hxd" pod "31af0199-a45d-4302-ac71-7e6b9e7ed8b0" (UID: "31af0199-a45d-4302-ac71-7e6b9e7ed8b0"). InnerVolumeSpecName "kube-api-access-z2hxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:53:54.966555 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.966537    5508 scope.go:122] "RemoveContainer" containerID="f181a77f0fce18ad8106bd4a0e34c20d96bea612158fa89bf08f0ffcd19be728"
Nov 03 01:53:54.966846 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:53:54.966826    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f181a77f0fce18ad8106bd4a0e34c20d96bea612158fa89bf08f0ffcd19be728\": container with ID starting with f181a77f0fce18ad8106bd4a0e34c20d96bea612158fa89bf08f0ffcd19be728 not found: ID does not exist" containerID="f181a77f0fce18ad8106bd4a0e34c20d96bea612158fa89bf08f0ffcd19be728"
Nov 03 01:53:54.966947 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:54.966854    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f181a77f0fce18ad8106bd4a0e34c20d96bea612158fa89bf08f0ffcd19be728"} err="failed to get container status \"f181a77f0fce18ad8106bd4a0e34c20d96bea612158fa89bf08f0ffcd19be728\": rpc error: code = NotFound desc = could not find container \"f181a77f0fce18ad8106bd4a0e34c20d96bea612158fa89bf08f0ffcd19be728\": container with ID starting with f181a77f0fce18ad8106bd4a0e34c20d96bea612158fa89bf08f0ffcd19be728 not found: ID does not exist"
Nov 03 01:53:55.063371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:55.062800    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-plugins-config-volume\") pod \"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d\" (UID: \"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d\") "
Nov 03 01:53:55.063371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:55.063217    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-plugins-config-volume" pod "c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d" (UID: "c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:53:55.063371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:55.063335    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-kube-api-access-z9rbf\" (UniqueName: \"kubernetes.io/projected/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-kube-api-access-z9rbf\") pod \"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d\" (UID: \"c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d\") "
Nov 03 01:53:55.077367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:55.063938    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2hxd\" (UniqueName: \"kubernetes.io/projected/31af0199-a45d-4302-ac71-7e6b9e7ed8b0-kube-api-access-z2hxd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:53:55.077367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:55.063963    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:53:55.077367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:55.072768    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-kube-api-access-z9rbf" pod "c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d" (UID: "c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d"). InnerVolumeSpecName "kube-api-access-z9rbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:53:55.091727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:55.088459    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"]
Nov 03 01:53:55.099389 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:55.099023    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7bcbbb-c5cf9c6d6-lt84d"]
Nov 03 01:53:55.165555 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:55.165527    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9rbf\" (UniqueName: \"kubernetes.io/projected/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d-kube-api-access-z9rbf\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:53:55.354399 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:55.354368    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"]
Nov 03 01:53:55.358857 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:55.358832    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7bcbbb-c74dfb847-q282t"]
Nov 03 01:53:55.470307 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:55.470275    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"]
Nov 03 01:53:55.473833 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:55.473810    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7bcbbb-795b5cfd74-x2szp"]
Nov 03 01:53:56.693751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:56.693716    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31af0199-a45d-4302-ac71-7e6b9e7ed8b0" path="/var/lib/kubelet/pods/31af0199-a45d-4302-ac71-7e6b9e7ed8b0/volumes"
Nov 03 01:53:56.694184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:56.694170    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a092c54-5ed5-4bb9-b65b-e80a1a5122cb" path="/var/lib/kubelet/pods/8a092c54-5ed5-4bb9-b65b-e80a1a5122cb/volumes"
Nov 03 01:53:56.694586 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:53:56.694573    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d" path="/var/lib/kubelet/pods/c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d/volumes"
Nov 03 01:54:00.317779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:00.317747    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"]
Nov 03 01:54:00.318239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:00.318029    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerName="istio-proxy" containerID="cri-o://fb4544c9d3253a50e2a6dd4bacc035ab07e94b48881884a3c7d06d134edc449c" gracePeriod=30
Nov 03 01:54:04.160868 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:04.160833    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"]
Nov 03 01:54:04.161439 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:04.161045    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy" containerID="cri-o://a8d01124001ebf7cd1352b7da1a5ecd2e05b15ef3bb2fbc33efd3cf2ed1aa553" gracePeriod=30
Nov 03 01:54:04.347377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:04.347331    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:54:04.347576 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:04.347403    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:54:05.595376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.595298    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:54:05.703374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.702423    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-token\" (UniqueName: \"kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-token\") pod \"872fb22d-056c-4e0c-9555-e285d73cd87f\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") "
Nov 03 01:54:05.703374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.702502    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-envoy\") pod \"872fb22d-056c-4e0c-9555-e285d73cd87f\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") "
Nov 03 01:54:05.703374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.702679    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872fb22d-056c-4e0c-9555-e285d73cd87f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872fb22d-056c-4e0c-9555-e285d73cd87f-istiod-ca-cert\") pod \"872fb22d-056c-4e0c-9555-e285d73cd87f\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") "
Nov 03 01:54:05.703374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.702754    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-data\") pod \"872fb22d-056c-4e0c-9555-e285d73cd87f\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") "
Nov 03 01:54:05.703374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.702845    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-certs\") pod \"872fb22d-056c-4e0c-9555-e285d73cd87f\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") "
Nov 03 01:54:05.703374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.702907    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-podinfo\") pod \"872fb22d-056c-4e0c-9555-e285d73cd87f\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") "
Nov 03 01:54:05.703374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.703027    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-socket\") pod \"872fb22d-056c-4e0c-9555-e285d73cd87f\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") "
Nov 03 01:54:05.703374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.703067    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-kube-api-access-6pz7z\" (UniqueName: \"kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-kube-api-access-6pz7z\") pod \"872fb22d-056c-4e0c-9555-e285d73cd87f\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") "
Nov 03 01:54:05.703374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.703122    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-credential-socket\") pod \"872fb22d-056c-4e0c-9555-e285d73cd87f\" (UID: \"872fb22d-056c-4e0c-9555-e285d73cd87f\") "
Nov 03 01:54:05.704002 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.703702    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-data" pod "872fb22d-056c-4e0c-9555-e285d73cd87f" (UID: "872fb22d-056c-4e0c-9555-e285d73cd87f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:05.704535 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.704508    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-certs" pod "872fb22d-056c-4e0c-9555-e285d73cd87f" (UID: "872fb22d-056c-4e0c-9555-e285d73cd87f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:05.704832 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.704796    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-credential-socket" pod "872fb22d-056c-4e0c-9555-e285d73cd87f" (UID: "872fb22d-056c-4e0c-9555-e285d73cd87f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:05.704960 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.704937    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:05.705028 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.704968    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:05.705275 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.705241    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-socket" pod "872fb22d-056c-4e0c-9555-e285d73cd87f" (UID: "872fb22d-056c-4e0c-9555-e285d73cd87f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:05.705655 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.705632    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872fb22d-056c-4e0c-9555-e285d73cd87f-istiod-ca-cert" pod "872fb22d-056c-4e0c-9555-e285d73cd87f" (UID: "872fb22d-056c-4e0c-9555-e285d73cd87f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:54:05.718550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.718520    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-kube-api-access-6pz7z" pod "872fb22d-056c-4e0c-9555-e285d73cd87f" (UID: "872fb22d-056c-4e0c-9555-e285d73cd87f"). InnerVolumeSpecName "kube-api-access-6pz7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:05.721903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.721877    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-envoy" pod "872fb22d-056c-4e0c-9555-e285d73cd87f" (UID: "872fb22d-056c-4e0c-9555-e285d73cd87f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:05.722196 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.722142    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-token" pod "872fb22d-056c-4e0c-9555-e285d73cd87f" (UID: "872fb22d-056c-4e0c-9555-e285d73cd87f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:05.727511 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.726754    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-podinfo" pod "872fb22d-056c-4e0c-9555-e285d73cd87f" (UID: "872fb22d-056c-4e0c-9555-e285d73cd87f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:54:05.806091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.806056    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:05.806091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.806091    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:05.806325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.806105    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:05.806325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.806120    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pz7z\" (UniqueName: \"kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-kube-api-access-6pz7z\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:05.806325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.806131    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:05.806325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.806142    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872fb22d-056c-4e0c-9555-e285d73cd87f-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:05.806325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:05.806155    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872fb22d-056c-4e0c-9555-e285d73cd87f-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:06.035062 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:06.035031    5508 generic.go:356] "Generic (PLEG): container finished" podID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerID="fb4544c9d3253a50e2a6dd4bacc035ab07e94b48881884a3c7d06d134edc449c" exitCode=0
Nov 03 01:54:06.035267 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:06.035075    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"
Nov 03 01:54:06.035267 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:06.035112    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" event={"ID":"872fb22d-056c-4e0c-9555-e285d73cd87f","Type":"ContainerDied","Data":"fb4544c9d3253a50e2a6dd4bacc035ab07e94b48881884a3c7d06d134edc449c"}
Nov 03 01:54:06.035267 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:06.035170    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98" event={"ID":"872fb22d-056c-4e0c-9555-e285d73cd87f","Type":"ContainerDied","Data":"d91a3287e92023bde9b85f522da51e9bf26c41c20c89ea7637faff09d42b233b"}
Nov 03 01:54:06.035267 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:06.035198    5508 scope.go:122] "RemoveContainer" containerID="fb4544c9d3253a50e2a6dd4bacc035ab07e94b48881884a3c7d06d134edc449c"
Nov 03 01:54:06.045560 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:06.045543    5508 scope.go:122] "RemoveContainer" containerID="d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0"
Nov 03 01:54:06.056002 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:06.055976    5508 scope.go:122] "RemoveContainer" containerID="fb4544c9d3253a50e2a6dd4bacc035ab07e94b48881884a3c7d06d134edc449c"
Nov 03 01:54:06.056269 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:54:06.056248    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4544c9d3253a50e2a6dd4bacc035ab07e94b48881884a3c7d06d134edc449c\": container with ID starting with fb4544c9d3253a50e2a6dd4bacc035ab07e94b48881884a3c7d06d134edc449c not found: ID does not exist" containerID="fb4544c9d3253a50e2a6dd4bacc035ab07e94b48881884a3c7d06d134edc449c"
Nov 03 01:54:06.056379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:06.056279    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4544c9d3253a50e2a6dd4bacc035ab07e94b48881884a3c7d06d134edc449c"} err="failed to get container status \"fb4544c9d3253a50e2a6dd4bacc035ab07e94b48881884a3c7d06d134edc449c\": rpc error: code = NotFound desc = could not find container \"fb4544c9d3253a50e2a6dd4bacc035ab07e94b48881884a3c7d06d134edc449c\": container with ID starting with fb4544c9d3253a50e2a6dd4bacc035ab07e94b48881884a3c7d06d134edc449c not found: ID does not exist"
Nov 03 01:54:06.056379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:06.056302    5508 scope.go:122] "RemoveContainer" containerID="d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0"
Nov 03 01:54:06.056555 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:54:06.056534    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0\": container with ID starting with d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0 not found: ID does not exist" containerID="d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0"
Nov 03 01:54:06.056615 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:06.056563    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0"} err="failed to get container status \"d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0\": rpc error: code = NotFound desc = could not find container \"d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0\": container with ID starting with d4a3e50b6ec63b8925ae2a6f6b673eb05c8c8995a52551903ce6d78f3ee941c0 not found: ID does not exist"
Nov 03 01:54:06.189769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:06.189738    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"]
Nov 03 01:54:06.195101 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:06.195076    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7bcbbb-openshift-default-6c4464d4b7-82s98"]
Nov 03 01:54:06.696425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:06.696393    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" path="/var/lib/kubelet/pods/872fb22d-056c-4e0c-9555-e285d73cd87f/volumes"
Nov 03 01:54:09.450773 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.450739    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:54:09.563607 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.563559    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istiod-ca-cert\") pod \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") "
Nov 03 01:54:09.563800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.563720    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-data\") pod \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") "
Nov 03 01:54:09.563800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.563756    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-kube-api-access-k95hw\" (UniqueName: \"kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-kube-api-access-k95hw\") pod \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") "
Nov 03 01:54:09.563867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.563797    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-socket\") pod \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") "
Nov 03 01:54:09.563903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.563872    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-podinfo\") pod \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") "
Nov 03 01:54:09.563956 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.563937    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-token\" (UniqueName: \"kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-token\") pod \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") "
Nov 03 01:54:09.564064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.564047    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-credential-socket\") pod \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") "
Nov 03 01:54:09.564103 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.564086    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-envoy\") pod \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") "
Nov 03 01:54:09.564162 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.564146    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-certs\") pod \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\" (UID: \"ba0b8bfb-8d40-46b7-8277-03d07f5e401a\") "
Nov 03 01:54:09.564546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.564513    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-socket" pod "ba0b8bfb-8d40-46b7-8277-03d07f5e401a" (UID: "ba0b8bfb-8d40-46b7-8277-03d07f5e401a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:09.564971 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.564939    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-certs" pod "ba0b8bfb-8d40-46b7-8277-03d07f5e401a" (UID: "ba0b8bfb-8d40-46b7-8277-03d07f5e401a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:09.565095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.564954    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-credential-socket" pod "ba0b8bfb-8d40-46b7-8277-03d07f5e401a" (UID: "ba0b8bfb-8d40-46b7-8277-03d07f5e401a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:09.565172 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.565142    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-data" pod "ba0b8bfb-8d40-46b7-8277-03d07f5e401a" (UID: "ba0b8bfb-8d40-46b7-8277-03d07f5e401a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:09.565241 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.564952    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:09.565641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.565608    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istiod-ca-cert" pod "ba0b8bfb-8d40-46b7-8277-03d07f5e401a" (UID: "ba0b8bfb-8d40-46b7-8277-03d07f5e401a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:54:09.568154 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.568130    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-envoy" pod "ba0b8bfb-8d40-46b7-8277-03d07f5e401a" (UID: "ba0b8bfb-8d40-46b7-8277-03d07f5e401a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:09.568278 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.568125    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-podinfo" pod "ba0b8bfb-8d40-46b7-8277-03d07f5e401a" (UID: "ba0b8bfb-8d40-46b7-8277-03d07f5e401a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:54:09.568278 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.568251    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-token" pod "ba0b8bfb-8d40-46b7-8277-03d07f5e401a" (UID: "ba0b8bfb-8d40-46b7-8277-03d07f5e401a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:09.568509 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.568489    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-kube-api-access-k95hw" pod "ba0b8bfb-8d40-46b7-8277-03d07f5e401a" (UID: "ba0b8bfb-8d40-46b7-8277-03d07f5e401a"). InnerVolumeSpecName "kube-api-access-k95hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:09.666184 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.666144    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:09.666379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.666204    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k95hw\" (UniqueName: \"kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-kube-api-access-k95hw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:09.666379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.666222    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:09.666379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.666233    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:09.666379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.666246    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:09.666379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.666258    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:09.666379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.666270    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:09.666379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:09.666282    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0b8bfb-8d40-46b7-8277-03d07f5e401a-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:10.068586 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:10.068553    5508 generic.go:356] "Generic (PLEG): container finished" podID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerID="a8d01124001ebf7cd1352b7da1a5ecd2e05b15ef3bb2fbc33efd3cf2ed1aa553" exitCode=0
Nov 03 01:54:10.068779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:10.068622    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"
Nov 03 01:54:10.068779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:10.068642    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" event={"ID":"ba0b8bfb-8d40-46b7-8277-03d07f5e401a","Type":"ContainerDied","Data":"a8d01124001ebf7cd1352b7da1a5ecd2e05b15ef3bb2fbc33efd3cf2ed1aa553"}
Nov 03 01:54:10.068779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:10.068707    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt" event={"ID":"ba0b8bfb-8d40-46b7-8277-03d07f5e401a","Type":"ContainerDied","Data":"87a4531c69789b5f04f7e0bd06e524a06dcf3690e3a43c56c29e77c093b60c7a"}
Nov 03 01:54:10.068779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:10.068737    5508 scope.go:122] "RemoveContainer" containerID="a8d01124001ebf7cd1352b7da1a5ecd2e05b15ef3bb2fbc33efd3cf2ed1aa553"
Nov 03 01:54:10.080239 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:10.080213    5508 scope.go:122] "RemoveContainer" containerID="ef28432c89aa40bbcc22d0198b2917633de262180d70f3432ccba38da79191c5"
Nov 03 01:54:10.088895 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:10.088880    5508 scope.go:122] "RemoveContainer" containerID="a8d01124001ebf7cd1352b7da1a5ecd2e05b15ef3bb2fbc33efd3cf2ed1aa553"
Nov 03 01:54:10.089176 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:54:10.089149    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d01124001ebf7cd1352b7da1a5ecd2e05b15ef3bb2fbc33efd3cf2ed1aa553\": container with ID starting with a8d01124001ebf7cd1352b7da1a5ecd2e05b15ef3bb2fbc33efd3cf2ed1aa553 not found: ID does not exist" containerID="a8d01124001ebf7cd1352b7da1a5ecd2e05b15ef3bb2fbc33efd3cf2ed1aa553"
Nov 03 01:54:10.089249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:10.089183    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d01124001ebf7cd1352b7da1a5ecd2e05b15ef3bb2fbc33efd3cf2ed1aa553"} err="failed to get container status \"a8d01124001ebf7cd1352b7da1a5ecd2e05b15ef3bb2fbc33efd3cf2ed1aa553\": rpc error: code = NotFound desc = could not find container \"a8d01124001ebf7cd1352b7da1a5ecd2e05b15ef3bb2fbc33efd3cf2ed1aa553\": container with ID starting with a8d01124001ebf7cd1352b7da1a5ecd2e05b15ef3bb2fbc33efd3cf2ed1aa553 not found: ID does not exist"
Nov 03 01:54:10.089249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:10.089203    5508 scope.go:122] "RemoveContainer" containerID="ef28432c89aa40bbcc22d0198b2917633de262180d70f3432ccba38da79191c5"
Nov 03 01:54:10.089476 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:54:10.089451    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef28432c89aa40bbcc22d0198b2917633de262180d70f3432ccba38da79191c5\": container with ID starting with ef28432c89aa40bbcc22d0198b2917633de262180d70f3432ccba38da79191c5 not found: ID does not exist" containerID="ef28432c89aa40bbcc22d0198b2917633de262180d70f3432ccba38da79191c5"
Nov 03 01:54:10.089534 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:10.089487    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef28432c89aa40bbcc22d0198b2917633de262180d70f3432ccba38da79191c5"} err="failed to get container status \"ef28432c89aa40bbcc22d0198b2917633de262180d70f3432ccba38da79191c5\": rpc error: code = NotFound desc = could not find container \"ef28432c89aa40bbcc22d0198b2917633de262180d70f3432ccba38da79191c5\": container with ID starting with ef28432c89aa40bbcc22d0198b2917633de262180d70f3432ccba38da79191c5 not found: ID does not exist"
Nov 03 01:54:10.270318 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:10.270278    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"]
Nov 03 01:54:10.274093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:10.274068    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-v2fwt"]
Nov 03 01:54:10.694621 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:10.694583    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" path="/var/lib/kubelet/pods/ba0b8bfb-8d40-46b7-8277-03d07f5e401a/volumes"
Nov 03 01:54:13.811492 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:54:13.811458    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:54:17.091510 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.091477    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"]
Nov 03 01:54:17.199452 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.199421    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"]
Nov 03 01:54:17.202961 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.202459    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"]
Nov 03 01:54:17.203374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203323    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy"
Nov 03 01:54:17.203374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203365    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy"
Nov 03 01:54:17.203646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203424    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d" containerName="payload-pre-processing"
Nov 03 01:54:17.203646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203442    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d" containerName="payload-pre-processing"
Nov 03 01:54:17.203646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203459    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8a092c54-5ed5-4bb9-b65b-e80a1a5122cb" containerName="maas-api"
Nov 03 01:54:17.203646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203467    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a092c54-5ed5-4bb9-b65b-e80a1a5122cb" containerName="maas-api"
Nov 03 01:54:17.203646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203508    5508 cpu_manager.go:414] "containerMap: removing container" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerName="istio-proxy"
Nov 03 01:54:17.203646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203515    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerName="istio-proxy"
Nov 03 01:54:17.203646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203526    5508 cpu_manager.go:414] "containerMap: removing container" podUID="31af0199-a45d-4302-ac71-7e6b9e7ed8b0" containerName="payload-processing"
Nov 03 01:54:17.203646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203534    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31af0199-a45d-4302-ac71-7e6b9e7ed8b0" containerName="payload-processing"
Nov 03 01:54:17.203646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203549    5508 cpu_manager.go:414] "containerMap: removing container" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerName="istio-proxy"
Nov 03 01:54:17.203646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203556    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerName="istio-proxy"
Nov 03 01:54:17.203646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203580    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy"
Nov 03 01:54:17.203646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203587    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy"
Nov 03 01:54:17.204315 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203743    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="31af0199-a45d-4302-ac71-7e6b9e7ed8b0" containerName="payload-processing"
Nov 03 01:54:17.204315 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203759    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerName="istio-proxy"
Nov 03 01:54:17.204315 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203771    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy"
Nov 03 01:54:17.204315 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203789    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b7e5cf-d7cc-4ada-88d1-ba5a6a2db98d" containerName="payload-pre-processing"
Nov 03 01:54:17.204315 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.203800    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a092c54-5ed5-4bb9-b65b-e80a1a5122cb" containerName="maas-api"
Nov 03 01:54:17.209143 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.209124    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"
Nov 03 01:54:17.254882 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.254860    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"]
Nov 03 01:54:17.257789 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.257733    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-71746a\""
Nov 03 01:54:17.257915 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.257759    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-71746a-dockercfg-p9frs\""
Nov 03 01:54:17.384704 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.384619    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p\" (UID: \"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"
Nov 03 01:54:17.385177 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.385158    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvwz5\" (UniqueName: \"kubernetes.io/projected/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-kube-api-access-qvwz5\") pod \"payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p\" (UID: \"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"
Nov 03 01:54:17.486580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.486546    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvwz5\" (UniqueName: \"kubernetes.io/projected/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-kube-api-access-qvwz5\") pod \"payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p\" (UID: \"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"
Nov 03 01:54:17.486930 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.486907    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p\" (UID: \"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"
Nov 03 01:54:17.488070 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.488044    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p\" (UID: \"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"
Nov 03 01:54:17.543742 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.543710    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvwz5\" (UniqueName: \"kubernetes.io/projected/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-kube-api-access-qvwz5\") pod \"payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p\" (UID: \"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"
Nov 03 01:54:17.822246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.822218    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"
Nov 03 01:54:17.971033 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.970999    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"]
Nov 03 01:54:17.971895 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.971874    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="872fb22d-056c-4e0c-9555-e285d73cd87f" containerName="istio-proxy"
Nov 03 01:54:17.971895 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.971897    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0b8bfb-8d40-46b7-8277-03d07f5e401a" containerName="istio-proxy"
Nov 03 01:54:17.975466 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:17.975447    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"
Nov 03 01:54:18.028198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.028173    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"]
Nov 03 01:54:18.096033 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.095959    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25\" (UID: \"a67f1cc6-9db6-4464-9f0e-1392b5be85cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"
Nov 03 01:54:18.096443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.096164    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzrdc\" (UniqueName: \"kubernetes.io/projected/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-kube-api-access-hzrdc\") pod \"payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25\" (UID: \"a67f1cc6-9db6-4464-9f0e-1392b5be85cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"
Nov 03 01:54:18.197435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.197398    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25\" (UID: \"a67f1cc6-9db6-4464-9f0e-1392b5be85cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"
Nov 03 01:54:18.197614 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.197590    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzrdc\" (UniqueName: \"kubernetes.io/projected/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-kube-api-access-hzrdc\") pod \"payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25\" (UID: \"a67f1cc6-9db6-4464-9f0e-1392b5be85cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"
Nov 03 01:54:18.198418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.198241    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25\" (UID: \"a67f1cc6-9db6-4464-9f0e-1392b5be85cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"
Nov 03 01:54:18.202281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.202255    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"]
Nov 03 01:54:18.208311 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:54:18.208286    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b4fd3a_9cf2_489f_89cd_59b57cddfc06.slice/crio-88a7bf471e6fae4b5c9dcf6375804494a30c319c0c7c866d1aab70b5ded8b036 WatchSource:0}: Error finding container 88a7bf471e6fae4b5c9dcf6375804494a30c319c0c7c866d1aab70b5ded8b036: Status 404 returned error can't find the container with id 88a7bf471e6fae4b5c9dcf6375804494a30c319c0c7c866d1aab70b5ded8b036
Nov 03 01:54:18.255684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.255654    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzrdc\" (UniqueName: \"kubernetes.io/projected/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-kube-api-access-hzrdc\") pod \"payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25\" (UID: \"a67f1cc6-9db6-4464-9f0e-1392b5be85cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"
Nov 03 01:54:18.289760 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.289737    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"
Nov 03 01:54:18.626306 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.626241    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"]
Nov 03 01:54:18.626494 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.626469    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n" podUID="d5a3f669-6790-4ba7-b21d-abee6025f055" containerName="payload-processing" containerID="cri-o://b600c24ccaa948c5525cfddd6745bc3de57fb5f8bf9ee0d34b71bb618825ba05" gracePeriod=30
Nov 03 01:54:18.667802 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.667772    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"]
Nov 03 01:54:18.678601 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.678581    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"]
Nov 03 01:54:18.678813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.678791    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62" podUID="15bdec5e-7ba9-48c7-a3b8-2e1488567579" containerName="payload-pre-processing" containerID="cri-o://c94b03d34ca0727b64d5700469dfea1a338ed7efb3955469292b81472fe52374" gracePeriod=30
Nov 03 01:54:18.957648 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:18.957612    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"
Nov 03 01:54:19.008508 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.008482    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"
Nov 03 01:54:19.111929 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.111889    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a3f669-6790-4ba7-b21d-abee6025f055-kube-api-access-nrlch\" (UniqueName: \"kubernetes.io/projected/d5a3f669-6790-4ba7-b21d-abee6025f055-kube-api-access-nrlch\") pod \"d5a3f669-6790-4ba7-b21d-abee6025f055\" (UID: \"d5a3f669-6790-4ba7-b21d-abee6025f055\") "
Nov 03 01:54:19.112428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.111977    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a3f669-6790-4ba7-b21d-abee6025f055-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a3f669-6790-4ba7-b21d-abee6025f055-plugins-config-volume\") pod \"d5a3f669-6790-4ba7-b21d-abee6025f055\" (UID: \"d5a3f669-6790-4ba7-b21d-abee6025f055\") "
Nov 03 01:54:19.112428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.112114    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15bdec5e-7ba9-48c7-a3b8-2e1488567579-kube-api-access-zt5h5\" (UniqueName: \"kubernetes.io/projected/15bdec5e-7ba9-48c7-a3b8-2e1488567579-kube-api-access-zt5h5\") pod \"15bdec5e-7ba9-48c7-a3b8-2e1488567579\" (UID: \"15bdec5e-7ba9-48c7-a3b8-2e1488567579\") "
Nov 03 01:54:19.112428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.112167    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15bdec5e-7ba9-48c7-a3b8-2e1488567579-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bdec5e-7ba9-48c7-a3b8-2e1488567579-plugins-config-volume\") pod \"15bdec5e-7ba9-48c7-a3b8-2e1488567579\" (UID: \"15bdec5e-7ba9-48c7-a3b8-2e1488567579\") "
Nov 03 01:54:19.112428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.112395    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a3f669-6790-4ba7-b21d-abee6025f055-plugins-config-volume" pod "d5a3f669-6790-4ba7-b21d-abee6025f055" (UID: "d5a3f669-6790-4ba7-b21d-abee6025f055"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:54:19.112684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.112616    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15bdec5e-7ba9-48c7-a3b8-2e1488567579-plugins-config-volume" pod "15bdec5e-7ba9-48c7-a3b8-2e1488567579" (UID: "15bdec5e-7ba9-48c7-a3b8-2e1488567579"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:54:19.112801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.112779    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a3f669-6790-4ba7-b21d-abee6025f055-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:19.112871 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.112806    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15bdec5e-7ba9-48c7-a3b8-2e1488567579-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:19.114100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.114077    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a3f669-6790-4ba7-b21d-abee6025f055-kube-api-access-nrlch" pod "d5a3f669-6790-4ba7-b21d-abee6025f055" (UID: "d5a3f669-6790-4ba7-b21d-abee6025f055"). InnerVolumeSpecName "kube-api-access-nrlch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:19.114212 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.114151    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bdec5e-7ba9-48c7-a3b8-2e1488567579-kube-api-access-zt5h5" pod "15bdec5e-7ba9-48c7-a3b8-2e1488567579" (UID: "15bdec5e-7ba9-48c7-a3b8-2e1488567579"). InnerVolumeSpecName "kube-api-access-zt5h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:19.130334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.130217    5508 generic.go:356] "Generic (PLEG): container finished" podID="15bdec5e-7ba9-48c7-a3b8-2e1488567579" containerID="c94b03d34ca0727b64d5700469dfea1a338ed7efb3955469292b81472fe52374" exitCode=0
Nov 03 01:54:19.130334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.130254    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62" event={"ID":"15bdec5e-7ba9-48c7-a3b8-2e1488567579","Type":"ContainerDied","Data":"c94b03d34ca0727b64d5700469dfea1a338ed7efb3955469292b81472fe52374"}
Nov 03 01:54:19.130334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.130281    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"
Nov 03 01:54:19.130334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.130315    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62" event={"ID":"15bdec5e-7ba9-48c7-a3b8-2e1488567579","Type":"ContainerDied","Data":"483cff41bf8967a7a7aa567b76dcdf8435943d98ea1917c182693dde307ce5a6"}
Nov 03 01:54:19.130621 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.130375    5508 scope.go:122] "RemoveContainer" containerID="c94b03d34ca0727b64d5700469dfea1a338ed7efb3955469292b81472fe52374"
Nov 03 01:54:19.131972 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.131952    5508 generic.go:356] "Generic (PLEG): container finished" podID="d5a3f669-6790-4ba7-b21d-abee6025f055" containerID="b600c24ccaa948c5525cfddd6745bc3de57fb5f8bf9ee0d34b71bb618825ba05" exitCode=0
Nov 03 01:54:19.132075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.131996    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"
Nov 03 01:54:19.132075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.132067    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n" event={"ID":"d5a3f669-6790-4ba7-b21d-abee6025f055","Type":"ContainerDied","Data":"b600c24ccaa948c5525cfddd6745bc3de57fb5f8bf9ee0d34b71bb618825ba05"}
Nov 03 01:54:19.132189 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.132110    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n" event={"ID":"d5a3f669-6790-4ba7-b21d-abee6025f055","Type":"ContainerDied","Data":"3c69163a1a77c5042e36eb8c79388bfb20cefdb8c6c34917ab43981c09b6aa50"}
Nov 03 01:54:19.134452 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.134072    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p" event={"ID":"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06","Type":"ContainerStarted","Data":"87a5880a56857da473302a0d3777578adca1dd8dad665d001307a5a85c5cca5d"}
Nov 03 01:54:19.134452 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.134125    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"
Nov 03 01:54:19.134452 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.134151    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p" event={"ID":"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06","Type":"ContainerStarted","Data":"88a7bf471e6fae4b5c9dcf6375804494a30c319c0c7c866d1aab70b5ded8b036"}
Nov 03 01:54:19.150369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.140478    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25" event={"ID":"a67f1cc6-9db6-4464-9f0e-1392b5be85cd","Type":"ContainerStarted","Data":"a01305753f04c3659fbc7b61cb41d83e4562d8398459bfd60cb38f2fe02b536f"}
Nov 03 01:54:19.150369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.140525    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25" event={"ID":"a67f1cc6-9db6-4464-9f0e-1392b5be85cd","Type":"ContainerStarted","Data":"4525b809c6af4d6583b1c4a7f075abbd439c1f272fa9deb6b2a7ff0077bd914e"}
Nov 03 01:54:19.150369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.141152    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"
Nov 03 01:54:19.150369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.147282    5508 scope.go:122] "RemoveContainer" containerID="c94b03d34ca0727b64d5700469dfea1a338ed7efb3955469292b81472fe52374"
Nov 03 01:54:19.150369 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:54:19.149685    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94b03d34ca0727b64d5700469dfea1a338ed7efb3955469292b81472fe52374\": container with ID starting with c94b03d34ca0727b64d5700469dfea1a338ed7efb3955469292b81472fe52374 not found: ID does not exist" containerID="c94b03d34ca0727b64d5700469dfea1a338ed7efb3955469292b81472fe52374"
Nov 03 01:54:19.150369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.149716    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94b03d34ca0727b64d5700469dfea1a338ed7efb3955469292b81472fe52374"} err="failed to get container status \"c94b03d34ca0727b64d5700469dfea1a338ed7efb3955469292b81472fe52374\": rpc error: code = NotFound desc = could not find container \"c94b03d34ca0727b64d5700469dfea1a338ed7efb3955469292b81472fe52374\": container with ID starting with c94b03d34ca0727b64d5700469dfea1a338ed7efb3955469292b81472fe52374 not found: ID does not exist"
Nov 03 01:54:19.150369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.149738    5508 scope.go:122] "RemoveContainer" containerID="b600c24ccaa948c5525cfddd6745bc3de57fb5f8bf9ee0d34b71bb618825ba05"
Nov 03 01:54:19.221370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.213796    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrlch\" (UniqueName: \"kubernetes.io/projected/d5a3f669-6790-4ba7-b21d-abee6025f055-kube-api-access-nrlch\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:19.221370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.213827    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt5h5\" (UniqueName: \"kubernetes.io/projected/15bdec5e-7ba9-48c7-a3b8-2e1488567579-kube-api-access-zt5h5\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:19.271900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.270541    5508 scope.go:122] "RemoveContainer" containerID="b600c24ccaa948c5525cfddd6745bc3de57fb5f8bf9ee0d34b71bb618825ba05"
Nov 03 01:54:19.272589 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:54:19.272510    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b600c24ccaa948c5525cfddd6745bc3de57fb5f8bf9ee0d34b71bb618825ba05\": container with ID starting with b600c24ccaa948c5525cfddd6745bc3de57fb5f8bf9ee0d34b71bb618825ba05 not found: ID does not exist" containerID="b600c24ccaa948c5525cfddd6745bc3de57fb5f8bf9ee0d34b71bb618825ba05"
Nov 03 01:54:19.272589 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.272551    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b600c24ccaa948c5525cfddd6745bc3de57fb5f8bf9ee0d34b71bb618825ba05"} err="failed to get container status \"b600c24ccaa948c5525cfddd6745bc3de57fb5f8bf9ee0d34b71bb618825ba05\": rpc error: code = NotFound desc = could not find container \"b600c24ccaa948c5525cfddd6745bc3de57fb5f8bf9ee0d34b71bb618825ba05\": container with ID starting with b600c24ccaa948c5525cfddd6745bc3de57fb5f8bf9ee0d34b71bb618825ba05 not found: ID does not exist"
Nov 03 01:54:19.332082 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.332049    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"]
Nov 03 01:54:19.334801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.334777    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-86e679-bc5d7c486-29k2n"]
Nov 03 01:54:19.441072 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.440978    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p" podStartSLOduration=2.44096048 podStartE2EDuration="2.44096048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:19.437775764 +0000 UTC m=+2138.863382227" watchObservedRunningTime="2026-11-03 01:54:19.44096048 +0000 UTC m=+2138.866566944"
Nov 03 01:54:19.542480 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.542450    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"]
Nov 03 01:54:19.546361 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.546326    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-86e679-7fd87774c5-bzm62"]
Nov 03 01:54:19.659035 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:19.658971    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25" podStartSLOduration=2.658951546 podStartE2EDuration="2.658951546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:19.653875799 +0000 UTC m=+2139.079482264" watchObservedRunningTime="2026-11-03 01:54:19.658951546 +0000 UTC m=+2139.084558012"
Nov 03 01:54:20.694948 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:20.694914    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15bdec5e-7ba9-48c7-a3b8-2e1488567579" path="/var/lib/kubelet/pods/15bdec5e-7ba9-48c7-a3b8-2e1488567579/volumes"
Nov 03 01:54:20.695449 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:20.695423    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a3f669-6790-4ba7-b21d-abee6025f055" path="/var/lib/kubelet/pods/d5a3f669-6790-4ba7-b21d-abee6025f055/volumes"
Nov 03 01:54:22.092046 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.092003    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd" podUID="a1b3fffd-f8f2-432e-867a-4179539374f7" containerName="payload-processing" containerID="cri-o://de7b428d5ea82f88225e190483978dee54203edf42c8abd217ab164890d4cced" gracePeriod=25
Nov 03 01:54:22.200938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.200907    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2" podUID="a899ee86-26da-4460-aac2-bfdde9d2d6bf" containerName="payload-pre-processing" containerID="cri-o://4730fe5ec1cb3330e88d313321185568c0093bb00d6a68459e07f18c4ad9208c" gracePeriod=25
Nov 03 01:54:22.366740 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.366708    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"
Nov 03 01:54:22.453489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.453467    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1b3fffd-f8f2-432e-867a-4179539374f7-kube-api-access-fxdxl\" (UniqueName: \"kubernetes.io/projected/a1b3fffd-f8f2-432e-867a-4179539374f7-kube-api-access-fxdxl\") pod \"a1b3fffd-f8f2-432e-867a-4179539374f7\" (UID: \"a1b3fffd-f8f2-432e-867a-4179539374f7\") "
Nov 03 01:54:22.453601 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.453530    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1b3fffd-f8f2-432e-867a-4179539374f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b3fffd-f8f2-432e-867a-4179539374f7-plugins-config-volume\") pod \"a1b3fffd-f8f2-432e-867a-4179539374f7\" (UID: \"a1b3fffd-f8f2-432e-867a-4179539374f7\") "
Nov 03 01:54:22.453888 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.453867    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b3fffd-f8f2-432e-867a-4179539374f7-plugins-config-volume" pod "a1b3fffd-f8f2-432e-867a-4179539374f7" (UID: "a1b3fffd-f8f2-432e-867a-4179539374f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:54:22.455494 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.455465    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b3fffd-f8f2-432e-867a-4179539374f7-kube-api-access-fxdxl" pod "a1b3fffd-f8f2-432e-867a-4179539374f7" (UID: "a1b3fffd-f8f2-432e-867a-4179539374f7"). InnerVolumeSpecName "kube-api-access-fxdxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:22.471750 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.471729    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"
Nov 03 01:54:22.554682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.554659    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a899ee86-26da-4460-aac2-bfdde9d2d6bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a899ee86-26da-4460-aac2-bfdde9d2d6bf-plugins-config-volume\") pod \"a899ee86-26da-4460-aac2-bfdde9d2d6bf\" (UID: \"a899ee86-26da-4460-aac2-bfdde9d2d6bf\") "
Nov 03 01:54:22.554820 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.554747    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a899ee86-26da-4460-aac2-bfdde9d2d6bf-kube-api-access-4mt2j\" (UniqueName: \"kubernetes.io/projected/a899ee86-26da-4460-aac2-bfdde9d2d6bf-kube-api-access-4mt2j\") pod \"a899ee86-26da-4460-aac2-bfdde9d2d6bf\" (UID: \"a899ee86-26da-4460-aac2-bfdde9d2d6bf\") "
Nov 03 01:54:22.554996 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.554985    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxdxl\" (UniqueName: \"kubernetes.io/projected/a1b3fffd-f8f2-432e-867a-4179539374f7-kube-api-access-fxdxl\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:22.555033 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.554998    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b3fffd-f8f2-432e-867a-4179539374f7-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:22.555079 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.555056    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a899ee86-26da-4460-aac2-bfdde9d2d6bf-plugins-config-volume" pod "a899ee86-26da-4460-aac2-bfdde9d2d6bf" (UID: "a899ee86-26da-4460-aac2-bfdde9d2d6bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:54:22.556667 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.556652    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a899ee86-26da-4460-aac2-bfdde9d2d6bf-kube-api-access-4mt2j" pod "a899ee86-26da-4460-aac2-bfdde9d2d6bf" (UID: "a899ee86-26da-4460-aac2-bfdde9d2d6bf"). InnerVolumeSpecName "kube-api-access-4mt2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:22.663942 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.656307    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a899ee86-26da-4460-aac2-bfdde9d2d6bf-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:22.663942 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:22.656344    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mt2j\" (UniqueName: \"kubernetes.io/projected/a899ee86-26da-4460-aac2-bfdde9d2d6bf-kube-api-access-4mt2j\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:23.172273 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.172237    5508 generic.go:356] "Generic (PLEG): container finished" podID="a1b3fffd-f8f2-432e-867a-4179539374f7" containerID="de7b428d5ea82f88225e190483978dee54203edf42c8abd217ab164890d4cced" exitCode=0
Nov 03 01:54:23.172737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.172309    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd" event={"ID":"a1b3fffd-f8f2-432e-867a-4179539374f7","Type":"ContainerDied","Data":"de7b428d5ea82f88225e190483978dee54203edf42c8abd217ab164890d4cced"}
Nov 03 01:54:23.172737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.172334    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"
Nov 03 01:54:23.172737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.172390    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd" event={"ID":"a1b3fffd-f8f2-432e-867a-4179539374f7","Type":"ContainerDied","Data":"ee8325cf3417e323d96d50d9a50f5b2a2d096155141883f211ce8537ee58e8d6"}
Nov 03 01:54:23.172737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.172426    5508 scope.go:122] "RemoveContainer" containerID="de7b428d5ea82f88225e190483978dee54203edf42c8abd217ab164890d4cced"
Nov 03 01:54:23.174014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.173990    5508 generic.go:356] "Generic (PLEG): container finished" podID="a899ee86-26da-4460-aac2-bfdde9d2d6bf" containerID="4730fe5ec1cb3330e88d313321185568c0093bb00d6a68459e07f18c4ad9208c" exitCode=0
Nov 03 01:54:23.174121 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.174059    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"
Nov 03 01:54:23.174121 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.174081    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2" event={"ID":"a899ee86-26da-4460-aac2-bfdde9d2d6bf","Type":"ContainerDied","Data":"4730fe5ec1cb3330e88d313321185568c0093bb00d6a68459e07f18c4ad9208c"}
Nov 03 01:54:23.174194 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.174134    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2" event={"ID":"a899ee86-26da-4460-aac2-bfdde9d2d6bf","Type":"ContainerDied","Data":"9cb5c5d51c7f48cb3c9d5ade5cebe34d71555c32b0775d45f06fbabb4d1d29ae"}
Nov 03 01:54:23.182792 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.182770    5508 scope.go:122] "RemoveContainer" containerID="de7b428d5ea82f88225e190483978dee54203edf42c8abd217ab164890d4cced"
Nov 03 01:54:23.183062 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:54:23.183044    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7b428d5ea82f88225e190483978dee54203edf42c8abd217ab164890d4cced\": container with ID starting with de7b428d5ea82f88225e190483978dee54203edf42c8abd217ab164890d4cced not found: ID does not exist" containerID="de7b428d5ea82f88225e190483978dee54203edf42c8abd217ab164890d4cced"
Nov 03 01:54:23.183118 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.183070    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7b428d5ea82f88225e190483978dee54203edf42c8abd217ab164890d4cced"} err="failed to get container status \"de7b428d5ea82f88225e190483978dee54203edf42c8abd217ab164890d4cced\": rpc error: code = NotFound desc = could not find container \"de7b428d5ea82f88225e190483978dee54203edf42c8abd217ab164890d4cced\": container with ID starting with de7b428d5ea82f88225e190483978dee54203edf42c8abd217ab164890d4cced not found: ID does not exist"
Nov 03 01:54:23.183118 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.183086    5508 scope.go:122] "RemoveContainer" containerID="4730fe5ec1cb3330e88d313321185568c0093bb00d6a68459e07f18c4ad9208c"
Nov 03 01:54:23.191558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.191541    5508 scope.go:122] "RemoveContainer" containerID="4730fe5ec1cb3330e88d313321185568c0093bb00d6a68459e07f18c4ad9208c"
Nov 03 01:54:23.191816 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:54:23.191796    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4730fe5ec1cb3330e88d313321185568c0093bb00d6a68459e07f18c4ad9208c\": container with ID starting with 4730fe5ec1cb3330e88d313321185568c0093bb00d6a68459e07f18c4ad9208c not found: ID does not exist" containerID="4730fe5ec1cb3330e88d313321185568c0093bb00d6a68459e07f18c4ad9208c"
Nov 03 01:54:23.191900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.191821    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4730fe5ec1cb3330e88d313321185568c0093bb00d6a68459e07f18c4ad9208c"} err="failed to get container status \"4730fe5ec1cb3330e88d313321185568c0093bb00d6a68459e07f18c4ad9208c\": rpc error: code = NotFound desc = could not find container \"4730fe5ec1cb3330e88d313321185568c0093bb00d6a68459e07f18c4ad9208c\": container with ID starting with 4730fe5ec1cb3330e88d313321185568c0093bb00d6a68459e07f18c4ad9208c not found: ID does not exist"
Nov 03 01:54:23.326816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.326780    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"]
Nov 03 01:54:23.333363 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.333331    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e0c1fb-675d7c6849-4czfd"]
Nov 03 01:54:23.434737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.434707    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"]
Nov 03 01:54:23.439032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.439011    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e0c1fb-74c7d85495-cndb2"]
Nov 03 01:54:23.765424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.765332    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"]
Nov 03 01:54:23.765619 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:23.765590    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" containerName="istio-proxy" containerID="cri-o://1fe97b5d9a8a06a54a19ba786088abae401b38b2aaa8eba7b4633c2e670bb0d9" gracePeriod=30
Nov 03 01:54:24.694157 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:24.694125    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1b3fffd-f8f2-432e-867a-4179539374f7" path="/var/lib/kubelet/pods/a1b3fffd-f8f2-432e-867a-4179539374f7/volumes"
Nov 03 01:54:24.694640 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:24.694624    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a899ee86-26da-4460-aac2-bfdde9d2d6bf" path="/var/lib/kubelet/pods/a899ee86-26da-4460-aac2-bfdde9d2d6bf/volumes"
Nov 03 01:54:25.934176 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.934144    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"]
Nov 03 01:54:25.934947 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.934927    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a899ee86-26da-4460-aac2-bfdde9d2d6bf" containerName="payload-pre-processing"
Nov 03 01:54:25.934947 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.934946    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a899ee86-26da-4460-aac2-bfdde9d2d6bf" containerName="payload-pre-processing"
Nov 03 01:54:25.935126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.934979    5508 cpu_manager.go:414] "containerMap: removing container" podUID="15bdec5e-7ba9-48c7-a3b8-2e1488567579" containerName="payload-pre-processing"
Nov 03 01:54:25.935126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.934986    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15bdec5e-7ba9-48c7-a3b8-2e1488567579" containerName="payload-pre-processing"
Nov 03 01:54:25.935126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.934999    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d5a3f669-6790-4ba7-b21d-abee6025f055" containerName="payload-processing"
Nov 03 01:54:25.935126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.935006    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a3f669-6790-4ba7-b21d-abee6025f055" containerName="payload-processing"
Nov 03 01:54:25.935126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.935021    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a1b3fffd-f8f2-432e-867a-4179539374f7" containerName="payload-processing"
Nov 03 01:54:25.935126 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.935028    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1b3fffd-f8f2-432e-867a-4179539374f7" containerName="payload-processing"
Nov 03 01:54:25.935320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.935140    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a899ee86-26da-4460-aac2-bfdde9d2d6bf" containerName="payload-pre-processing"
Nov 03 01:54:25.935320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.935154    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="15bdec5e-7ba9-48c7-a3b8-2e1488567579" containerName="payload-pre-processing"
Nov 03 01:54:25.935320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.935166    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a3f669-6790-4ba7-b21d-abee6025f055" containerName="payload-processing"
Nov 03 01:54:25.935320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.935176    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1b3fffd-f8f2-432e-867a-4179539374f7" containerName="payload-processing"
Nov 03 01:54:25.947302 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.947271    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:25.996150 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:25.996119    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-5d9025-openshift-default-dockercfg-jmp7k\""
Nov 03 01:54:26.001953 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.001930    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-podinfo\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.002055 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.001965    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa8e146-3961-4111-9712-e1323fdf7a08-istiod-ca-cert\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.002104 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.002082    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-token\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.002187 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.002172    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-socket\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.002229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.002198    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxgzh\" (UniqueName: \"kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-kube-api-access-gxgzh\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.002262 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.002224    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-credential-socket\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.002369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.002328    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-envoy\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.002414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.002390    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-certs\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.002468 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.002456    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-data\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.047041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.047010    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"]
Nov 03 01:54:26.104297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.103369    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-socket\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.104297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.103421    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxgzh\" (UniqueName: \"kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-kube-api-access-gxgzh\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.104297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.103475    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-credential-socket\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.104297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.103513    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-envoy\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.104297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.103536    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-certs\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.104297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.103598    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-data\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.104297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.103651    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-podinfo\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.104297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.103692    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa8e146-3961-4111-9712-e1323fdf7a08-istiod-ca-cert\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.104297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.103805    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-token\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.104297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.103849    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-socket\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.104991 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.104484    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-data\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.104991 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.104730    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-credential-socket\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.105163 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.105112    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-certs\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.105637 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.105610    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa8e146-3961-4111-9712-e1323fdf7a08-istiod-ca-cert\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.109485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.109464    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-envoy\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.115026 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.114985    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-podinfo\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.161057 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.161021    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-token\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.161309 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.161286    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxgzh\" (UniqueName: \"kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-kube-api-access-gxgzh\") pod \"e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") " pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.262414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.262315    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:26.695320 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.695287    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"]
Nov 03 01:54:26.702180 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:54:26.702156    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa8e146_3961_4111_9712_e1323fdf7a08.slice/crio-98a66d403475022d25c4a68ce536a044d3cfeb5c3d59161b9f982205358bc139 WatchSource:0}: Error finding container 98a66d403475022d25c4a68ce536a044d3cfeb5c3d59161b9f982205358bc139: Status 404 returned error can't find the container with id 98a66d403475022d25c4a68ce536a044d3cfeb5c3d59161b9f982205358bc139
Nov 03 01:54:26.704693 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.704665    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:54:26.704787 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.704726    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:54:26.704787 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:26.704756    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:54:27.204552 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:27.204518    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8" event={"ID":"3aa8e146-3961-4111-9712-e1323fdf7a08","Type":"ContainerStarted","Data":"ca902b641aaf8e5779c64a63d6db5f3814de24e9941cf528112d99006b101b8c"}
Nov 03 01:54:27.204986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:27.204566    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8" event={"ID":"3aa8e146-3961-4111-9712-e1323fdf7a08","Type":"ContainerStarted","Data":"98a66d403475022d25c4a68ce536a044d3cfeb5c3d59161b9f982205358bc139"}
Nov 03 01:54:27.263399 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:27.263376    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:27.268857 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:27.268833    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:27.366847 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:27.366795    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8" podStartSLOduration=2.366779089 podStartE2EDuration="2.366779089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:27.363488149 +0000 UTC m=+2146.789094641" watchObservedRunningTime="2026-11-03 01:54:27.366779089 +0000 UTC m=+2146.792385552"
Nov 03 01:54:28.211753 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:28.211726    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:28.212700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:28.212677    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:54:29.029492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.029434    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:54:29.134071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134043    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-socket\") pod \"d24d34b3-8841-4389-b20f-297008aa92a5\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") "
Nov 03 01:54:29.134215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134105    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-data\") pod \"d24d34b3-8841-4389-b20f-297008aa92a5\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") "
Nov 03 01:54:29.134215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134129    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d24d34b3-8841-4389-b20f-297008aa92a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24d34b3-8841-4389-b20f-297008aa92a5-istiod-ca-cert\") pod \"d24d34b3-8841-4389-b20f-297008aa92a5\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") "
Nov 03 01:54:29.134215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134169    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-kube-api-access-hnkg7\" (UniqueName: \"kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-kube-api-access-hnkg7\") pod \"d24d34b3-8841-4389-b20f-297008aa92a5\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") "
Nov 03 01:54:29.134440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134241    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-envoy\") pod \"d24d34b3-8841-4389-b20f-297008aa92a5\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") "
Nov 03 01:54:29.134440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134262    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-credential-socket\") pod \"d24d34b3-8841-4389-b20f-297008aa92a5\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") "
Nov 03 01:54:29.134440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134301    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-certs\") pod \"d24d34b3-8841-4389-b20f-297008aa92a5\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") "
Nov 03 01:54:29.134440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134378    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-socket" pod "d24d34b3-8841-4389-b20f-297008aa92a5" (UID: "d24d34b3-8841-4389-b20f-297008aa92a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:29.134440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134408    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-data" pod "d24d34b3-8841-4389-b20f-297008aa92a5" (UID: "d24d34b3-8841-4389-b20f-297008aa92a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:29.134440 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134431    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d24d34b3-8841-4389-b20f-297008aa92a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24d34b3-8841-4389-b20f-297008aa92a5-istio-podinfo\") pod \"d24d34b3-8841-4389-b20f-297008aa92a5\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") "
Nov 03 01:54:29.134727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134507    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-istio-token\" (UniqueName: \"kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-istio-token\") pod \"d24d34b3-8841-4389-b20f-297008aa92a5\" (UID: \"d24d34b3-8841-4389-b20f-297008aa92a5\") "
Nov 03 01:54:29.134727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134614    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24d34b3-8841-4389-b20f-297008aa92a5-istiod-ca-cert" pod "d24d34b3-8841-4389-b20f-297008aa92a5" (UID: "d24d34b3-8841-4389-b20f-297008aa92a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:54:29.134727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134617    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-credential-socket" pod "d24d34b3-8841-4389-b20f-297008aa92a5" (UID: "d24d34b3-8841-4389-b20f-297008aa92a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:29.134962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134926    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-certs" pod "d24d34b3-8841-4389-b20f-297008aa92a5" (UID: "d24d34b3-8841-4389-b20f-297008aa92a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:29.135098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.134990    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:29.135098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.135009    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:29.135098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.135024    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:29.135098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.135037    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d24d34b3-8841-4389-b20f-297008aa92a5-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:29.136674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.136650    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-kube-api-access-hnkg7" pod "d24d34b3-8841-4389-b20f-297008aa92a5" (UID: "d24d34b3-8841-4389-b20f-297008aa92a5"). InnerVolumeSpecName "kube-api-access-hnkg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:29.136674 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.136663    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-envoy" pod "d24d34b3-8841-4389-b20f-297008aa92a5" (UID: "d24d34b3-8841-4389-b20f-297008aa92a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:54:29.136922 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.136906    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d24d34b3-8841-4389-b20f-297008aa92a5-istio-podinfo" pod "d24d34b3-8841-4389-b20f-297008aa92a5" (UID: "d24d34b3-8841-4389-b20f-297008aa92a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:54:29.136974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.136920    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-istio-token" pod "d24d34b3-8841-4389-b20f-297008aa92a5" (UID: "d24d34b3-8841-4389-b20f-297008aa92a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:54:29.221228 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.221186    5508 generic.go:356] "Generic (PLEG): container finished" podID="d24d34b3-8841-4389-b20f-297008aa92a5" containerID="1fe97b5d9a8a06a54a19ba786088abae401b38b2aaa8eba7b4633c2e670bb0d9" exitCode=0
Nov 03 01:54:29.221678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.221247    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"
Nov 03 01:54:29.221678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.221276    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" event={"ID":"d24d34b3-8841-4389-b20f-297008aa92a5","Type":"ContainerDied","Data":"1fe97b5d9a8a06a54a19ba786088abae401b38b2aaa8eba7b4633c2e670bb0d9"}
Nov 03 01:54:29.221678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.221341    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk" event={"ID":"d24d34b3-8841-4389-b20f-297008aa92a5","Type":"ContainerDied","Data":"0b3dc16b2498b59f5e01e9560609127c13f0d2b875bf036fed5de7968558f3b4"}
Nov 03 01:54:29.221678 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.221375    5508 scope.go:122] "RemoveContainer" containerID="1fe97b5d9a8a06a54a19ba786088abae401b38b2aaa8eba7b4633c2e670bb0d9"
Nov 03 01:54:29.237077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.236186    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:29.237077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.236213    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d24d34b3-8841-4389-b20f-297008aa92a5-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:29.237077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.236228    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d24d34b3-8841-4389-b20f-297008aa92a5-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:29.237077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.236241    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:29.237077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.236256    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnkg7\" (UniqueName: \"kubernetes.io/projected/d24d34b3-8841-4389-b20f-297008aa92a5-kube-api-access-hnkg7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:54:29.240289 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.240265    5508 scope.go:122] "RemoveContainer" containerID="48682b93a6ccc6d2a688417ccb7fe4ffcf89f32dc54845b74ee24ff96ba72fd6"
Nov 03 01:54:29.249058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.249041    5508 scope.go:122] "RemoveContainer" containerID="1fe97b5d9a8a06a54a19ba786088abae401b38b2aaa8eba7b4633c2e670bb0d9"
Nov 03 01:54:29.249323 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:54:29.249304    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe97b5d9a8a06a54a19ba786088abae401b38b2aaa8eba7b4633c2e670bb0d9\": container with ID starting with 1fe97b5d9a8a06a54a19ba786088abae401b38b2aaa8eba7b4633c2e670bb0d9 not found: ID does not exist" containerID="1fe97b5d9a8a06a54a19ba786088abae401b38b2aaa8eba7b4633c2e670bb0d9"
Nov 03 01:54:29.249397 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.249332    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe97b5d9a8a06a54a19ba786088abae401b38b2aaa8eba7b4633c2e670bb0d9"} err="failed to get container status \"1fe97b5d9a8a06a54a19ba786088abae401b38b2aaa8eba7b4633c2e670bb0d9\": rpc error: code = NotFound desc = could not find container \"1fe97b5d9a8a06a54a19ba786088abae401b38b2aaa8eba7b4633c2e670bb0d9\": container with ID starting with 1fe97b5d9a8a06a54a19ba786088abae401b38b2aaa8eba7b4633c2e670bb0d9 not found: ID does not exist"
Nov 03 01:54:29.249397 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.249362    5508 scope.go:122] "RemoveContainer" containerID="48682b93a6ccc6d2a688417ccb7fe4ffcf89f32dc54845b74ee24ff96ba72fd6"
Nov 03 01:54:29.249612 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:54:29.249595    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48682b93a6ccc6d2a688417ccb7fe4ffcf89f32dc54845b74ee24ff96ba72fd6\": container with ID starting with 48682b93a6ccc6d2a688417ccb7fe4ffcf89f32dc54845b74ee24ff96ba72fd6 not found: ID does not exist" containerID="48682b93a6ccc6d2a688417ccb7fe4ffcf89f32dc54845b74ee24ff96ba72fd6"
Nov 03 01:54:29.249675 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.249624    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48682b93a6ccc6d2a688417ccb7fe4ffcf89f32dc54845b74ee24ff96ba72fd6"} err="failed to get container status \"48682b93a6ccc6d2a688417ccb7fe4ffcf89f32dc54845b74ee24ff96ba72fd6\": rpc error: code = NotFound desc = could not find container \"48682b93a6ccc6d2a688417ccb7fe4ffcf89f32dc54845b74ee24ff96ba72fd6\": container with ID starting with 48682b93a6ccc6d2a688417ccb7fe4ffcf89f32dc54845b74ee24ff96ba72fd6 not found: ID does not exist"
Nov 03 01:54:29.375867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.375807    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"]
Nov 03 01:54:29.381646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:29.381623    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-86e679-openshift-default-b4ffbdb87-6xdsk"]
Nov 03 01:54:30.151461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:30.151426    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"
Nov 03 01:54:30.693913 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:30.693878    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" path="/var/lib/kubelet/pods/d24d34b3-8841-4389-b20f-297008aa92a5/volumes"
Nov 03 01:54:31.158102 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:31.158076    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"
Nov 03 01:54:33.774902 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.774869    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"]
Nov 03 01:54:33.775424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.775411    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" containerName="istio-proxy"
Nov 03 01:54:33.775482 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.775426    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" containerName="istio-proxy"
Nov 03 01:54:33.775482 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.775437    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" containerName="istio-proxy"
Nov 03 01:54:33.775482 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.775442    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" containerName="istio-proxy"
Nov 03 01:54:33.775569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.775550    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" containerName="istio-proxy"
Nov 03 01:54:33.775569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.775557    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24d34b3-8841-4389-b20f-297008aa92a5" containerName="istio-proxy"
Nov 03 01:54:33.779400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.779381    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"
Nov 03 01:54:33.786513 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.786489    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p\" (UID: \"7ea32c0f-32ac-40b3-9af9-e19ba93a708b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"
Nov 03 01:54:33.786782 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.786757    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwdfx\" (UniqueName: \"kubernetes.io/projected/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-kube-api-access-jwdfx\") pod \"payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p\" (UID: \"7ea32c0f-32ac-40b3-9af9-e19ba93a708b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"
Nov 03 01:54:33.833461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.833439    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"]
Nov 03 01:54:33.880626 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.880591    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-5d9025-dockercfg-msvsq\""
Nov 03 01:54:33.880896 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.880879    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-5d9025\""
Nov 03 01:54:33.919280 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.918433    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p\" (UID: \"7ea32c0f-32ac-40b3-9af9-e19ba93a708b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"
Nov 03 01:54:33.919280 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.918720    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwdfx\" (UniqueName: \"kubernetes.io/projected/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-kube-api-access-jwdfx\") pod \"payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p\" (UID: \"7ea32c0f-32ac-40b3-9af9-e19ba93a708b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"
Nov 03 01:54:33.921179 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.921148    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p\" (UID: \"7ea32c0f-32ac-40b3-9af9-e19ba93a708b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"
Nov 03 01:54:33.940194 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.940160    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"]
Nov 03 01:54:33.944446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.944425    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"
Nov 03 01:54:33.977374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.977336    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwdfx\" (UniqueName: \"kubernetes.io/projected/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-kube-api-access-jwdfx\") pod \"payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p\" (UID: \"7ea32c0f-32ac-40b3-9af9-e19ba93a708b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"
Nov 03 01:54:33.998771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:33.998748    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"]
Nov 03 01:54:34.020337 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:34.020315    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805c5346-d73c-4d36-9f28-8b030b76e4fe-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7\" (UID: \"805c5346-d73c-4d36-9f28-8b030b76e4fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"
Nov 03 01:54:34.020451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:34.020375    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzbc8\" (UniqueName: \"kubernetes.io/projected/805c5346-d73c-4d36-9f28-8b030b76e4fe-kube-api-access-zzbc8\") pod \"payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7\" (UID: \"805c5346-d73c-4d36-9f28-8b030b76e4fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"
Nov 03 01:54:34.092751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:34.092685    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"
Nov 03 01:54:34.121336 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:34.121306    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805c5346-d73c-4d36-9f28-8b030b76e4fe-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7\" (UID: \"805c5346-d73c-4d36-9f28-8b030b76e4fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"
Nov 03 01:54:34.121464 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:34.121403    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzbc8\" (UniqueName: \"kubernetes.io/projected/805c5346-d73c-4d36-9f28-8b030b76e4fe-kube-api-access-zzbc8\") pod \"payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7\" (UID: \"805c5346-d73c-4d36-9f28-8b030b76e4fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"
Nov 03 01:54:34.122063 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:34.122044    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805c5346-d73c-4d36-9f28-8b030b76e4fe-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7\" (UID: \"805c5346-d73c-4d36-9f28-8b030b76e4fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"
Nov 03 01:54:34.179290 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:34.179230    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzbc8\" (UniqueName: \"kubernetes.io/projected/805c5346-d73c-4d36-9f28-8b030b76e4fe-kube-api-access-zzbc8\") pod \"payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7\" (UID: \"805c5346-d73c-4d36-9f28-8b030b76e4fe\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"
Nov 03 01:54:34.259438 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:34.259408    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"
Nov 03 01:54:34.485760 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:34.485725    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"]
Nov 03 01:54:34.493116 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:54:34.493083    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea32c0f_32ac_40b3_9af9_e19ba93a708b.slice/crio-e0214949658893e1e93bb41a822980ccae35e727a9bcfef68a1c0ef1537ccb60 WatchSource:0}: Error finding container e0214949658893e1e93bb41a822980ccae35e727a9bcfef68a1c0ef1537ccb60: Status 404 returned error can't find the container with id e0214949658893e1e93bb41a822980ccae35e727a9bcfef68a1c0ef1537ccb60
Nov 03 01:54:34.635301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:34.635262    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"]
Nov 03 01:54:34.641328 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:54:34.641302    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805c5346_d73c_4d36_9f28_8b030b76e4fe.slice/crio-e95562dd05a5a459f5cf0876e5593da461bd2871a67eef9c1ba9cf27f1bb35b9 WatchSource:0}: Error finding container e95562dd05a5a459f5cf0876e5593da461bd2871a67eef9c1ba9cf27f1bb35b9: Status 404 returned error can't find the container with id e95562dd05a5a459f5cf0876e5593da461bd2871a67eef9c1ba9cf27f1bb35b9
Nov 03 01:54:35.263517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:35.263481    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7" event={"ID":"805c5346-d73c-4d36-9f28-8b030b76e4fe","Type":"ContainerStarted","Data":"d6cfd015007c588c3d5df1b700b708d402af92ca60f9dd4c0e4b5eaae2c7d0c3"}
Nov 03 01:54:35.263964 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:35.263546    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7" event={"ID":"805c5346-d73c-4d36-9f28-8b030b76e4fe","Type":"ContainerStarted","Data":"e95562dd05a5a459f5cf0876e5593da461bd2871a67eef9c1ba9cf27f1bb35b9"}
Nov 03 01:54:35.263964 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:35.263653    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"
Nov 03 01:54:35.265228 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:35.265204    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p" event={"ID":"7ea32c0f-32ac-40b3-9af9-e19ba93a708b","Type":"ContainerStarted","Data":"25dab1846bc443cd2e5251d805248dec5aebf5506d60ad852264f07b4ff4b1f9"}
Nov 03 01:54:35.265376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:35.265243    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p" event={"ID":"7ea32c0f-32ac-40b3-9af9-e19ba93a708b","Type":"ContainerStarted","Data":"e0214949658893e1e93bb41a822980ccae35e727a9bcfef68a1c0ef1537ccb60"}
Nov 03 01:54:35.265376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:35.265264    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"
Nov 03 01:54:35.376636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:35.376578    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7" podStartSLOduration=2.37655978 podStartE2EDuration="2.37655978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:35.372012236 +0000 UTC m=+2154.797618702" watchObservedRunningTime="2026-11-03 01:54:35.37655978 +0000 UTC m=+2154.802166245"
Nov 03 01:54:35.484075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:35.484012    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p" podStartSLOduration=2.483993206 podStartE2EDuration="2.483993206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:35.479503327 +0000 UTC m=+2154.905109815" watchObservedRunningTime="2026-11-03 01:54:35.483993206 +0000 UTC m=+2154.909599672"
Nov 03 01:54:39.462401 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:39.462370    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:54:43.839648 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:54:43.839613    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:54:46.275608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:46.275526    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"
Nov 03 01:54:46.276077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:46.275757    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"
Nov 03 01:54:49.191435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:49.191394    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"]
Nov 03 01:54:49.197732 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:49.197707    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"
Nov 03 01:54:49.211017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:49.210992    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw\" (UID: \"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"
Nov 03 01:54:49.211175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:49.211028    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx72z\" (UniqueName: \"kubernetes.io/projected/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-kube-api-access-lx72z\") pod \"payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw\" (UID: \"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"
Nov 03 01:54:49.293045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:49.293010    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b37ba5-dockercfg-rgvn6\""
Nov 03 01:54:49.293264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:49.293137    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b37ba5\""
Nov 03 01:54:49.296073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:49.296044    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"]
Nov 03 01:54:49.312991 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:49.312957    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw\" (UID: \"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"
Nov 03 01:54:49.313164 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:49.313002    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx72z\" (UniqueName: \"kubernetes.io/projected/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-kube-api-access-lx72z\") pod \"payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw\" (UID: \"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"
Nov 03 01:54:49.313877 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:49.313853    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw\" (UID: \"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"
Nov 03 01:54:49.370185 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:49.370148    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx72z\" (UniqueName: \"kubernetes.io/projected/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-kube-api-access-lx72z\") pod \"payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw\" (UID: \"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"
Nov 03 01:54:49.512538 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:49.512456    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"
Nov 03 01:54:49.884616 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:49.884534    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"]
Nov 03 01:54:49.890526 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:54:49.890499    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2dcfcd8_ff6a_41fb_b9bd_f3ad2d809db5.slice/crio-550d03952c517236a5b7855d507ffa265d389bb8175a53fb09b480dca35fef32 WatchSource:0}: Error finding container 550d03952c517236a5b7855d507ffa265d389bb8175a53fb09b480dca35fef32: Status 404 returned error can't find the container with id 550d03952c517236a5b7855d507ffa265d389bb8175a53fb09b480dca35fef32
Nov 03 01:54:50.003821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.003794    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"]
Nov 03 01:54:50.008056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.008041    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"
Nov 03 01:54:50.021038 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.021017    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc20924a-8101-43ab-95f1-8f0848d649f9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v\" (UID: \"cc20924a-8101-43ab-95f1-8f0848d649f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"
Nov 03 01:54:50.021144 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.021056    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z28sj\" (UniqueName: \"kubernetes.io/projected/cc20924a-8101-43ab-95f1-8f0848d649f9-kube-api-access-z28sj\") pod \"payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v\" (UID: \"cc20924a-8101-43ab-95f1-8f0848d649f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"
Nov 03 01:54:50.062139 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.062110    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"]
Nov 03 01:54:50.122379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.122322    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc20924a-8101-43ab-95f1-8f0848d649f9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v\" (UID: \"cc20924a-8101-43ab-95f1-8f0848d649f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"
Nov 03 01:54:50.122545 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.122417    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z28sj\" (UniqueName: \"kubernetes.io/projected/cc20924a-8101-43ab-95f1-8f0848d649f9-kube-api-access-z28sj\") pod \"payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v\" (UID: \"cc20924a-8101-43ab-95f1-8f0848d649f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"
Nov 03 01:54:50.122977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.122958    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc20924a-8101-43ab-95f1-8f0848d649f9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v\" (UID: \"cc20924a-8101-43ab-95f1-8f0848d649f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"
Nov 03 01:54:50.179870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.179847    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z28sj\" (UniqueName: \"kubernetes.io/projected/cc20924a-8101-43ab-95f1-8f0848d649f9-kube-api-access-z28sj\") pod \"payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v\" (UID: \"cc20924a-8101-43ab-95f1-8f0848d649f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"
Nov 03 01:54:50.321508 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.321479    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"
Nov 03 01:54:50.378732 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.378693    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw" event={"ID":"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5","Type":"ContainerStarted","Data":"0509316348f441f6970302ededc57b2d61d808b481fb16774f48f8fd10b84df4"}
Nov 03 01:54:50.378891 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.378758    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw" event={"ID":"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5","Type":"ContainerStarted","Data":"550d03952c517236a5b7855d507ffa265d389bb8175a53fb09b480dca35fef32"}
Nov 03 01:54:50.379677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.379655    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"
Nov 03 01:54:50.489461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.489368    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw" podStartSLOduration=1.4893338059999999 podStartE2EDuration="1.489333806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:50.486954067 +0000 UTC m=+2169.912560531" watchObservedRunningTime="2026-11-03 01:54:50.489333806 +0000 UTC m=+2169.914940270"
Nov 03 01:54:50.713260 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:50.713229    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"]
Nov 03 01:54:50.720028 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:54:50.720002    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc20924a_8101_43ab_95f1_8f0848d649f9.slice/crio-c94cf5d364a59e388752ddc47c6e1f3f917d14eea13ef86fbf6f336f0fe8d2aa WatchSource:0}: Error finding container c94cf5d364a59e388752ddc47c6e1f3f917d14eea13ef86fbf6f336f0fe8d2aa: Status 404 returned error can't find the container with id c94cf5d364a59e388752ddc47c6e1f3f917d14eea13ef86fbf6f336f0fe8d2aa
Nov 03 01:54:51.386931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:51.386827    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v" event={"ID":"cc20924a-8101-43ab-95f1-8f0848d649f9","Type":"ContainerStarted","Data":"5f3da6483a0618dd4827e8e363c1647cd219cc741e93a05551439b763ef39851"}
Nov 03 01:54:51.386931 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:51.386891    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v" event={"ID":"cc20924a-8101-43ab-95f1-8f0848d649f9","Type":"ContainerStarted","Data":"c94cf5d364a59e388752ddc47c6e1f3f917d14eea13ef86fbf6f336f0fe8d2aa"}
Nov 03 01:54:51.387515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:51.387089    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"
Nov 03 01:54:51.497197 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:51.497130    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v" podStartSLOduration=2.497111685 podStartE2EDuration="2.497111685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:54:51.495462018 +0000 UTC m=+2170.921068482" watchObservedRunningTime="2026-11-03 01:54:51.497111685 +0000 UTC m=+2170.922718154"
Nov 03 01:54:59.704382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:59.702681    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"]
Nov 03 01:54:59.704382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:54:59.702909    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz" podUID="5e472801-3966-49f6-979d-3c2943800101" containerName="maas-api" containerID="cri-o://4d10e741076b0f8ce81783a4d9ea532a62e1204b4fd71de03acd76462e7e100b" gracePeriod=30
Nov 03 01:55:00.015804 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.015774    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:55:00.115463 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.115429    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"]
Nov 03 01:55:00.115671 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.115636    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7" podUID="636dd427-8666-46d8-a788-2915dc378d5f" containerName="payload-processing" containerID="cri-o://883d0ad8d3388728a51a17d74b655d1596be3add55a6a1af2bf7ecbc26cce7e9" gracePeriod=30
Nov 03 01:55:00.138577 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.138548    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-maas-api-tls\") pod \"5e472801-3966-49f6-979d-3c2943800101\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") "
Nov 03 01:55:00.138692 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.138650    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e472801-3966-49f6-979d-3c2943800101-kube-api-access-xmpt4\" (UniqueName: \"kubernetes.io/projected/5e472801-3966-49f6-979d-3c2943800101-kube-api-access-xmpt4\") pod \"5e472801-3966-49f6-979d-3c2943800101\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") "
Nov 03 01:55:00.138805 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.138769    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-metrics-certs\") pod \"5e472801-3966-49f6-979d-3c2943800101\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") "
Nov 03 01:55:00.138805 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.138790    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e472801-3966-49f6-979d-3c2943800101-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e472801-3966-49f6-979d-3c2943800101-openshift-service-ca\") pod \"5e472801-3966-49f6-979d-3c2943800101\" (UID: \"5e472801-3966-49f6-979d-3c2943800101\") "
Nov 03 01:55:00.139418 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.139391    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e472801-3966-49f6-979d-3c2943800101-openshift-service-ca" pod "5e472801-3966-49f6-979d-3c2943800101" (UID: "5e472801-3966-49f6-979d-3c2943800101"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:00.140479 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.140455    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-maas-api-tls" pod "5e472801-3966-49f6-979d-3c2943800101" (UID: "5e472801-3966-49f6-979d-3c2943800101"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:55:00.140758 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.140735    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e472801-3966-49f6-979d-3c2943800101-kube-api-access-xmpt4" pod "5e472801-3966-49f6-979d-3c2943800101" (UID: "5e472801-3966-49f6-979d-3c2943800101"). InnerVolumeSpecName "kube-api-access-xmpt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:00.140947 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.140932    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-metrics-certs" pod "5e472801-3966-49f6-979d-3c2943800101" (UID: "5e472801-3966-49f6-979d-3c2943800101"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:55:00.168124 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.168099    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"]
Nov 03 01:55:00.168301 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.168281    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm" podUID="b09aad0d-ebb8-4fec-898f-a1e47d1dfd95" containerName="payload-pre-processing" containerID="cri-o://7e0087e5b146cfca428c2605bba7f941824b01708664a4f924eb051ed65ee8a6" gracePeriod=30
Nov 03 01:55:00.240310 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.240244    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:00.240310 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.240272    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmpt4\" (UniqueName: \"kubernetes.io/projected/5e472801-3966-49f6-979d-3c2943800101-kube-api-access-xmpt4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:00.240310 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.240282    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e472801-3966-49f6-979d-3c2943800101-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:00.240310 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.240293    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e472801-3966-49f6-979d-3c2943800101-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:00.436019 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.435994    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"
Nov 03 01:55:00.447771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.447749    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"
Nov 03 01:55:00.452533 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.452509    5508 generic.go:356] "Generic (PLEG): container finished" podID="5e472801-3966-49f6-979d-3c2943800101" containerID="4d10e741076b0f8ce81783a4d9ea532a62e1204b4fd71de03acd76462e7e100b" exitCode=0
Nov 03 01:55:00.452658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.452576    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"
Nov 03 01:55:00.452658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.452615    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz" event={"ID":"5e472801-3966-49f6-979d-3c2943800101","Type":"ContainerDied","Data":"4d10e741076b0f8ce81783a4d9ea532a62e1204b4fd71de03acd76462e7e100b"}
Nov 03 01:55:00.452779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.452665    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz" event={"ID":"5e472801-3966-49f6-979d-3c2943800101","Type":"ContainerDied","Data":"7cdde1073e70bb06169d098b716715ffb76b2add1840d328473815b40a6beda8"}
Nov 03 01:55:00.452779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.452694    5508 scope.go:122] "RemoveContainer" containerID="4d10e741076b0f8ce81783a4d9ea532a62e1204b4fd71de03acd76462e7e100b"
Nov 03 01:55:00.454164 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.454143    5508 generic.go:356] "Generic (PLEG): container finished" podID="b09aad0d-ebb8-4fec-898f-a1e47d1dfd95" containerID="7e0087e5b146cfca428c2605bba7f941824b01708664a4f924eb051ed65ee8a6" exitCode=0
Nov 03 01:55:00.454270 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.454183    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"
Nov 03 01:55:00.454331 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.454294    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm" event={"ID":"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95","Type":"ContainerDied","Data":"7e0087e5b146cfca428c2605bba7f941824b01708664a4f924eb051ed65ee8a6"}
Nov 03 01:55:00.454435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.454330    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm" event={"ID":"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95","Type":"ContainerDied","Data":"722b72136ff68af48d7627ba614bc87185a3828a93fc0cbe977554f0accd0ba5"}
Nov 03 01:55:00.455641 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.455621    5508 generic.go:356] "Generic (PLEG): container finished" podID="636dd427-8666-46d8-a788-2915dc378d5f" containerID="883d0ad8d3388728a51a17d74b655d1596be3add55a6a1af2bf7ecbc26cce7e9" exitCode=0
Nov 03 01:55:00.455755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.455668    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"
Nov 03 01:55:00.455755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.455674    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7" event={"ID":"636dd427-8666-46d8-a788-2915dc378d5f","Type":"ContainerDied","Data":"883d0ad8d3388728a51a17d74b655d1596be3add55a6a1af2bf7ecbc26cce7e9"}
Nov 03 01:55:00.455755 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.455709    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7" event={"ID":"636dd427-8666-46d8-a788-2915dc378d5f","Type":"ContainerDied","Data":"e8fd656cb4b0a10c370101965765677a0bfcbf3ee333724ce359f01d65208901"}
Nov 03 01:55:00.465379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.465332    5508 scope.go:122] "RemoveContainer" containerID="4d10e741076b0f8ce81783a4d9ea532a62e1204b4fd71de03acd76462e7e100b"
Nov 03 01:55:00.465623 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:00.465603    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d10e741076b0f8ce81783a4d9ea532a62e1204b4fd71de03acd76462e7e100b\": container with ID starting with 4d10e741076b0f8ce81783a4d9ea532a62e1204b4fd71de03acd76462e7e100b not found: ID does not exist" containerID="4d10e741076b0f8ce81783a4d9ea532a62e1204b4fd71de03acd76462e7e100b"
Nov 03 01:55:00.465708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.465629    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d10e741076b0f8ce81783a4d9ea532a62e1204b4fd71de03acd76462e7e100b"} err="failed to get container status \"4d10e741076b0f8ce81783a4d9ea532a62e1204b4fd71de03acd76462e7e100b\": rpc error: code = NotFound desc = could not find container \"4d10e741076b0f8ce81783a4d9ea532a62e1204b4fd71de03acd76462e7e100b\": container with ID starting with 4d10e741076b0f8ce81783a4d9ea532a62e1204b4fd71de03acd76462e7e100b not found: ID does not exist"
Nov 03 01:55:00.465708 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.465647    5508 scope.go:122] "RemoveContainer" containerID="7e0087e5b146cfca428c2605bba7f941824b01708664a4f924eb051ed65ee8a6"
Nov 03 01:55:00.474695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.474674    5508 scope.go:122] "RemoveContainer" containerID="7e0087e5b146cfca428c2605bba7f941824b01708664a4f924eb051ed65ee8a6"
Nov 03 01:55:00.474966 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:00.474947    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0087e5b146cfca428c2605bba7f941824b01708664a4f924eb051ed65ee8a6\": container with ID starting with 7e0087e5b146cfca428c2605bba7f941824b01708664a4f924eb051ed65ee8a6 not found: ID does not exist" containerID="7e0087e5b146cfca428c2605bba7f941824b01708664a4f924eb051ed65ee8a6"
Nov 03 01:55:00.475046 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.474972    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0087e5b146cfca428c2605bba7f941824b01708664a4f924eb051ed65ee8a6"} err="failed to get container status \"7e0087e5b146cfca428c2605bba7f941824b01708664a4f924eb051ed65ee8a6\": rpc error: code = NotFound desc = could not find container \"7e0087e5b146cfca428c2605bba7f941824b01708664a4f924eb051ed65ee8a6\": container with ID starting with 7e0087e5b146cfca428c2605bba7f941824b01708664a4f924eb051ed65ee8a6 not found: ID does not exist"
Nov 03 01:55:00.475046 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.474987    5508 scope.go:122] "RemoveContainer" containerID="883d0ad8d3388728a51a17d74b655d1596be3add55a6a1af2bf7ecbc26cce7e9"
Nov 03 01:55:00.483149 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.483120    5508 scope.go:122] "RemoveContainer" containerID="883d0ad8d3388728a51a17d74b655d1596be3add55a6a1af2bf7ecbc26cce7e9"
Nov 03 01:55:00.483432 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:00.483412    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883d0ad8d3388728a51a17d74b655d1596be3add55a6a1af2bf7ecbc26cce7e9\": container with ID starting with 883d0ad8d3388728a51a17d74b655d1596be3add55a6a1af2bf7ecbc26cce7e9 not found: ID does not exist" containerID="883d0ad8d3388728a51a17d74b655d1596be3add55a6a1af2bf7ecbc26cce7e9"
Nov 03 01:55:00.483499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.483438    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883d0ad8d3388728a51a17d74b655d1596be3add55a6a1af2bf7ecbc26cce7e9"} err="failed to get container status \"883d0ad8d3388728a51a17d74b655d1596be3add55a6a1af2bf7ecbc26cce7e9\": rpc error: code = NotFound desc = could not find container \"883d0ad8d3388728a51a17d74b655d1596be3add55a6a1af2bf7ecbc26cce7e9\": container with ID starting with 883d0ad8d3388728a51a17d74b655d1596be3add55a6a1af2bf7ecbc26cce7e9 not found: ID does not exist"
Nov 03 01:55:00.543069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.543022    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636dd427-8666-46d8-a788-2915dc378d5f-kube-api-access-hfbrm\" (UniqueName: \"kubernetes.io/projected/636dd427-8666-46d8-a788-2915dc378d5f-kube-api-access-hfbrm\") pod \"636dd427-8666-46d8-a788-2915dc378d5f\" (UID: \"636dd427-8666-46d8-a788-2915dc378d5f\") "
Nov 03 01:55:00.543069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.543053    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-plugins-config-volume\") pod \"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95\" (UID: \"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95\") "
Nov 03 01:55:00.543200 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.543098    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636dd427-8666-46d8-a788-2915dc378d5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636dd427-8666-46d8-a788-2915dc378d5f-plugins-config-volume\") pod \"636dd427-8666-46d8-a788-2915dc378d5f\" (UID: \"636dd427-8666-46d8-a788-2915dc378d5f\") "
Nov 03 01:55:00.543200 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.543165    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-kube-api-access-2v8pw\" (UniqueName: \"kubernetes.io/projected/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-kube-api-access-2v8pw\") pod \"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95\" (UID: \"b09aad0d-ebb8-4fec-898f-a1e47d1dfd95\") "
Nov 03 01:55:00.543466 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.543439    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-plugins-config-volume" pod "b09aad0d-ebb8-4fec-898f-a1e47d1dfd95" (UID: "b09aad0d-ebb8-4fec-898f-a1e47d1dfd95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:00.543569 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.543514    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636dd427-8666-46d8-a788-2915dc378d5f-plugins-config-volume" pod "636dd427-8666-46d8-a788-2915dc378d5f" (UID: "636dd427-8666-46d8-a788-2915dc378d5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:00.543646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.543625    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:00.543686 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.543649    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/636dd427-8666-46d8-a788-2915dc378d5f-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:00.545197 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.545169    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636dd427-8666-46d8-a788-2915dc378d5f-kube-api-access-hfbrm" pod "636dd427-8666-46d8-a788-2915dc378d5f" (UID: "636dd427-8666-46d8-a788-2915dc378d5f"). InnerVolumeSpecName "kube-api-access-hfbrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:00.545322 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.545255    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-kube-api-access-2v8pw" pod "b09aad0d-ebb8-4fec-898f-a1e47d1dfd95" (UID: "b09aad0d-ebb8-4fec-898f-a1e47d1dfd95"). InnerVolumeSpecName "kube-api-access-2v8pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:00.655381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.653180    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfbrm\" (UniqueName: \"kubernetes.io/projected/636dd427-8666-46d8-a788-2915dc378d5f-kube-api-access-hfbrm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:00.655381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.653239    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v8pw\" (UniqueName: \"kubernetes.io/projected/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95-kube-api-access-2v8pw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:00.726570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.726533    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"]
Nov 03 01:55:00.729283 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.729256    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-23e673-7c968d8988-2d7sz"]
Nov 03 01:55:00.803034 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.802975    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"]
Nov 03 01:55:00.803175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.803143    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f" podUID="e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" containerName="istio-proxy" containerID="cri-o://f753c15fa19c15434f0cc885f7250947c1fa3b125d55d6fe3f103c3f0f9c6e12" gracePeriod=30
Nov 03 01:55:00.944453 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.944422    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"]
Nov 03 01:55:00.949321 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:00.949295    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-23e673-77bbbf4466-n5gt7"]
Nov 03 01:55:01.049465 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:01.049434    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"]
Nov 03 01:55:01.053846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:01.053787    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-23e673-56996898cf-d2wxm"]
Nov 03 01:55:02.397865 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:02.397824    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"
Nov 03 01:55:02.398237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:02.397906    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"
Nov 03 01:55:02.693981 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:02.693895    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e472801-3966-49f6-979d-3c2943800101" path="/var/lib/kubelet/pods/5e472801-3966-49f6-979d-3c2943800101/volumes"
Nov 03 01:55:02.694417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:02.694395    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636dd427-8666-46d8-a788-2915dc378d5f" path="/var/lib/kubelet/pods/636dd427-8666-46d8-a788-2915dc378d5f/volumes"
Nov 03 01:55:02.694859 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:02.694839    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b09aad0d-ebb8-4fec-898f-a1e47d1dfd95" path="/var/lib/kubelet/pods/b09aad0d-ebb8-4fec-898f-a1e47d1dfd95/volumes"
Nov 03 01:55:03.077053 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:03.076968    5508 patch_prober.go:29] interesting pod/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:55:03.077053 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:03.077029    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f" podUID="e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:55:04.459319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:04.459283    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"]
Nov 03 01:55:04.561281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:04.561249    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"]
Nov 03 01:55:06.140610 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.140581    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:55:06.243370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.241791    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-credential-socket\") pod \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") "
Nov 03 01:55:06.243370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.241864    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-token\" (UniqueName: \"kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-token\") pod \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") "
Nov 03 01:55:06.243370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.241947    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-podinfo\") pod \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") "
Nov 03 01:55:06.243370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.242000    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-envoy\") pod \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") "
Nov 03 01:55:06.243370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.242059    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-certs\") pod \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") "
Nov 03 01:55:06.243370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.242125    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istiod-ca-cert\") pod \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") "
Nov 03 01:55:06.243370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.242180    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-socket\") pod \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") "
Nov 03 01:55:06.243370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.242235    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-data\") pod \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") "
Nov 03 01:55:06.243370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.242285    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-kube-api-access-bnnxq\" (UniqueName: \"kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-kube-api-access-bnnxq\") pod \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\" (UID: \"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857\") "
Nov 03 01:55:06.247934 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.247302    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-socket" pod "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" (UID: "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:06.247934 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.247520    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-data" pod "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" (UID: "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:06.247934 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.247699    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-certs" pod "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" (UID: "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:06.248431 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.248392    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-credential-socket" pod "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" (UID: "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:06.254572 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.254546    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istiod-ca-cert" pod "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" (UID: "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:06.269631 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.269488    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-kube-api-access-bnnxq" pod "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" (UID: "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857"). InnerVolumeSpecName "kube-api-access-bnnxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:06.269830 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.269795    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-podinfo" pod "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" (UID: "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:55:06.269918 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.269825    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-token" pod "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" (UID: "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:06.274335 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.271470    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-envoy" pod "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" (UID: "e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:06.344049 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.344021    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:06.344199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.344068    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:06.344199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.344077    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnnxq\" (UniqueName: \"kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-kube-api-access-bnnxq\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:06.344199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.344088    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:06.344199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.344097    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:06.344199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.344106    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:06.344199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.344113    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:06.344199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.344121    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:06.344199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.344129    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:06.513522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.513482    5508 generic.go:356] "Generic (PLEG): container finished" podID="e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" containerID="f753c15fa19c15434f0cc885f7250947c1fa3b125d55d6fe3f103c3f0f9c6e12" exitCode=0
Nov 03 01:55:06.513760 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.513740    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f" event={"ID":"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857","Type":"ContainerDied","Data":"f753c15fa19c15434f0cc885f7250947c1fa3b125d55d6fe3f103c3f0f9c6e12"}
Nov 03 01:55:06.513854 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.513837    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f" event={"ID":"e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857","Type":"ContainerDied","Data":"5b68817c1bfbac10dbdeecc6d11303ea2ce9b142d53bb1bab69f6d0557314f5d"}
Nov 03 01:55:06.513914 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.513869    5508 scope.go:122] "RemoveContainer" containerID="f753c15fa19c15434f0cc885f7250947c1fa3b125d55d6fe3f103c3f0f9c6e12"
Nov 03 01:55:06.514045 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.514023    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"
Nov 03 01:55:06.541977 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.541958    5508 scope.go:122] "RemoveContainer" containerID="f753c15fa19c15434f0cc885f7250947c1fa3b125d55d6fe3f103c3f0f9c6e12"
Nov 03 01:55:06.542463 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:06.542436    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f753c15fa19c15434f0cc885f7250947c1fa3b125d55d6fe3f103c3f0f9c6e12\": container with ID starting with f753c15fa19c15434f0cc885f7250947c1fa3b125d55d6fe3f103c3f0f9c6e12 not found: ID does not exist" containerID="f753c15fa19c15434f0cc885f7250947c1fa3b125d55d6fe3f103c3f0f9c6e12"
Nov 03 01:55:06.542567 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.542475    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f753c15fa19c15434f0cc885f7250947c1fa3b125d55d6fe3f103c3f0f9c6e12"} err="failed to get container status \"f753c15fa19c15434f0cc885f7250947c1fa3b125d55d6fe3f103c3f0f9c6e12\": rpc error: code = NotFound desc = could not find container \"f753c15fa19c15434f0cc885f7250947c1fa3b125d55d6fe3f103c3f0f9c6e12\": container with ID starting with f753c15fa19c15434f0cc885f7250947c1fa3b125d55d6fe3f103c3f0f9c6e12 not found: ID does not exist"
Nov 03 01:55:06.677813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.677776    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"]
Nov 03 01:55:06.683857 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.683828    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-41aa5f-openshift-default-55dc4f4f9-rpb6f"]
Nov 03 01:55:06.691951 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:06.691925    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" path="/var/lib/kubelet/pods/e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857/volumes"
Nov 03 01:55:07.161162 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:07.161122    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"]
Nov 03 01:55:07.161594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:07.161417    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy" containerID="cri-o://1f6c9691160fa88ac5720164ad35dfb882aeb2027d265a19153b53c030563bfc" gracePeriod=30
Nov 03 01:55:09.459946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.459899    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t" podUID="63a58d16-d2ab-46bb-bb83-9339e9f1e0b4" containerName="payload-processing" containerID="cri-o://8bcb998798013965f4631b246bfbb2fbf023e15e7af19af17223b5bc75f2f8bf" gracePeriod=25
Nov 03 01:55:09.562117 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.562077    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8" podUID="11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96" containerName="payload-pre-processing" containerID="cri-o://384e2434d841e819ec672c5b26b42ee3c71d6285cdb93d8a7748df70f7e5b6d3" gracePeriod=25
Nov 03 01:55:09.734693 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.734669    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"
Nov 03 01:55:09.797987 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.797956    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-kube-api-access-7ggqc\" (UniqueName: \"kubernetes.io/projected/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-kube-api-access-7ggqc\") pod \"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4\" (UID: \"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4\") "
Nov 03 01:55:09.798152 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.798108    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-plugins-config-volume\") pod \"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4\" (UID: \"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4\") "
Nov 03 01:55:09.798488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.798465    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-plugins-config-volume" pod "63a58d16-d2ab-46bb-bb83-9339e9f1e0b4" (UID: "63a58d16-d2ab-46bb-bb83-9339e9f1e0b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:09.800636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.800610    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-kube-api-access-7ggqc" pod "63a58d16-d2ab-46bb-bb83-9339e9f1e0b4" (UID: "63a58d16-d2ab-46bb-bb83-9339e9f1e0b4"). InnerVolumeSpecName "kube-api-access-7ggqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:09.830694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.830674    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"
Nov 03 01:55:09.899005 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.898977    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-kube-api-access-j8zcv\" (UniqueName: \"kubernetes.io/projected/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-kube-api-access-j8zcv\") pod \"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96\" (UID: \"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96\") "
Nov 03 01:55:09.899167 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.899068    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-plugins-config-volume\") pod \"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96\" (UID: \"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96\") "
Nov 03 01:55:09.899505 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.899480    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-plugins-config-volume" pod "11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96" (UID: "11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:09.899596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.899575    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:09.899596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.899588    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ggqc\" (UniqueName: \"kubernetes.io/projected/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4-kube-api-access-7ggqc\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:09.899709 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.899603    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:09.901033 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:09.901010    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-kube-api-access-j8zcv" pod "11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96" (UID: "11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96"). InnerVolumeSpecName "kube-api-access-j8zcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:10.000812 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.000735    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8zcv\" (UniqueName: \"kubernetes.io/projected/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96-kube-api-access-j8zcv\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:10.541921 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.541891    5508 generic.go:356] "Generic (PLEG): container finished" podID="63a58d16-d2ab-46bb-bb83-9339e9f1e0b4" containerID="8bcb998798013965f4631b246bfbb2fbf023e15e7af19af17223b5bc75f2f8bf" exitCode=0
Nov 03 01:55:10.542385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.541935    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"
Nov 03 01:55:10.542385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.541973    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t" event={"ID":"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4","Type":"ContainerDied","Data":"8bcb998798013965f4631b246bfbb2fbf023e15e7af19af17223b5bc75f2f8bf"}
Nov 03 01:55:10.542385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.542054    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t" event={"ID":"63a58d16-d2ab-46bb-bb83-9339e9f1e0b4","Type":"ContainerDied","Data":"110aa445f5d36fb01cc86f754be2ed7be3286744fae4794dbce4fe9c9cb3a1ae"}
Nov 03 01:55:10.542385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.542080    5508 scope.go:122] "RemoveContainer" containerID="8bcb998798013965f4631b246bfbb2fbf023e15e7af19af17223b5bc75f2f8bf"
Nov 03 01:55:10.543488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.543438    5508 generic.go:356] "Generic (PLEG): container finished" podID="11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96" containerID="384e2434d841e819ec672c5b26b42ee3c71d6285cdb93d8a7748df70f7e5b6d3" exitCode=0
Nov 03 01:55:10.543563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.543491    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"
Nov 03 01:55:10.543622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.543570    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8" event={"ID":"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96","Type":"ContainerDied","Data":"384e2434d841e819ec672c5b26b42ee3c71d6285cdb93d8a7748df70f7e5b6d3"}
Nov 03 01:55:10.543622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.543615    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8" event={"ID":"11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96","Type":"ContainerDied","Data":"c1014433bc36beb50ee0d4ea9f063c8af17cb0c3335ddcb472d389239709a2a9"}
Nov 03 01:55:10.553444 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.553418    5508 scope.go:122] "RemoveContainer" containerID="8bcb998798013965f4631b246bfbb2fbf023e15e7af19af17223b5bc75f2f8bf"
Nov 03 01:55:10.554118 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:10.554002    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bcb998798013965f4631b246bfbb2fbf023e15e7af19af17223b5bc75f2f8bf\": container with ID starting with 8bcb998798013965f4631b246bfbb2fbf023e15e7af19af17223b5bc75f2f8bf not found: ID does not exist" containerID="8bcb998798013965f4631b246bfbb2fbf023e15e7af19af17223b5bc75f2f8bf"
Nov 03 01:55:10.554118 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.554035    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bcb998798013965f4631b246bfbb2fbf023e15e7af19af17223b5bc75f2f8bf"} err="failed to get container status \"8bcb998798013965f4631b246bfbb2fbf023e15e7af19af17223b5bc75f2f8bf\": rpc error: code = NotFound desc = could not find container \"8bcb998798013965f4631b246bfbb2fbf023e15e7af19af17223b5bc75f2f8bf\": container with ID starting with 8bcb998798013965f4631b246bfbb2fbf023e15e7af19af17223b5bc75f2f8bf not found: ID does not exist"
Nov 03 01:55:10.554118 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.554056    5508 scope.go:122] "RemoveContainer" containerID="384e2434d841e819ec672c5b26b42ee3c71d6285cdb93d8a7748df70f7e5b6d3"
Nov 03 01:55:10.563231 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.563215    5508 scope.go:122] "RemoveContainer" containerID="384e2434d841e819ec672c5b26b42ee3c71d6285cdb93d8a7748df70f7e5b6d3"
Nov 03 01:55:10.563568 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:10.563549    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384e2434d841e819ec672c5b26b42ee3c71d6285cdb93d8a7748df70f7e5b6d3\": container with ID starting with 384e2434d841e819ec672c5b26b42ee3c71d6285cdb93d8a7748df70f7e5b6d3 not found: ID does not exist" containerID="384e2434d841e819ec672c5b26b42ee3c71d6285cdb93d8a7748df70f7e5b6d3"
Nov 03 01:55:10.563644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.563576    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"384e2434d841e819ec672c5b26b42ee3c71d6285cdb93d8a7748df70f7e5b6d3"} err="failed to get container status \"384e2434d841e819ec672c5b26b42ee3c71d6285cdb93d8a7748df70f7e5b6d3\": rpc error: code = NotFound desc = could not find container \"384e2434d841e819ec672c5b26b42ee3c71d6285cdb93d8a7748df70f7e5b6d3\": container with ID starting with 384e2434d841e819ec672c5b26b42ee3c71d6285cdb93d8a7748df70f7e5b6d3 not found: ID does not exist"
Nov 03 01:55:10.698523 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.698486    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"]
Nov 03 01:55:10.743380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.743291    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-5d4d9d8479-92bk8"]
Nov 03 01:55:10.847528 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.847495    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"]
Nov 03 01:55:10.854400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:10.854369    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738acb-dd5dbfff9-jth5t"]
Nov 03 01:55:12.449672 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.449604    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:55:12.526017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.525984    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-certs\") pod \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") "
Nov 03 01:55:12.526196 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526050    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-credential-socket\") pod \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") "
Nov 03 01:55:12.526196 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526080    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-socket\") pod \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") "
Nov 03 01:55:12.526196 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526106    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-envoy\") pod \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") "
Nov 03 01:55:12.526196 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526151    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-podinfo\") pod \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") "
Nov 03 01:55:12.526196 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526182    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-kube-api-access-bhw9q\" (UniqueName: \"kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-kube-api-access-bhw9q\") pod \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") "
Nov 03 01:55:12.526514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526261    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istiod-ca-cert\") pod \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") "
Nov 03 01:55:12.526514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526291    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-certs" pod "774d3ce9-baeb-4de2-837b-62b8a559b7b3" (UID: "774d3ce9-baeb-4de2-837b-62b8a559b7b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:12.526514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526315    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-token\" (UniqueName: \"kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-token\") pod \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") "
Nov 03 01:55:12.526514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526388    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-socket" pod "774d3ce9-baeb-4de2-837b-62b8a559b7b3" (UID: "774d3ce9-baeb-4de2-837b-62b8a559b7b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:12.526514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526404    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-credential-socket" pod "774d3ce9-baeb-4de2-837b-62b8a559b7b3" (UID: "774d3ce9-baeb-4de2-837b-62b8a559b7b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:12.526514 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526433    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-data\") pod \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\" (UID: \"774d3ce9-baeb-4de2-837b-62b8a559b7b3\") "
Nov 03 01:55:12.526821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526668    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istiod-ca-cert" pod "774d3ce9-baeb-4de2-837b-62b8a559b7b3" (UID: "774d3ce9-baeb-4de2-837b-62b8a559b7b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:12.526821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526747    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-data" pod "774d3ce9-baeb-4de2-837b-62b8a559b7b3" (UID: "774d3ce9-baeb-4de2-837b-62b8a559b7b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:12.526942 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526918    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:12.526996 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526939    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:12.526996 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526954    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:12.526996 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526969    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:12.526996 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.526981    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:12.528705 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.528677    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-kube-api-access-bhw9q" pod "774d3ce9-baeb-4de2-837b-62b8a559b7b3" (UID: "774d3ce9-baeb-4de2-837b-62b8a559b7b3"). InnerVolumeSpecName "kube-api-access-bhw9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:12.529381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.528796    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-podinfo" pod "774d3ce9-baeb-4de2-837b-62b8a559b7b3" (UID: "774d3ce9-baeb-4de2-837b-62b8a559b7b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:55:12.529381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.528800    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-token" pod "774d3ce9-baeb-4de2-837b-62b8a559b7b3" (UID: "774d3ce9-baeb-4de2-837b-62b8a559b7b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:12.529381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.528882    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-envoy" pod "774d3ce9-baeb-4de2-837b-62b8a559b7b3" (UID: "774d3ce9-baeb-4de2-837b-62b8a559b7b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:12.560565 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.560542    5508 generic.go:356] "Generic (PLEG): container finished" podID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerID="1f6c9691160fa88ac5720164ad35dfb882aeb2027d265a19153b53c030563bfc" exitCode=0
Nov 03 01:55:12.560681 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.560596    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" event={"ID":"774d3ce9-baeb-4de2-837b-62b8a559b7b3","Type":"ContainerDied","Data":"1f6c9691160fa88ac5720164ad35dfb882aeb2027d265a19153b53c030563bfc"}
Nov 03 01:55:12.560681 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.560614    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"
Nov 03 01:55:12.560681 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.560654    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2" event={"ID":"774d3ce9-baeb-4de2-837b-62b8a559b7b3","Type":"ContainerDied","Data":"216a9d82958aa5399eb56c0c18d5ce433359713bb398a2b67aa617522a10c277"}
Nov 03 01:55:12.560681 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.560677    5508 scope.go:122] "RemoveContainer" containerID="1f6c9691160fa88ac5720164ad35dfb882aeb2027d265a19153b53c030563bfc"
Nov 03 01:55:12.570171 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.570152    5508 scope.go:122] "RemoveContainer" containerID="50fba1e546916d1e949d2ddd2234df40bb647848220f5e96ea1aefb0f2069702"
Nov 03 01:55:12.579300 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.579284    5508 scope.go:122] "RemoveContainer" containerID="1f6c9691160fa88ac5720164ad35dfb882aeb2027d265a19153b53c030563bfc"
Nov 03 01:55:12.579578 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:12.579553    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6c9691160fa88ac5720164ad35dfb882aeb2027d265a19153b53c030563bfc\": container with ID starting with 1f6c9691160fa88ac5720164ad35dfb882aeb2027d265a19153b53c030563bfc not found: ID does not exist" containerID="1f6c9691160fa88ac5720164ad35dfb882aeb2027d265a19153b53c030563bfc"
Nov 03 01:55:12.579684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.579588    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6c9691160fa88ac5720164ad35dfb882aeb2027d265a19153b53c030563bfc"} err="failed to get container status \"1f6c9691160fa88ac5720164ad35dfb882aeb2027d265a19153b53c030563bfc\": rpc error: code = NotFound desc = could not find container \"1f6c9691160fa88ac5720164ad35dfb882aeb2027d265a19153b53c030563bfc\": container with ID starting with 1f6c9691160fa88ac5720164ad35dfb882aeb2027d265a19153b53c030563bfc not found: ID does not exist"
Nov 03 01:55:12.579684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.579605    5508 scope.go:122] "RemoveContainer" containerID="50fba1e546916d1e949d2ddd2234df40bb647848220f5e96ea1aefb0f2069702"
Nov 03 01:55:12.579872 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:12.579857    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50fba1e546916d1e949d2ddd2234df40bb647848220f5e96ea1aefb0f2069702\": container with ID starting with 50fba1e546916d1e949d2ddd2234df40bb647848220f5e96ea1aefb0f2069702 not found: ID does not exist" containerID="50fba1e546916d1e949d2ddd2234df40bb647848220f5e96ea1aefb0f2069702"
Nov 03 01:55:12.579912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.579875    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50fba1e546916d1e949d2ddd2234df40bb647848220f5e96ea1aefb0f2069702"} err="failed to get container status \"50fba1e546916d1e949d2ddd2234df40bb647848220f5e96ea1aefb0f2069702\": rpc error: code = NotFound desc = could not find container \"50fba1e546916d1e949d2ddd2234df40bb647848220f5e96ea1aefb0f2069702\": container with ID starting with 50fba1e546916d1e949d2ddd2234df40bb647848220f5e96ea1aefb0f2069702 not found: ID does not exist"
Nov 03 01:55:12.628388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.628342    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:12.628388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.628378    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:12.628388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.628387    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774d3ce9-baeb-4de2-837b-62b8a559b7b3-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:12.628388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.628396    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhw9q\" (UniqueName: \"kubernetes.io/projected/774d3ce9-baeb-4de2-837b-62b8a559b7b3-kube-api-access-bhw9q\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:12.693451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.693423    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96" path="/var/lib/kubelet/pods/11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96/volumes"
Nov 03 01:55:12.693886 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.693864    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a58d16-d2ab-46bb-bb83-9339e9f1e0b4" path="/var/lib/kubelet/pods/63a58d16-d2ab-46bb-bb83-9339e9f1e0b4/volumes"
Nov 03 01:55:12.716372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.716301    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"]
Nov 03 01:55:12.720584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.720553    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-23e673-openshift-default-6bf677976c-7bck2"]
Nov 03 01:55:12.798726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:12.798693    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:55:13.811150 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:13.811115    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:55:14.693056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:14.693026    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" path="/var/lib/kubelet/pods/774d3ce9-baeb-4de2-837b-62b8a559b7b3/volumes"
Nov 03 01:55:21.873762 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:21.873725    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"]
Nov 03 01:55:22.494400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.494363    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"]
Nov 03 01:55:22.494626 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.494586    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25" podUID="a67f1cc6-9db6-4464-9f0e-1392b5be85cd" containerName="payload-processing" containerID="cri-o://a01305753f04c3659fbc7b61cb41d83e4562d8398459bfd60cb38f2fe02b536f" gracePeriod=30
Nov 03 01:55:22.595078 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.595046    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"]
Nov 03 01:55:22.595311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.595254    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p" podUID="c3b4fd3a-9cf2-489f-89cd-59b57cddfc06" containerName="payload-pre-processing" containerID="cri-o://87a5880a56857da473302a0d3777578adca1dd8dad665d001307a5a85c5cca5d" gracePeriod=30
Nov 03 01:55:22.625621 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.625592    5508 generic.go:356] "Generic (PLEG): container finished" podID="a67f1cc6-9db6-4464-9f0e-1392b5be85cd" containerID="a01305753f04c3659fbc7b61cb41d83e4562d8398459bfd60cb38f2fe02b536f" exitCode=0
Nov 03 01:55:22.625757 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.625675    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25" event={"ID":"a67f1cc6-9db6-4464-9f0e-1392b5be85cd","Type":"ContainerDied","Data":"a01305753f04c3659fbc7b61cb41d83e4562d8398459bfd60cb38f2fe02b536f"}
Nov 03 01:55:22.794059 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.794031    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"
Nov 03 01:55:22.858286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.858256    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-kube-api-access-hzrdc\" (UniqueName: \"kubernetes.io/projected/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-kube-api-access-hzrdc\") pod \"a67f1cc6-9db6-4464-9f0e-1392b5be85cd\" (UID: \"a67f1cc6-9db6-4464-9f0e-1392b5be85cd\") "
Nov 03 01:55:22.858521 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.858502    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-plugins-config-volume\") pod \"a67f1cc6-9db6-4464-9f0e-1392b5be85cd\" (UID: \"a67f1cc6-9db6-4464-9f0e-1392b5be85cd\") "
Nov 03 01:55:22.858845 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.858820    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-plugins-config-volume" pod "a67f1cc6-9db6-4464-9f0e-1392b5be85cd" (UID: "a67f1cc6-9db6-4464-9f0e-1392b5be85cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:22.858993 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.858911    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:22.861090 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.861063    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-kube-api-access-hzrdc" pod "a67f1cc6-9db6-4464-9f0e-1392b5be85cd" (UID: "a67f1cc6-9db6-4464-9f0e-1392b5be85cd"). InnerVolumeSpecName "kube-api-access-hzrdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:22.885576 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.885557    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"
Nov 03 01:55:22.959838 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.959793    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-kube-api-access-qvwz5\" (UniqueName: \"kubernetes.io/projected/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-kube-api-access-qvwz5\") pod \"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06\" (UID: \"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06\") "
Nov 03 01:55:22.960058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.959916    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-plugins-config-volume\") pod \"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06\" (UID: \"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06\") "
Nov 03 01:55:22.960311 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.960285    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-plugins-config-volume" pod "c3b4fd3a-9cf2-489f-89cd-59b57cddfc06" (UID: "c3b4fd3a-9cf2-489f-89cd-59b57cddfc06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:22.960413 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.960373    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzrdc\" (UniqueName: \"kubernetes.io/projected/a67f1cc6-9db6-4464-9f0e-1392b5be85cd-kube-api-access-hzrdc\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:22.961920 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:22.961884    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-kube-api-access-qvwz5" pod "c3b4fd3a-9cf2-489f-89cd-59b57cddfc06" (UID: "c3b4fd3a-9cf2-489f-89cd-59b57cddfc06"). InnerVolumeSpecName "kube-api-access-qvwz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:23.077425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.063183    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:23.077425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.063219    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvwz5\" (UniqueName: \"kubernetes.io/projected/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06-kube-api-access-qvwz5\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:23.634477 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.634439    5508 generic.go:356] "Generic (PLEG): container finished" podID="c3b4fd3a-9cf2-489f-89cd-59b57cddfc06" containerID="87a5880a56857da473302a0d3777578adca1dd8dad665d001307a5a85c5cca5d" exitCode=0
Nov 03 01:55:23.634654 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.634488    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"
Nov 03 01:55:23.634654 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.634527    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p" event={"ID":"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06","Type":"ContainerDied","Data":"87a5880a56857da473302a0d3777578adca1dd8dad665d001307a5a85c5cca5d"}
Nov 03 01:55:23.634654 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.634579    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p" event={"ID":"c3b4fd3a-9cf2-489f-89cd-59b57cddfc06","Type":"ContainerDied","Data":"88a7bf471e6fae4b5c9dcf6375804494a30c319c0c7c866d1aab70b5ded8b036"}
Nov 03 01:55:23.634654 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.634616    5508 scope.go:122] "RemoveContainer" containerID="87a5880a56857da473302a0d3777578adca1dd8dad665d001307a5a85c5cca5d"
Nov 03 01:55:23.636224 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.636018    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"
Nov 03 01:55:23.636224 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.636026    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25" event={"ID":"a67f1cc6-9db6-4464-9f0e-1392b5be85cd","Type":"ContainerDied","Data":"4525b809c6af4d6583b1c4a7f075abbd439c1f272fa9deb6b2a7ff0077bd914e"}
Nov 03 01:55:23.646816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.646790    5508 scope.go:122] "RemoveContainer" containerID="87a5880a56857da473302a0d3777578adca1dd8dad665d001307a5a85c5cca5d"
Nov 03 01:55:23.647565 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:23.647532    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a5880a56857da473302a0d3777578adca1dd8dad665d001307a5a85c5cca5d\": container with ID starting with 87a5880a56857da473302a0d3777578adca1dd8dad665d001307a5a85c5cca5d not found: ID does not exist" containerID="87a5880a56857da473302a0d3777578adca1dd8dad665d001307a5a85c5cca5d"
Nov 03 01:55:23.647687 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.647578    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a5880a56857da473302a0d3777578adca1dd8dad665d001307a5a85c5cca5d"} err="failed to get container status \"87a5880a56857da473302a0d3777578adca1dd8dad665d001307a5a85c5cca5d\": rpc error: code = NotFound desc = could not find container \"87a5880a56857da473302a0d3777578adca1dd8dad665d001307a5a85c5cca5d\": container with ID starting with 87a5880a56857da473302a0d3777578adca1dd8dad665d001307a5a85c5cca5d not found: ID does not exist"
Nov 03 01:55:23.647687 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.647603    5508 scope.go:122] "RemoveContainer" containerID="a01305753f04c3659fbc7b61cb41d83e4562d8398459bfd60cb38f2fe02b536f"
Nov 03 01:55:23.788518 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.788437    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"]
Nov 03 01:55:23.834599 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.834570    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-71746a-65fbbf887f-jvs9p"]
Nov 03 01:55:23.894597 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.894571    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"]
Nov 03 01:55:23.900179 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:23.900158    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-71746a-7bbb4f8b7b-n4z25"]
Nov 03 01:55:24.693599 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:24.693566    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67f1cc6-9db6-4464-9f0e-1392b5be85cd" path="/var/lib/kubelet/pods/a67f1cc6-9db6-4464-9f0e-1392b5be85cd/volumes"
Nov 03 01:55:24.694110 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:24.694093    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b4fd3a-9cf2-489f-89cd-59b57cddfc06" path="/var/lib/kubelet/pods/c3b4fd3a-9cf2-489f-89cd-59b57cddfc06/volumes"
Nov 03 01:55:27.626661 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:27.626586    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w" podUID="98db450f-aa09-4dc2-9166-2a9d1e8f76ad" containerName="payload-processing" containerID="cri-o://e4a8a46010fc8857e69f67c048783117caf90dc96d84216d4ebf44efb6a55fba" gracePeriod=25
Nov 03 01:55:27.891369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:27.891300    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"
Nov 03 01:55:28.026000 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.025964    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-plugins-config-volume\") pod \"98db450f-aa09-4dc2-9166-2a9d1e8f76ad\" (UID: \"98db450f-aa09-4dc2-9166-2a9d1e8f76ad\") "
Nov 03 01:55:28.026169 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.026119    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-kube-api-access-r9c7h\" (UniqueName: \"kubernetes.io/projected/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-kube-api-access-r9c7h\") pod \"98db450f-aa09-4dc2-9166-2a9d1e8f76ad\" (UID: \"98db450f-aa09-4dc2-9166-2a9d1e8f76ad\") "
Nov 03 01:55:28.026474 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.026432    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-plugins-config-volume" pod "98db450f-aa09-4dc2-9166-2a9d1e8f76ad" (UID: "98db450f-aa09-4dc2-9166-2a9d1e8f76ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:28.026647 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.026623    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:28.028189 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.028160    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-kube-api-access-r9c7h" pod "98db450f-aa09-4dc2-9166-2a9d1e8f76ad" (UID: "98db450f-aa09-4dc2-9166-2a9d1e8f76ad"). InnerVolumeSpecName "kube-api-access-r9c7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:28.127753 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.127719    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9c7h\" (UniqueName: \"kubernetes.io/projected/98db450f-aa09-4dc2-9166-2a9d1e8f76ad-kube-api-access-r9c7h\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:28.581563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.581531    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"]
Nov 03 01:55:28.581776 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.581750    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm" podUID="8474ff81-60d1-4464-bd05-e24fd663f34f" containerName="maas-api" containerID="cri-o://0ce64158fdfa193acd26660a1f4b26024e6e953ceb5a93f345873075826fe630" gracePeriod=30
Nov 03 01:55:28.676159 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.676124    5508 generic.go:356] "Generic (PLEG): container finished" podID="98db450f-aa09-4dc2-9166-2a9d1e8f76ad" containerID="e4a8a46010fc8857e69f67c048783117caf90dc96d84216d4ebf44efb6a55fba" exitCode=0
Nov 03 01:55:28.676633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.676318    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w" event={"ID":"98db450f-aa09-4dc2-9166-2a9d1e8f76ad","Type":"ContainerDied","Data":"e4a8a46010fc8857e69f67c048783117caf90dc96d84216d4ebf44efb6a55fba"}
Nov 03 01:55:28.676633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.676381    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w" event={"ID":"98db450f-aa09-4dc2-9166-2a9d1e8f76ad","Type":"ContainerDied","Data":"ccbd458c51bc2a8689f8d1dfce6a1c01f6fec821592e1fc8ef943cc93b81ce75"}
Nov 03 01:55:28.676633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.676405    5508 scope.go:122] "RemoveContainer" containerID="e4a8a46010fc8857e69f67c048783117caf90dc96d84216d4ebf44efb6a55fba"
Nov 03 01:55:28.676633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.676531    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"
Nov 03 01:55:28.700652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.700630    5508 scope.go:122] "RemoveContainer" containerID="e4a8a46010fc8857e69f67c048783117caf90dc96d84216d4ebf44efb6a55fba"
Nov 03 01:55:28.700921 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:28.700902    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a8a46010fc8857e69f67c048783117caf90dc96d84216d4ebf44efb6a55fba\": container with ID starting with e4a8a46010fc8857e69f67c048783117caf90dc96d84216d4ebf44efb6a55fba not found: ID does not exist" containerID="e4a8a46010fc8857e69f67c048783117caf90dc96d84216d4ebf44efb6a55fba"
Nov 03 01:55:28.701006 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.700931    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a8a46010fc8857e69f67c048783117caf90dc96d84216d4ebf44efb6a55fba"} err="failed to get container status \"e4a8a46010fc8857e69f67c048783117caf90dc96d84216d4ebf44efb6a55fba\": rpc error: code = NotFound desc = could not find container \"e4a8a46010fc8857e69f67c048783117caf90dc96d84216d4ebf44efb6a55fba\": container with ID starting with e4a8a46010fc8857e69f67c048783117caf90dc96d84216d4ebf44efb6a55fba not found: ID does not exist"
Nov 03 01:55:28.852793 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.852716    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"]
Nov 03 01:55:28.853012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.852997    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0e0d61-bdc476bfc-2dj4w"]
Nov 03 01:55:28.886586 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:28.886562    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:55:29.049567 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.049537    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-metrics-certs\") pod \"8474ff81-60d1-4464-bd05-e24fd663f34f\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") "
Nov 03 01:55:29.049770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.049707    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8474ff81-60d1-4464-bd05-e24fd663f34f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8474ff81-60d1-4464-bd05-e24fd663f34f-openshift-service-ca\") pod \"8474ff81-60d1-4464-bd05-e24fd663f34f\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") "
Nov 03 01:55:29.049770 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.049753    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-maas-api-tls\") pod \"8474ff81-60d1-4464-bd05-e24fd663f34f\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") "
Nov 03 01:55:29.050211 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.049958    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8474ff81-60d1-4464-bd05-e24fd663f34f-kube-api-access-k9nr4\" (UniqueName: \"kubernetes.io/projected/8474ff81-60d1-4464-bd05-e24fd663f34f-kube-api-access-k9nr4\") pod \"8474ff81-60d1-4464-bd05-e24fd663f34f\" (UID: \"8474ff81-60d1-4464-bd05-e24fd663f34f\") "
Nov 03 01:55:29.050211 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.050087    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8474ff81-60d1-4464-bd05-e24fd663f34f-openshift-service-ca" pod "8474ff81-60d1-4464-bd05-e24fd663f34f" (UID: "8474ff81-60d1-4464-bd05-e24fd663f34f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:29.050562 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.050541    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8474ff81-60d1-4464-bd05-e24fd663f34f-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:29.052025 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.051996    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8474ff81-60d1-4464-bd05-e24fd663f34f-kube-api-access-k9nr4" pod "8474ff81-60d1-4464-bd05-e24fd663f34f" (UID: "8474ff81-60d1-4464-bd05-e24fd663f34f"). InnerVolumeSpecName "kube-api-access-k9nr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:29.052508 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.052485    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-maas-api-tls" pod "8474ff81-60d1-4464-bd05-e24fd663f34f" (UID: "8474ff81-60d1-4464-bd05-e24fd663f34f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:55:29.052605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.052589    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-metrics-certs" pod "8474ff81-60d1-4464-bd05-e24fd663f34f" (UID: "8474ff81-60d1-4464-bd05-e24fd663f34f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:55:29.056428 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.056406    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"]
Nov 03 01:55:29.056616 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.056596    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw" podUID="b54ac1ac-b825-4335-9cff-bbd34a00ca89" containerName="payload-processing" containerID="cri-o://c74b599f17cb7a5986a3108242092436761aaa837a0f1a300ce34c5bf2413078" gracePeriod=30
Nov 03 01:55:29.108613 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.108555    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"]
Nov 03 01:55:29.108779 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.108747    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq" podUID="29ba26de-2eae-4bf1-a8bb-928fe164bf67" containerName="payload-pre-processing" containerID="cri-o://61eb9278e16f6faca31d2236cd1d4bb1b9005a85083448eba9c2f99c3658023e" gracePeriod=30
Nov 03 01:55:29.151729 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.151704    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9nr4\" (UniqueName: \"kubernetes.io/projected/8474ff81-60d1-4464-bd05-e24fd663f34f-kube-api-access-k9nr4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:29.151809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.151732    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:29.151809 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.151746    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8474ff81-60d1-4464-bd05-e24fd663f34f-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:29.364073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.363996    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"
Nov 03 01:55:29.397055 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.397027    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"
Nov 03 01:55:29.455287 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.455256    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b54ac1ac-b825-4335-9cff-bbd34a00ca89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54ac1ac-b825-4335-9cff-bbd34a00ca89-plugins-config-volume\") pod \"b54ac1ac-b825-4335-9cff-bbd34a00ca89\" (UID: \"b54ac1ac-b825-4335-9cff-bbd34a00ca89\") "
Nov 03 01:55:29.455488 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.455319    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b54ac1ac-b825-4335-9cff-bbd34a00ca89-kube-api-access-r7rdn\" (UniqueName: \"kubernetes.io/projected/b54ac1ac-b825-4335-9cff-bbd34a00ca89-kube-api-access-r7rdn\") pod \"b54ac1ac-b825-4335-9cff-bbd34a00ca89\" (UID: \"b54ac1ac-b825-4335-9cff-bbd34a00ca89\") "
Nov 03 01:55:29.455695 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.455668    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54ac1ac-b825-4335-9cff-bbd34a00ca89-plugins-config-volume" pod "b54ac1ac-b825-4335-9cff-bbd34a00ca89" (UID: "b54ac1ac-b825-4335-9cff-bbd34a00ca89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:29.457727 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.457704    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54ac1ac-b825-4335-9cff-bbd34a00ca89-kube-api-access-r7rdn" pod "b54ac1ac-b825-4335-9cff-bbd34a00ca89" (UID: "b54ac1ac-b825-4335-9cff-bbd34a00ca89"). InnerVolumeSpecName "kube-api-access-r7rdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:29.556247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.556216    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ba26de-2eae-4bf1-a8bb-928fe164bf67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ba26de-2eae-4bf1-a8bb-928fe164bf67-plugins-config-volume\") pod \"29ba26de-2eae-4bf1-a8bb-928fe164bf67\" (UID: \"29ba26de-2eae-4bf1-a8bb-928fe164bf67\") "
Nov 03 01:55:29.556247 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.556256    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ba26de-2eae-4bf1-a8bb-928fe164bf67-kube-api-access-8pfdf\" (UniqueName: \"kubernetes.io/projected/29ba26de-2eae-4bf1-a8bb-928fe164bf67-kube-api-access-8pfdf\") pod \"29ba26de-2eae-4bf1-a8bb-928fe164bf67\" (UID: \"29ba26de-2eae-4bf1-a8bb-928fe164bf67\") "
Nov 03 01:55:29.556551 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.556537    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54ac1ac-b825-4335-9cff-bbd34a00ca89-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:29.556615 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.556552    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7rdn\" (UniqueName: \"kubernetes.io/projected/b54ac1ac-b825-4335-9cff-bbd34a00ca89-kube-api-access-r7rdn\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:29.556683 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.556658    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ba26de-2eae-4bf1-a8bb-928fe164bf67-plugins-config-volume" pod "29ba26de-2eae-4bf1-a8bb-928fe164bf67" (UID: "29ba26de-2eae-4bf1-a8bb-928fe164bf67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:29.558357 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.558319    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ba26de-2eae-4bf1-a8bb-928fe164bf67-kube-api-access-8pfdf" pod "29ba26de-2eae-4bf1-a8bb-928fe164bf67" (UID: "29ba26de-2eae-4bf1-a8bb-928fe164bf67"). InnerVolumeSpecName "kube-api-access-8pfdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:29.657600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.657564    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29ba26de-2eae-4bf1-a8bb-928fe164bf67-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:29.657600 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.657593    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pfdf\" (UniqueName: \"kubernetes.io/projected/29ba26de-2eae-4bf1-a8bb-928fe164bf67-kube-api-access-8pfdf\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:29.710371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.710010    5508 generic.go:356] "Generic (PLEG): container finished" podID="8474ff81-60d1-4464-bd05-e24fd663f34f" containerID="0ce64158fdfa193acd26660a1f4b26024e6e953ceb5a93f345873075826fe630" exitCode=0
Nov 03 01:55:29.710371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.710132    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm" event={"ID":"8474ff81-60d1-4464-bd05-e24fd663f34f","Type":"ContainerDied","Data":"0ce64158fdfa193acd26660a1f4b26024e6e953ceb5a93f345873075826fe630"}
Nov 03 01:55:29.710371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.710193    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm" event={"ID":"8474ff81-60d1-4464-bd05-e24fd663f34f","Type":"ContainerDied","Data":"38787ccedcc22556792aa23c84ce08f3d999709ca4c4f599fedc3d59fcfdc5e0"}
Nov 03 01:55:29.710371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.710225    5508 scope.go:122] "RemoveContainer" containerID="0ce64158fdfa193acd26660a1f4b26024e6e953ceb5a93f345873075826fe630"
Nov 03 01:55:29.711026 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.710402    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"
Nov 03 01:55:29.740825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.738172    5508 generic.go:356] "Generic (PLEG): container finished" podID="29ba26de-2eae-4bf1-a8bb-928fe164bf67" containerID="61eb9278e16f6faca31d2236cd1d4bb1b9005a85083448eba9c2f99c3658023e" exitCode=0
Nov 03 01:55:29.740825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.738433    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq" event={"ID":"29ba26de-2eae-4bf1-a8bb-928fe164bf67","Type":"ContainerDied","Data":"61eb9278e16f6faca31d2236cd1d4bb1b9005a85083448eba9c2f99c3658023e"}
Nov 03 01:55:29.740825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.738512    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq" event={"ID":"29ba26de-2eae-4bf1-a8bb-928fe164bf67","Type":"ContainerDied","Data":"ee2e56e8d9658def264b3baa89caedce82a59da3c726d49f170d78bbce343010"}
Nov 03 01:55:29.740825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.738617    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"
Nov 03 01:55:29.743527 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.741209    5508 generic.go:356] "Generic (PLEG): container finished" podID="b54ac1ac-b825-4335-9cff-bbd34a00ca89" containerID="c74b599f17cb7a5986a3108242092436761aaa837a0f1a300ce34c5bf2413078" exitCode=0
Nov 03 01:55:29.743527 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.741376    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw" event={"ID":"b54ac1ac-b825-4335-9cff-bbd34a00ca89","Type":"ContainerDied","Data":"c74b599f17cb7a5986a3108242092436761aaa837a0f1a300ce34c5bf2413078"}
Nov 03 01:55:29.743527 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.741427    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw" event={"ID":"b54ac1ac-b825-4335-9cff-bbd34a00ca89","Type":"ContainerDied","Data":"5e2a90dac47977d12db501968dfebe7b7102fe82021abacb1da8503b03ada519"}
Nov 03 01:55:29.743527 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.741507    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"
Nov 03 01:55:29.758726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.755947    5508 scope.go:122] "RemoveContainer" containerID="0ce64158fdfa193acd26660a1f4b26024e6e953ceb5a93f345873075826fe630"
Nov 03 01:55:29.758726 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:29.756482    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce64158fdfa193acd26660a1f4b26024e6e953ceb5a93f345873075826fe630\": container with ID starting with 0ce64158fdfa193acd26660a1f4b26024e6e953ceb5a93f345873075826fe630 not found: ID does not exist" containerID="0ce64158fdfa193acd26660a1f4b26024e6e953ceb5a93f345873075826fe630"
Nov 03 01:55:29.758726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.756516    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce64158fdfa193acd26660a1f4b26024e6e953ceb5a93f345873075826fe630"} err="failed to get container status \"0ce64158fdfa193acd26660a1f4b26024e6e953ceb5a93f345873075826fe630\": rpc error: code = NotFound desc = could not find container \"0ce64158fdfa193acd26660a1f4b26024e6e953ceb5a93f345873075826fe630\": container with ID starting with 0ce64158fdfa193acd26660a1f4b26024e6e953ceb5a93f345873075826fe630 not found: ID does not exist"
Nov 03 01:55:29.758726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.756539    5508 scope.go:122] "RemoveContainer" containerID="61eb9278e16f6faca31d2236cd1d4bb1b9005a85083448eba9c2f99c3658023e"
Nov 03 01:55:29.771519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.771498    5508 scope.go:122] "RemoveContainer" containerID="61eb9278e16f6faca31d2236cd1d4bb1b9005a85083448eba9c2f99c3658023e"
Nov 03 01:55:29.771786 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:29.771769    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61eb9278e16f6faca31d2236cd1d4bb1b9005a85083448eba9c2f99c3658023e\": container with ID starting with 61eb9278e16f6faca31d2236cd1d4bb1b9005a85083448eba9c2f99c3658023e not found: ID does not exist" containerID="61eb9278e16f6faca31d2236cd1d4bb1b9005a85083448eba9c2f99c3658023e"
Nov 03 01:55:29.771866 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.771793    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61eb9278e16f6faca31d2236cd1d4bb1b9005a85083448eba9c2f99c3658023e"} err="failed to get container status \"61eb9278e16f6faca31d2236cd1d4bb1b9005a85083448eba9c2f99c3658023e\": rpc error: code = NotFound desc = could not find container \"61eb9278e16f6faca31d2236cd1d4bb1b9005a85083448eba9c2f99c3658023e\": container with ID starting with 61eb9278e16f6faca31d2236cd1d4bb1b9005a85083448eba9c2f99c3658023e not found: ID does not exist"
Nov 03 01:55:29.771866 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.771810    5508 scope.go:122] "RemoveContainer" containerID="c74b599f17cb7a5986a3108242092436761aaa837a0f1a300ce34c5bf2413078"
Nov 03 01:55:29.780564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.780539    5508 scope.go:122] "RemoveContainer" containerID="c74b599f17cb7a5986a3108242092436761aaa837a0f1a300ce34c5bf2413078"
Nov 03 01:55:29.780861 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:29.780837    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74b599f17cb7a5986a3108242092436761aaa837a0f1a300ce34c5bf2413078\": container with ID starting with c74b599f17cb7a5986a3108242092436761aaa837a0f1a300ce34c5bf2413078 not found: ID does not exist" containerID="c74b599f17cb7a5986a3108242092436761aaa837a0f1a300ce34c5bf2413078"
Nov 03 01:55:29.780962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.780870    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74b599f17cb7a5986a3108242092436761aaa837a0f1a300ce34c5bf2413078"} err="failed to get container status \"c74b599f17cb7a5986a3108242092436761aaa837a0f1a300ce34c5bf2413078\": rpc error: code = NotFound desc = could not find container \"c74b599f17cb7a5986a3108242092436761aaa837a0f1a300ce34c5bf2413078\": container with ID starting with c74b599f17cb7a5986a3108242092436761aaa837a0f1a300ce34c5bf2413078 not found: ID does not exist"
Nov 03 01:55:29.869223 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.869087    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"]
Nov 03 01:55:29.874912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.874889    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36c2c1-65d55894b6-s4tpm"]
Nov 03 01:55:29.978052 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.977936    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"]
Nov 03 01:55:29.981012 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:29.980989    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36c2c1-b65bcf9c5-cltlq"]
Nov 03 01:55:30.083132 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.083097    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"]
Nov 03 01:55:30.087858 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.087829    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36c2c1-5686bfc6cd-66zhw"]
Nov 03 01:55:30.294581 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.294493    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"]
Nov 03 01:55:30.294986 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.294970    5508 cpu_manager.go:414] "containerMap: removing container" podUID="29ba26de-2eae-4bf1-a8bb-928fe164bf67" containerName="payload-pre-processing"
Nov 03 01:55:30.295050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.294986    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ba26de-2eae-4bf1-a8bb-928fe164bf67" containerName="payload-pre-processing"
Nov 03 01:55:30.295050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295001    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5e472801-3966-49f6-979d-3c2943800101" containerName="maas-api"
Nov 03 01:55:30.295050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295006    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e472801-3966-49f6-979d-3c2943800101" containerName="maas-api"
Nov 03 01:55:30.295050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295019    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a67f1cc6-9db6-4464-9f0e-1392b5be85cd" containerName="payload-processing"
Nov 03 01:55:30.295050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295024    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67f1cc6-9db6-4464-9f0e-1392b5be85cd" containerName="payload-processing"
Nov 03 01:55:30.295050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295032    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c3b4fd3a-9cf2-489f-89cd-59b57cddfc06" containerName="payload-pre-processing"
Nov 03 01:55:30.295050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295037    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b4fd3a-9cf2-489f-89cd-59b57cddfc06" containerName="payload-pre-processing"
Nov 03 01:55:30.295050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295044    5508 cpu_manager.go:414] "containerMap: removing container" podUID="63a58d16-d2ab-46bb-bb83-9339e9f1e0b4" containerName="payload-processing"
Nov 03 01:55:30.295050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295049    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a58d16-d2ab-46bb-bb83-9339e9f1e0b4" containerName="payload-processing"
Nov 03 01:55:30.295050 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295056    5508 cpu_manager.go:414] "containerMap: removing container" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295061    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295074    5508 cpu_manager.go:414] "containerMap: removing container" podUID="11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96" containerName="payload-pre-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295078    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96" containerName="payload-pre-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295085    5508 cpu_manager.go:414] "containerMap: removing container" podUID="636dd427-8666-46d8-a788-2915dc378d5f" containerName="payload-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295089    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636dd427-8666-46d8-a788-2915dc378d5f" containerName="payload-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295098    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8474ff81-60d1-4464-bd05-e24fd663f34f" containerName="maas-api"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295102    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8474ff81-60d1-4464-bd05-e24fd663f34f" containerName="maas-api"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295110    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" containerName="istio-proxy"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295114    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" containerName="istio-proxy"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295123    5508 cpu_manager.go:414] "containerMap: removing container" podUID="98db450f-aa09-4dc2-9166-2a9d1e8f76ad" containerName="payload-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295127    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98db450f-aa09-4dc2-9166-2a9d1e8f76ad" containerName="payload-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295134    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b54ac1ac-b825-4335-9cff-bbd34a00ca89" containerName="payload-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295138    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b54ac1ac-b825-4335-9cff-bbd34a00ca89" containerName="payload-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295149    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b09aad0d-ebb8-4fec-898f-a1e47d1dfd95" containerName="payload-pre-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295154    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b09aad0d-ebb8-4fec-898f-a1e47d1dfd95" containerName="payload-pre-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295219    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b09aad0d-ebb8-4fec-898f-a1e47d1dfd95" containerName="payload-pre-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295230    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ba26de-2eae-4bf1-a8bb-928fe164bf67" containerName="payload-pre-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295236    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e472801-3966-49f6-979d-3c2943800101" containerName="maas-api"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295241    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67f1cc6-9db6-4464-9f0e-1392b5be85cd" containerName="payload-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295246    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b4fd3a-9cf2-489f-89cd-59b57cddfc06" containerName="payload-pre-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295252    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a58d16-d2ab-46bb-bb83-9339e9f1e0b4" containerName="payload-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295257    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295263    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c8bbe6-bb8a-4096-a4f5-cc7d4e712a96" containerName="payload-pre-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295269    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="636dd427-8666-46d8-a788-2915dc378d5f" containerName="payload-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295275    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8474ff81-60d1-4464-bd05-e24fd663f34f" containerName="maas-api"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295279    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3fb1ba8-ef8c-4ef2-a78d-2f01d9ea0857" containerName="istio-proxy"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295284    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="98db450f-aa09-4dc2-9166-2a9d1e8f76ad" containerName="payload-processing"
Nov 03 01:55:30.295592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.295288    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b54ac1ac-b825-4335-9cff-bbd34a00ca89" containerName="payload-processing"
Nov 03 01:55:30.298501 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.298483    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:30.350069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.350025    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-5d9025-serving-cert\""
Nov 03 01:55:30.354946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.354923    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"]
Nov 03 01:55:30.465914 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.465885    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:30.466106 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.465933    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt8q4\" (UniqueName: \"kubernetes.io/projected/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-kube-api-access-jt8q4\") pod \"maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:30.466177 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.466129    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-metrics-certs\") pod \"maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:30.466268 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.466245    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:30.567648 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.567530    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:30.568835 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.568812    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:30.569025 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.569009    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt8q4\" (UniqueName: \"kubernetes.io/projected/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-kube-api-access-jt8q4\") pod \"maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:30.569293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.569274    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-metrics-certs\") pod \"maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:30.570038 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.568730    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:30.579888 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.579858    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-metrics-certs\") pod \"maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:30.580403 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.580371    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:30.627617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.627595    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt8q4\" (UniqueName: \"kubernetes.io/projected/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-kube-api-access-jt8q4\") pod \"maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:30.693893 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.693852    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ba26de-2eae-4bf1-a8bb-928fe164bf67" path="/var/lib/kubelet/pods/29ba26de-2eae-4bf1-a8bb-928fe164bf67/volumes"
Nov 03 01:55:30.694478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.694458    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8474ff81-60d1-4464-bd05-e24fd663f34f" path="/var/lib/kubelet/pods/8474ff81-60d1-4464-bd05-e24fd663f34f/volumes"
Nov 03 01:55:30.694957 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.694940    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98db450f-aa09-4dc2-9166-2a9d1e8f76ad" path="/var/lib/kubelet/pods/98db450f-aa09-4dc2-9166-2a9d1e8f76ad/volumes"
Nov 03 01:55:30.695872 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.695850    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b54ac1ac-b825-4335-9cff-bbd34a00ca89" path="/var/lib/kubelet/pods/b54ac1ac-b825-4335-9cff-bbd34a00ca89/volumes"
Nov 03 01:55:30.913276 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:30.913244    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:31.294876 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:31.294794    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"]
Nov 03 01:55:31.300339 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:55:31.300313    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624b2b40_ab2d_4d9a_8899_b65b04b6d59c.slice/crio-618d839d13518c9416dbada4aa02434763f0cc478acfcf4f62566d39668ee1f8 WatchSource:0}: Error finding container 618d839d13518c9416dbada4aa02434763f0cc478acfcf4f62566d39668ee1f8: Status 404 returned error can't find the container with id 618d839d13518c9416dbada4aa02434763f0cc478acfcf4f62566d39668ee1f8
Nov 03 01:55:31.783000 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:31.782967    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg" event={"ID":"624b2b40-ab2d-4d9a-8899-b65b04b6d59c","Type":"ContainerStarted","Data":"618d839d13518c9416dbada4aa02434763f0cc478acfcf4f62566d39668ee1f8"}
Nov 03 01:55:32.797388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:32.797274    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg" event={"ID":"624b2b40-ab2d-4d9a-8899-b65b04b6d59c","Type":"ContainerStarted","Data":"07500cf82d484f8d20f5e5cdbe663f74d52a0d66263d2c9324bf2d655925492f"}
Nov 03 01:55:32.797388 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:32.797368    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:32.911264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:32.911212    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg" podStartSLOduration=1.684304008 podStartE2EDuration="2.911197932s" totalImagesPullingTime="1.226893924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:32.90864304 +0000 UTC m=+2212.334249524" watchObservedRunningTime="2026-11-03 01:55:32.911197932 +0000 UTC m=+2212.336804396"
Nov 03 01:55:35.486187 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.486154    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"]
Nov 03 01:55:35.486946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.486924    5508 cpu_manager.go:414] "containerMap: removing container" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy"
Nov 03 01:55:35.487021 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.486949    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy"
Nov 03 01:55:35.487094 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.487082    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="774d3ce9-baeb-4de2-837b-62b8a559b7b3" containerName="istio-proxy"
Nov 03 01:55:35.491326 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.491299    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.540084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.540057    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-706bdd-openshift-default-dockercfg-nx5vr\""
Nov 03 01:55:35.595401 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.590762    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"]
Nov 03 01:55:35.641214 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.641186    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-token\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.641377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.641233    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-credential-socket\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.641377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.641291    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-socket\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.641377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.641313    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-envoy\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.641377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.641343    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istiod-ca-cert\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.641539 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.641453    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-data\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.641539 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.641489    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-podinfo\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.641605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.641543    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-certs\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.641652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.641637    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpshn\" (UniqueName: \"kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-kube-api-access-jpshn\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.742716 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.742643    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpshn\" (UniqueName: \"kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-kube-api-access-jpshn\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.742938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.742742    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-token\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.742938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.742785    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-credential-socket\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.742938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.742816    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-socket\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.742938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.742847    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-envoy\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.742938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.742904    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istiod-ca-cert\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.742938 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.742927    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-data\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.743289 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.742949    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-podinfo\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.743289 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.743079    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-certs\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.743431 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.743288    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-socket\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.743517 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.743494    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-certs\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.743700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.743557    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-credential-socket\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.743700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.743622    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-data\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.743863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.743741    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istiod-ca-cert\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.745617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.745592    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-envoy\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.746094 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.746072    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-podinfo\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.800032 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.800011    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpshn\" (UniqueName: \"kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-kube-api-access-jpshn\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.800333 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.800315    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-token\") pod \"e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") " pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:35.805286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:35.805269    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:36.242267 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:36.242226    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"]
Nov 03 01:55:36.253537 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:55:36.253509    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9762f99e_d6cd_4369_bc2b_e11dbf56a32f.slice/crio-22e56758e90ad8beb198051cd5619a83c574173fccc95d72cafe7dd67ab6dd64 WatchSource:0}: Error finding container 22e56758e90ad8beb198051cd5619a83c574173fccc95d72cafe7dd67ab6dd64: Status 404 returned error can't find the container with id 22e56758e90ad8beb198051cd5619a83c574173fccc95d72cafe7dd67ab6dd64
Nov 03 01:55:36.256405 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:36.256375    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:55:36.256476 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:36.256455    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:55:36.256522 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:36.256483    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:55:36.825174 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:36.825140    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5" event={"ID":"9762f99e-d6cd-4369-bc2b-e11dbf56a32f","Type":"ContainerStarted","Data":"c5d098335a95cbc91d7edfe1f95d405e5255285c0b215761616242e1d0ae5c39"}
Nov 03 01:55:36.825631 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:36.825192    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5" event={"ID":"9762f99e-d6cd-4369-bc2b-e11dbf56a32f","Type":"ContainerStarted","Data":"22e56758e90ad8beb198051cd5619a83c574173fccc95d72cafe7dd67ab6dd64"}
Nov 03 01:55:36.939546 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:36.939491    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5" podStartSLOduration=1.939476656 podStartE2EDuration="1.939476656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:36.937969614 +0000 UTC m=+2216.363576078" watchObservedRunningTime="2026-11-03 01:55:36.939476656 +0000 UTC m=+2216.365083147"
Nov 03 01:55:37.458246 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:37.458214    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"]
Nov 03 01:55:37.458962 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:37.458926    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp" podUID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerName="istio-proxy" containerID="cri-o://fab8639e2fce1d37fff92cf0567b0e8802aaed56f9f8a71e1f705b86c23c14a1" gracePeriod=30
Nov 03 01:55:37.805743 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:37.805668    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:37.810929 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:37.810896    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:37.832206 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:37.832184    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:37.832883 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:37.832867    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:55:38.809717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:38.809687    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:55:42.725754 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.725691    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:55:42.853575 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.853534    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-token\" (UniqueName: \"kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-token\") pod \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") "
Nov 03 01:55:42.853766 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.853636    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-credential-socket\") pod \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") "
Nov 03 01:55:42.853766 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.853689    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-socket\") pod \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") "
Nov 03 01:55:42.853766 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.853716    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-kube-api-access-4hmbn\" (UniqueName: \"kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-kube-api-access-4hmbn\") pod \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") "
Nov 03 01:55:42.853948 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.853784    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-data\") pod \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") "
Nov 03 01:55:42.853948 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.853833    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-envoy\") pod \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") "
Nov 03 01:55:42.854055 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.853982    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-socket" pod "a37e873e-8f3e-4e1b-a409-ca3fe4192728" (UID: "a37e873e-8f3e-4e1b-a409-ca3fe4192728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:42.854114 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.854092    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-credential-socket" pod "a37e873e-8f3e-4e1b-a409-ca3fe4192728" (UID: "a37e873e-8f3e-4e1b-a409-ca3fe4192728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:42.854240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.854212    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-podinfo\") pod \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") "
Nov 03 01:55:42.854240 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.854229    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-data" pod "a37e873e-8f3e-4e1b-a409-ca3fe4192728" (UID: "a37e873e-8f3e-4e1b-a409-ca3fe4192728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:42.854366 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.854294    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-certs\") pod \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") "
Nov 03 01:55:42.854366 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.854327    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istiod-ca-cert\") pod \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\" (UID: \"a37e873e-8f3e-4e1b-a409-ca3fe4192728\") "
Nov 03 01:55:42.854788 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.854731    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-certs" pod "a37e873e-8f3e-4e1b-a409-ca3fe4192728" (UID: "a37e873e-8f3e-4e1b-a409-ca3fe4192728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:42.854925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.854853    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:42.854925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.854875    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:42.854925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.854891    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:42.854925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.854907    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:42.854925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.854905    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istiod-ca-cert" pod "a37e873e-8f3e-4e1b-a409-ca3fe4192728" (UID: "a37e873e-8f3e-4e1b-a409-ca3fe4192728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:42.856177 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.856150    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-token" pod "a37e873e-8f3e-4e1b-a409-ca3fe4192728" (UID: "a37e873e-8f3e-4e1b-a409-ca3fe4192728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:42.856319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.856256    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-envoy" pod "a37e873e-8f3e-4e1b-a409-ca3fe4192728" (UID: "a37e873e-8f3e-4e1b-a409-ca3fe4192728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:55:42.856476 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.856453    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-kube-api-access-4hmbn" pod "a37e873e-8f3e-4e1b-a409-ca3fe4192728" (UID: "a37e873e-8f3e-4e1b-a409-ca3fe4192728"). InnerVolumeSpecName "kube-api-access-4hmbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:42.856737 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.856717    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-podinfo" pod "a37e873e-8f3e-4e1b-a409-ca3fe4192728" (UID: "a37e873e-8f3e-4e1b-a409-ca3fe4192728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:55:42.867716 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.867689    5508 generic.go:356] "Generic (PLEG): container finished" podID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerID="fab8639e2fce1d37fff92cf0567b0e8802aaed56f9f8a71e1f705b86c23c14a1" exitCode=0
Nov 03 01:55:42.867821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.867759    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"
Nov 03 01:55:42.867874 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.867759    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp" event={"ID":"a37e873e-8f3e-4e1b-a409-ca3fe4192728","Type":"ContainerDied","Data":"fab8639e2fce1d37fff92cf0567b0e8802aaed56f9f8a71e1f705b86c23c14a1"}
Nov 03 01:55:42.867928 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.867901    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp" event={"ID":"a37e873e-8f3e-4e1b-a409-ca3fe4192728","Type":"ContainerDied","Data":"a967c5609eafbd680e87b7e8faa14d389d95f2a2227ddedbaa580c4007185d4a"}
Nov 03 01:55:42.867972 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.867945    5508 scope.go:122] "RemoveContainer" containerID="fab8639e2fce1d37fff92cf0567b0e8802aaed56f9f8a71e1f705b86c23c14a1"
Nov 03 01:55:42.884862 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.884844    5508 scope.go:122] "RemoveContainer" containerID="14b47e8eb3d0fd8d995c16f1a6bf4e130df23edd792531c29e8639db07f07dd8"
Nov 03 01:55:42.897598 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.897582    5508 scope.go:122] "RemoveContainer" containerID="fab8639e2fce1d37fff92cf0567b0e8802aaed56f9f8a71e1f705b86c23c14a1"
Nov 03 01:55:42.897940 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:42.897910    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab8639e2fce1d37fff92cf0567b0e8802aaed56f9f8a71e1f705b86c23c14a1\": container with ID starting with fab8639e2fce1d37fff92cf0567b0e8802aaed56f9f8a71e1f705b86c23c14a1 not found: ID does not exist" containerID="fab8639e2fce1d37fff92cf0567b0e8802aaed56f9f8a71e1f705b86c23c14a1"
Nov 03 01:55:42.898044 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.897950    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab8639e2fce1d37fff92cf0567b0e8802aaed56f9f8a71e1f705b86c23c14a1"} err="failed to get container status \"fab8639e2fce1d37fff92cf0567b0e8802aaed56f9f8a71e1f705b86c23c14a1\": rpc error: code = NotFound desc = could not find container \"fab8639e2fce1d37fff92cf0567b0e8802aaed56f9f8a71e1f705b86c23c14a1\": container with ID starting with fab8639e2fce1d37fff92cf0567b0e8802aaed56f9f8a71e1f705b86c23c14a1 not found: ID does not exist"
Nov 03 01:55:42.898044 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.897973    5508 scope.go:122] "RemoveContainer" containerID="14b47e8eb3d0fd8d995c16f1a6bf4e130df23edd792531c29e8639db07f07dd8"
Nov 03 01:55:42.898247 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:42.898217    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14b47e8eb3d0fd8d995c16f1a6bf4e130df23edd792531c29e8639db07f07dd8\": container with ID starting with 14b47e8eb3d0fd8d995c16f1a6bf4e130df23edd792531c29e8639db07f07dd8 not found: ID does not exist" containerID="14b47e8eb3d0fd8d995c16f1a6bf4e130df23edd792531c29e8639db07f07dd8"
Nov 03 01:55:42.898312 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.898243    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14b47e8eb3d0fd8d995c16f1a6bf4e130df23edd792531c29e8639db07f07dd8"} err="failed to get container status \"14b47e8eb3d0fd8d995c16f1a6bf4e130df23edd792531c29e8639db07f07dd8\": rpc error: code = NotFound desc = could not find container \"14b47e8eb3d0fd8d995c16f1a6bf4e130df23edd792531c29e8639db07f07dd8\": container with ID starting with 14b47e8eb3d0fd8d995c16f1a6bf4e130df23edd792531c29e8639db07f07dd8 not found: ID does not exist"
Nov 03 01:55:42.955870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.955834    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hmbn\" (UniqueName: \"kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-kube-api-access-4hmbn\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:42.955870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.955864    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:42.956052 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.955877    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:42.956052 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.955890    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:42.956052 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:42.955903    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a37e873e-8f3e-4e1b-a409-ca3fe4192728-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:43.023191 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.023054    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"]
Nov 03 01:55:43.034378 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.034325    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-36c2c1-openshift-default-5bf775bdcf-8c7dp"]
Nov 03 01:55:43.308037 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.307961    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"]
Nov 03 01:55:43.308684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.308661    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerName="istio-proxy"
Nov 03 01:55:43.308684 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.308683    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerName="istio-proxy"
Nov 03 01:55:43.308859 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.308699    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerName="istio-proxy"
Nov 03 01:55:43.308859 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.308707    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerName="istio-proxy"
Nov 03 01:55:43.308930 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.308871    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerName="istio-proxy"
Nov 03 01:55:43.313915 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.313894    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"
Nov 03 01:55:43.362855 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.362824    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-706bdd\""
Nov 03 01:55:43.363098 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.362898    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-706bdd-dockercfg-fbph6\""
Nov 03 01:55:43.364670 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.364642    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"]
Nov 03 01:55:43.366706 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.366682    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2psj7\" (UniqueName: \"kubernetes.io/projected/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-kube-api-access-2psj7\") pod \"payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7\" (UID: \"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"
Nov 03 01:55:43.366813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.366756    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7\" (UID: \"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"
Nov 03 01:55:43.475294 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.473160    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7\" (UID: \"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"
Nov 03 01:55:43.475294 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.473414    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2psj7\" (UniqueName: \"kubernetes.io/projected/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-kube-api-access-2psj7\") pod \"payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7\" (UID: \"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"
Nov 03 01:55:43.475294 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.474844    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7\" (UID: \"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"
Nov 03 01:55:43.479060 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.479023    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"]
Nov 03 01:55:43.480610 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.479921    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" containerName="istio-proxy"
Nov 03 01:55:43.485980 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.485720    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"
Nov 03 01:55:43.528446 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.528407    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"]
Nov 03 01:55:43.532398 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.532368    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2psj7\" (UniqueName: \"kubernetes.io/projected/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-kube-api-access-2psj7\") pod \"payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7\" (UID: \"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"
Nov 03 01:55:43.575102 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.575002    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed1fa2e-f822-4105-8b59-4da996a8ce65-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n\" (UID: \"9ed1fa2e-f822-4105-8b59-4da996a8ce65\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"
Nov 03 01:55:43.575281 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.575166    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7gpl\" (UniqueName: \"kubernetes.io/projected/9ed1fa2e-f822-4105-8b59-4da996a8ce65-kube-api-access-h7gpl\") pod \"payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n\" (UID: \"9ed1fa2e-f822-4105-8b59-4da996a8ce65\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"
Nov 03 01:55:43.631196 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.631104    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"
Nov 03 01:55:43.676567 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.676055    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed1fa2e-f822-4105-8b59-4da996a8ce65-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n\" (UID: \"9ed1fa2e-f822-4105-8b59-4da996a8ce65\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"
Nov 03 01:55:43.676567 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.676157    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7gpl\" (UniqueName: \"kubernetes.io/projected/9ed1fa2e-f822-4105-8b59-4da996a8ce65-kube-api-access-h7gpl\") pod \"payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n\" (UID: \"9ed1fa2e-f822-4105-8b59-4da996a8ce65\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"
Nov 03 01:55:43.676860 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.676807    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed1fa2e-f822-4105-8b59-4da996a8ce65-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n\" (UID: \"9ed1fa2e-f822-4105-8b59-4da996a8ce65\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"
Nov 03 01:55:43.734330 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.734265    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7gpl\" (UniqueName: \"kubernetes.io/projected/9ed1fa2e-f822-4105-8b59-4da996a8ce65-kube-api-access-h7gpl\") pod \"payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n\" (UID: \"9ed1fa2e-f822-4105-8b59-4da996a8ce65\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"
Nov 03 01:55:43.809646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:43.809377    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"
Nov 03 01:55:43.820772 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:43.820738    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:55:44.017283 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:44.017244    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"]
Nov 03 01:55:44.022614 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:55:44.022583    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ccb7d48_53d7_4fe6_aa77_c1675128fdc2.slice/crio-9d9564eaf8cecbf6148770fc40467929d9ce69f8b7798b09b5aeb7ebcff19cd7 WatchSource:0}: Error finding container 9d9564eaf8cecbf6148770fc40467929d9ce69f8b7798b09b5aeb7ebcff19cd7: Status 404 returned error can't find the container with id 9d9564eaf8cecbf6148770fc40467929d9ce69f8b7798b09b5aeb7ebcff19cd7
Nov 03 01:55:44.188815 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:44.188777    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"]
Nov 03 01:55:44.195490 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:55:44.195454    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed1fa2e_f822_4105_8b59_4da996a8ce65.slice/crio-d1688b7074d3a1f5f317bb227dcd8801ff43b74805ff9ff20a12f4c5ce762c0e WatchSource:0}: Error finding container d1688b7074d3a1f5f317bb227dcd8801ff43b74805ff9ff20a12f4c5ce762c0e: Status 404 returned error can't find the container with id d1688b7074d3a1f5f317bb227dcd8801ff43b74805ff9ff20a12f4c5ce762c0e
Nov 03 01:55:44.691862 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:44.691819    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37e873e-8f3e-4e1b-a409-ca3fe4192728" path="/var/lib/kubelet/pods/a37e873e-8f3e-4e1b-a409-ca3fe4192728/volumes"
Nov 03 01:55:44.890199 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:44.890154    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7" event={"ID":"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2","Type":"ContainerStarted","Data":"b8d24bc7c48ccd8819e3334e094936c865128c9acb8b35653e59481f76dbdb46"}
Nov 03 01:55:44.890682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:44.890212    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7" event={"ID":"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2","Type":"ContainerStarted","Data":"9d9564eaf8cecbf6148770fc40467929d9ce69f8b7798b09b5aeb7ebcff19cd7"}
Nov 03 01:55:44.890682 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:44.890411    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"
Nov 03 01:55:44.892429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:44.892403    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n" event={"ID":"9ed1fa2e-f822-4105-8b59-4da996a8ce65","Type":"ContainerStarted","Data":"51abda21b40d16006dca86992b1cc604c5d50cb0760eb57aae612489e9d013f5"}
Nov 03 01:55:44.892573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:44.892444    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n" event={"ID":"9ed1fa2e-f822-4105-8b59-4da996a8ce65","Type":"ContainerStarted","Data":"d1688b7074d3a1f5f317bb227dcd8801ff43b74805ff9ff20a12f4c5ce762c0e"}
Nov 03 01:55:44.892573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:44.892472    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"
Nov 03 01:55:44.998899 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:44.998783    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7" podStartSLOduration=1.998756475 podStartE2EDuration="1.998756475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:44.998080775 +0000 UTC m=+2224.423687249" watchObservedRunningTime="2026-11-03 01:55:44.998756475 +0000 UTC m=+2224.424362940"
Nov 03 01:55:45.107590 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:45.107520    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n" podStartSLOduration=2.107501508 podStartE2EDuration="2.107501508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:55:45.106256115 +0000 UTC m=+2224.531862579" watchObservedRunningTime="2026-11-03 01:55:45.107501508 +0000 UTC m=+2224.533107973"
Nov 03 01:55:47.912123 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:47.912091    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"]
Nov 03 01:55:47.912567 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:47.912340    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v" podUID="cc20924a-8101-43ab-95f1-8f0848d649f9" containerName="payload-processing" containerID="cri-o://5f3da6483a0618dd4827e8e363c1647cd219cc741e93a05551439b763ef39851" gracePeriod=30
Nov 03 01:55:47.966158 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:47.966132    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"]
Nov 03 01:55:47.966434 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:47.966413    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw" podUID="e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5" containerName="payload-pre-processing" containerID="cri-o://0509316348f441f6970302ededc57b2d61d808b481fb16774f48f8fd10b84df4" gracePeriod=30
Nov 03 01:55:48.221711 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.221684    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"
Nov 03 01:55:48.251111 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.251083    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"
Nov 03 01:55:48.333499 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.333469    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc20924a-8101-43ab-95f1-8f0848d649f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc20924a-8101-43ab-95f1-8f0848d649f9-plugins-config-volume\") pod \"cc20924a-8101-43ab-95f1-8f0848d649f9\" (UID: \"cc20924a-8101-43ab-95f1-8f0848d649f9\") "
Nov 03 01:55:48.333643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.333521    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-plugins-config-volume\") pod \"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5\" (UID: \"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5\") "
Nov 03 01:55:48.333643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.333562    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-kube-api-access-lx72z\" (UniqueName: \"kubernetes.io/projected/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-kube-api-access-lx72z\") pod \"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5\" (UID: \"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5\") "
Nov 03 01:55:48.333775 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.333641    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc20924a-8101-43ab-95f1-8f0848d649f9-kube-api-access-z28sj\" (UniqueName: \"kubernetes.io/projected/cc20924a-8101-43ab-95f1-8f0848d649f9-kube-api-access-z28sj\") pod \"cc20924a-8101-43ab-95f1-8f0848d649f9\" (UID: \"cc20924a-8101-43ab-95f1-8f0848d649f9\") "
Nov 03 01:55:48.334391 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.333950    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-plugins-config-volume" pod "e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5" (UID: "e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:48.334504 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.334315    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc20924a-8101-43ab-95f1-8f0848d649f9-plugins-config-volume" pod "cc20924a-8101-43ab-95f1-8f0848d649f9" (UID: "cc20924a-8101-43ab-95f1-8f0848d649f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:55:48.336021 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.335994    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-kube-api-access-lx72z" pod "e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5" (UID: "e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5"). InnerVolumeSpecName "kube-api-access-lx72z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:48.336460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.336439    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc20924a-8101-43ab-95f1-8f0848d649f9-kube-api-access-z28sj" pod "cc20924a-8101-43ab-95f1-8f0848d649f9" (UID: "cc20924a-8101-43ab-95f1-8f0848d649f9"). InnerVolumeSpecName "kube-api-access-z28sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:55:48.446843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.446748    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc20924a-8101-43ab-95f1-8f0848d649f9-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:48.446843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.446785    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:48.446843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.446801    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx72z\" (UniqueName: \"kubernetes.io/projected/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5-kube-api-access-lx72z\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:48.446843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.446815    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z28sj\" (UniqueName: \"kubernetes.io/projected/cc20924a-8101-43ab-95f1-8f0848d649f9-kube-api-access-z28sj\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:55:48.916818 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.916784    5508 generic.go:356] "Generic (PLEG): container finished" podID="e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5" containerID="0509316348f441f6970302ededc57b2d61d808b481fb16774f48f8fd10b84df4" exitCode=0
Nov 03 01:55:48.917270 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.916822    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"
Nov 03 01:55:48.917270 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.916862    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw" event={"ID":"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5","Type":"ContainerDied","Data":"0509316348f441f6970302ededc57b2d61d808b481fb16774f48f8fd10b84df4"}
Nov 03 01:55:48.917270 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.916922    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw" event={"ID":"e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5","Type":"ContainerDied","Data":"550d03952c517236a5b7855d507ffa265d389bb8175a53fb09b480dca35fef32"}
Nov 03 01:55:48.917270 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.916955    5508 scope.go:122] "RemoveContainer" containerID="0509316348f441f6970302ededc57b2d61d808b481fb16774f48f8fd10b84df4"
Nov 03 01:55:48.918914 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.918888    5508 generic.go:356] "Generic (PLEG): container finished" podID="cc20924a-8101-43ab-95f1-8f0848d649f9" containerID="5f3da6483a0618dd4827e8e363c1647cd219cc741e93a05551439b763ef39851" exitCode=0
Nov 03 01:55:48.919174 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.918976    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v" event={"ID":"cc20924a-8101-43ab-95f1-8f0848d649f9","Type":"ContainerDied","Data":"5f3da6483a0618dd4827e8e363c1647cd219cc741e93a05551439b763ef39851"}
Nov 03 01:55:48.919174 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.919013    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v" event={"ID":"cc20924a-8101-43ab-95f1-8f0848d649f9","Type":"ContainerDied","Data":"c94cf5d364a59e388752ddc47c6e1f3f917d14eea13ef86fbf6f336f0fe8d2aa"}
Nov 03 01:55:48.919419 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.919395    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"
Nov 03 01:55:48.926900 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.926882    5508 scope.go:122] "RemoveContainer" containerID="0509316348f441f6970302ededc57b2d61d808b481fb16774f48f8fd10b84df4"
Nov 03 01:55:48.927151 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:48.927134    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0509316348f441f6970302ededc57b2d61d808b481fb16774f48f8fd10b84df4\": container with ID starting with 0509316348f441f6970302ededc57b2d61d808b481fb16774f48f8fd10b84df4 not found: ID does not exist" containerID="0509316348f441f6970302ededc57b2d61d808b481fb16774f48f8fd10b84df4"
Nov 03 01:55:48.927233 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.927155    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0509316348f441f6970302ededc57b2d61d808b481fb16774f48f8fd10b84df4"} err="failed to get container status \"0509316348f441f6970302ededc57b2d61d808b481fb16774f48f8fd10b84df4\": rpc error: code = NotFound desc = could not find container \"0509316348f441f6970302ededc57b2d61d808b481fb16774f48f8fd10b84df4\": container with ID starting with 0509316348f441f6970302ededc57b2d61d808b481fb16774f48f8fd10b84df4 not found: ID does not exist"
Nov 03 01:55:48.927233 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.927170    5508 scope.go:122] "RemoveContainer" containerID="5f3da6483a0618dd4827e8e363c1647cd219cc741e93a05551439b763ef39851"
Nov 03 01:55:48.936882 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.936859    5508 scope.go:122] "RemoveContainer" containerID="5f3da6483a0618dd4827e8e363c1647cd219cc741e93a05551439b763ef39851"
Nov 03 01:55:48.937134 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:55:48.937118    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3da6483a0618dd4827e8e363c1647cd219cc741e93a05551439b763ef39851\": container with ID starting with 5f3da6483a0618dd4827e8e363c1647cd219cc741e93a05551439b763ef39851 not found: ID does not exist" containerID="5f3da6483a0618dd4827e8e363c1647cd219cc741e93a05551439b763ef39851"
Nov 03 01:55:48.937198 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:48.937140    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3da6483a0618dd4827e8e363c1647cd219cc741e93a05551439b763ef39851"} err="failed to get container status \"5f3da6483a0618dd4827e8e363c1647cd219cc741e93a05551439b763ef39851\": rpc error: code = NotFound desc = could not find container \"5f3da6483a0618dd4827e8e363c1647cd219cc741e93a05551439b763ef39851\": container with ID starting with 5f3da6483a0618dd4827e8e363c1647cd219cc741e93a05551439b763ef39851 not found: ID does not exist"
Nov 03 01:55:49.070636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:49.070564    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"]
Nov 03 01:55:49.081867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:49.074930    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b37ba5-98b55c6dd-mvd8v"]
Nov 03 01:55:49.176659 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:49.176623    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"]
Nov 03 01:55:49.182911 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:49.181959    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b37ba5-7b74c64cc5-7q2gw"]
Nov 03 01:55:50.692454 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:50.692419    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc20924a-8101-43ab-95f1-8f0848d649f9" path="/var/lib/kubelet/pods/cc20924a-8101-43ab-95f1-8f0848d649f9/volumes"
Nov 03 01:55:50.692969 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:50.692950    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5" path="/var/lib/kubelet/pods/e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5/volumes"
Nov 03 01:55:55.900762 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:55.900733    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"
Nov 03 01:55:55.901302 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:55.900805    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"
Nov 03 01:55:59.875904 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:59.875866    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"]
Nov 03 01:55:59.876647 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:59.876625    5508 cpu_manager.go:414] "containerMap: removing container" podUID="cc20924a-8101-43ab-95f1-8f0848d649f9" containerName="payload-processing"
Nov 03 01:55:59.876698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:59.876651    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc20924a-8101-43ab-95f1-8f0848d649f9" containerName="payload-processing"
Nov 03 01:55:59.876698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:59.876674    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5" containerName="payload-pre-processing"
Nov 03 01:55:59.876698 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:59.876682    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5" containerName="payload-pre-processing"
Nov 03 01:55:59.876816 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:59.876805    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc20924a-8101-43ab-95f1-8f0848d649f9" containerName="payload-processing"
Nov 03 01:55:59.876862 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:59.876823    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2dcfcd8-ff6a-41fb-b9bd-f3ad2d809db5" containerName="payload-pre-processing"
Nov 03 01:55:59.889308 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:59.882918    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"
Nov 03 01:55:59.976495 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:59.976456    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e2845e-0c51-428b-a854-a5a681d43341-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl\" (UID: \"66e2845e-0c51-428b-a854-a5a681d43341\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"
Nov 03 01:55:59.976694 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:59.976506    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqtq8\" (UniqueName: \"kubernetes.io/projected/66e2845e-0c51-428b-a854-a5a681d43341-kube-api-access-bqtq8\") pod \"payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl\" (UID: \"66e2845e-0c51-428b-a854-a5a681d43341\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"
Nov 03 01:55:59.978467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:59.978434    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-738acb-dockercfg-lgpzk\""
Nov 03 01:55:59.978574 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:55:59.978484    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-738acb\""
Nov 03 01:56:00.077835 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:00.077802    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e2845e-0c51-428b-a854-a5a681d43341-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl\" (UID: \"66e2845e-0c51-428b-a854-a5a681d43341\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"
Nov 03 01:56:00.077835 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:00.077840    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqtq8\" (UniqueName: \"kubernetes.io/projected/66e2845e-0c51-428b-a854-a5a681d43341-kube-api-access-bqtq8\") pod \"payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl\" (UID: \"66e2845e-0c51-428b-a854-a5a681d43341\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"
Nov 03 01:56:00.078740 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:00.078713    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e2845e-0c51-428b-a854-a5a681d43341-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl\" (UID: \"66e2845e-0c51-428b-a854-a5a681d43341\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"
Nov 03 01:56:00.136334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:00.136269    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqtq8\" (UniqueName: \"kubernetes.io/projected/66e2845e-0c51-428b-a854-a5a681d43341-kube-api-access-bqtq8\") pod \"payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl\" (UID: \"66e2845e-0c51-428b-a854-a5a681d43341\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"
Nov 03 01:56:00.201949 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:00.201921    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"
Nov 03 01:56:00.692435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:00.692405    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"]
Nov 03 01:56:00.696093 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:00.696070    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"
Nov 03 01:56:00.787925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:00.787884    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d79e96-1752-414a-acc6-5b80b18381ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv\" (UID: \"56d79e96-1752-414a-acc6-5b80b18381ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"
Nov 03 01:56:00.788090 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:00.788023    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpcv8\" (UniqueName: \"kubernetes.io/projected/56d79e96-1752-414a-acc6-5b80b18381ab-kube-api-access-xpcv8\") pod \"payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv\" (UID: \"56d79e96-1752-414a-acc6-5b80b18381ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"
Nov 03 01:56:00.889003 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:00.888965    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpcv8\" (UniqueName: \"kubernetes.io/projected/56d79e96-1752-414a-acc6-5b80b18381ab-kube-api-access-xpcv8\") pod \"payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv\" (UID: \"56d79e96-1752-414a-acc6-5b80b18381ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"
Nov 03 01:56:00.889447 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:00.889111    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d79e96-1752-414a-acc6-5b80b18381ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv\" (UID: \"56d79e96-1752-414a-acc6-5b80b18381ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"
Nov 03 01:56:00.889848 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:00.889826    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d79e96-1752-414a-acc6-5b80b18381ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv\" (UID: \"56d79e96-1752-414a-acc6-5b80b18381ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"
Nov 03 01:56:00.946601 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:00.946537    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpcv8\" (UniqueName: \"kubernetes.io/projected/56d79e96-1752-414a-acc6-5b80b18381ab-kube-api-access-xpcv8\") pod \"payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv\" (UID: \"56d79e96-1752-414a-acc6-5b80b18381ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"
Nov 03 01:56:01.010109 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:01.010039    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"
Nov 03 01:56:01.512772 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:01.512736    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:56:05.512455 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.512415    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"]
Nov 03 01:56:05.517826 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.517805    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.570380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.570334    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-70235a-openshift-default-dockercfg-h2hdh\""
Nov 03 01:56:05.641017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.640990    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-credential-socket\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.641161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.641048    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21141eda-543d-4fa6-9404-8289591a86d6-istio-podinfo\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.641161 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.641117    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-certs\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.641254 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.641179    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-envoy\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.641291 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.641272    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-istio-token\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.641325 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.641311    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-data\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.641382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.641340    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21141eda-543d-4fa6-9404-8289591a86d6-istiod-ca-cert\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.641430 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.641407    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-socket\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.641467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.641433    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rdbg\" (UniqueName: \"kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-kube-api-access-8rdbg\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.744644 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.744606    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rdbg\" (UniqueName: \"kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-kube-api-access-8rdbg\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.744929 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.744914    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-credential-socket\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.745077 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.745064    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21141eda-543d-4fa6-9404-8289591a86d6-istio-podinfo\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.745195 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.745183    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-certs\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.745381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.745365    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-envoy\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.745512 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.745500    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-istio-token\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.745643 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.745630    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-data\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.745746 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.745734    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21141eda-543d-4fa6-9404-8289591a86d6-istiod-ca-cert\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.745846 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.745834    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-socket\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.748270 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.748243    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-socket\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.749201 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.749177    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-credential-socket\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.750152 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.750131    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-data\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.762369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.750910    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21141eda-543d-4fa6-9404-8289591a86d6-istiod-ca-cert\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.764592 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.762839    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-certs\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.783381 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.783336    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21141eda-543d-4fa6-9404-8289591a86d6-istio-podinfo\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.813377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.806460    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-envoy\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.813377 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.810082    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rdbg\" (UniqueName: \"kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-kube-api-access-8rdbg\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.826379 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.822956    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-istio-token\") pod \"e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") " pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:05.832033 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:05.832006    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:10.377972 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.377935    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"]
Nov 03 01:56:10.378773 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.378219    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c" podUID="721dd14b-55de-4342-a676-e06e3d8ae2e3" containerName="maas-api" containerID="cri-o://5b8a30c2784bb2fcc7fdad1f45429855e3ba8afdb1fee6059c2b4c552178dd34" gracePeriod=30
Nov 03 01:56:10.675127 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.675105    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:56:10.833879 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.833847    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-maas-api-tls\") pod \"721dd14b-55de-4342-a676-e06e3d8ae2e3\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") "
Nov 03 01:56:10.836056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.833897    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721dd14b-55de-4342-a676-e06e3d8ae2e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/721dd14b-55de-4342-a676-e06e3d8ae2e3-openshift-service-ca\") pod \"721dd14b-55de-4342-a676-e06e3d8ae2e3\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") "
Nov 03 01:56:10.836056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.833973    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-metrics-certs\") pod \"721dd14b-55de-4342-a676-e06e3d8ae2e3\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") "
Nov 03 01:56:10.836056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.834014    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721dd14b-55de-4342-a676-e06e3d8ae2e3-kube-api-access-248lz\" (UniqueName: \"kubernetes.io/projected/721dd14b-55de-4342-a676-e06e3d8ae2e3-kube-api-access-248lz\") pod \"721dd14b-55de-4342-a676-e06e3d8ae2e3\" (UID: \"721dd14b-55de-4342-a676-e06e3d8ae2e3\") "
Nov 03 01:56:10.836056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.834295    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721dd14b-55de-4342-a676-e06e3d8ae2e3-openshift-service-ca" pod "721dd14b-55de-4342-a676-e06e3d8ae2e3" (UID: "721dd14b-55de-4342-a676-e06e3d8ae2e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:56:10.836056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.834499    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/721dd14b-55de-4342-a676-e06e3d8ae2e3-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:10.836056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.835976    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-maas-api-tls" pod "721dd14b-55de-4342-a676-e06e3d8ae2e3" (UID: "721dd14b-55de-4342-a676-e06e3d8ae2e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:56:10.836285 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.836087    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721dd14b-55de-4342-a676-e06e3d8ae2e3-kube-api-access-248lz" pod "721dd14b-55de-4342-a676-e06e3d8ae2e3" (UID: "721dd14b-55de-4342-a676-e06e3d8ae2e3"). InnerVolumeSpecName "kube-api-access-248lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:56:10.836285 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.836118    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-metrics-certs" pod "721dd14b-55de-4342-a676-e06e3d8ae2e3" (UID: "721dd14b-55de-4342-a676-e06e3d8ae2e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:56:10.935615 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.935587    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:10.935725 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.935621    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-248lz\" (UniqueName: \"kubernetes.io/projected/721dd14b-55de-4342-a676-e06e3d8ae2e3-kube-api-access-248lz\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:10.935725 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:10.935637    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/721dd14b-55de-4342-a676-e06e3d8ae2e3-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:11.053971 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.053879    5508 generic.go:356] "Generic (PLEG): container finished" podID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerID="ca902b641aaf8e5779c64a63d6db5f3814de24e9941cf528112d99006b101b8c" exitCode=0
Nov 03 01:56:11.054157 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.053991    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8" event={"ID":"3aa8e146-3961-4111-9712-e1323fdf7a08","Type":"ContainerDied","Data":"ca902b641aaf8e5779c64a63d6db5f3814de24e9941cf528112d99006b101b8c"}
Nov 03 01:56:11.054490 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.054468    5508 scope.go:122] "RemoveContainer" containerID="ca902b641aaf8e5779c64a63d6db5f3814de24e9941cf528112d99006b101b8c"
Nov 03 01:56:11.055534 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.055495    5508 generic.go:356] "Generic (PLEG): container finished" podID="721dd14b-55de-4342-a676-e06e3d8ae2e3" containerID="5b8a30c2784bb2fcc7fdad1f45429855e3ba8afdb1fee6059c2b4c552178dd34" exitCode=0
Nov 03 01:56:11.055636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.055545    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"
Nov 03 01:56:11.055636 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.055541    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c" event={"ID":"721dd14b-55de-4342-a676-e06e3d8ae2e3","Type":"ContainerDied","Data":"5b8a30c2784bb2fcc7fdad1f45429855e3ba8afdb1fee6059c2b4c552178dd34"}
Nov 03 01:56:11.055754 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.055676    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c" event={"ID":"721dd14b-55de-4342-a676-e06e3d8ae2e3","Type":"ContainerDied","Data":"21aebd959be8b71601559436f0d289978053fd9971dccae2bfb9d91f05aa421a"}
Nov 03 01:56:11.055754 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.055708    5508 scope.go:122] "RemoveContainer" containerID="5b8a30c2784bb2fcc7fdad1f45429855e3ba8afdb1fee6059c2b4c552178dd34"
Nov 03 01:56:11.055893 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.055864    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:11.056017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.055947    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:11.056017 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.055995    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:11.069209 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.069191    5508 scope.go:122] "RemoveContainer" containerID="5b8a30c2784bb2fcc7fdad1f45429855e3ba8afdb1fee6059c2b4c552178dd34"
Nov 03 01:56:11.069535 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:56:11.069514    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8a30c2784bb2fcc7fdad1f45429855e3ba8afdb1fee6059c2b4c552178dd34\": container with ID starting with 5b8a30c2784bb2fcc7fdad1f45429855e3ba8afdb1fee6059c2b4c552178dd34 not found: ID does not exist" containerID="5b8a30c2784bb2fcc7fdad1f45429855e3ba8afdb1fee6059c2b4c552178dd34"
Nov 03 01:56:11.069623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.069542    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8a30c2784bb2fcc7fdad1f45429855e3ba8afdb1fee6059c2b4c552178dd34"} err="failed to get container status \"5b8a30c2784bb2fcc7fdad1f45429855e3ba8afdb1fee6059c2b4c552178dd34\": rpc error: code = NotFound desc = could not find container \"5b8a30c2784bb2fcc7fdad1f45429855e3ba8afdb1fee6059c2b4c552178dd34\": container with ID starting with 5b8a30c2784bb2fcc7fdad1f45429855e3ba8afdb1fee6059c2b4c552178dd34 not found: ID does not exist"
Nov 03 01:56:11.273825 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.273749    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"]
Nov 03 01:56:11.278102 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:11.278074    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0e0d61-79c6945fdb-mw47c"]
Nov 03 01:56:12.077797 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:12.077756    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8" event={"ID":"3aa8e146-3961-4111-9712-e1323fdf7a08","Type":"ContainerStarted","Data":"165aab56cfe7c46f2fe10558a030a43e20f486f08dfacf844c41c465ea521bb4"}
Nov 03 01:56:12.271374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:12.270781    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:56:12.290420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:12.286779    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 01:56:12.290420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:12.286850    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8" podUID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 01:56:12.691821 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:12.691788    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721dd14b-55de-4342-a676-e06e3d8ae2e3" path="/var/lib/kubelet/pods/721dd14b-55de-4342-a676-e06e3d8ae2e3/volumes"
Nov 03 01:56:13.211854 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:13.211825    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:56:13.270892 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:13.270860    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 01:56:13.271147 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:13.271122    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8" podUID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 01:56:13.813981 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:56:13.813948    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:56:13.971765 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:13.971705    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"]
Nov 03 01:56:13.972276 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:13.972241    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv" podUID="063d1af5-8267-423a-be4b-a46a27c2e527" containerName="istio-proxy" containerID="cri-o://5dc32cabaac82471c5a28b2d74e4cddfd8febbaf07c86cee2081d8fb29578fcc" gracePeriod=30
Nov 03 01:56:14.268479 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:14.268441    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 01:56:14.269078 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:14.269045    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8" podUID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 01:56:15.262733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.262699    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 01:56:15.263003 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.262975    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8" podUID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 01:56:15.585424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.583111    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"]
Nov 03 01:56:15.585424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.583923    5508 cpu_manager.go:414] "containerMap: removing container" podUID="721dd14b-55de-4342-a676-e06e3d8ae2e3" containerName="maas-api"
Nov 03 01:56:15.585424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.583943    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721dd14b-55de-4342-a676-e06e3d8ae2e3" containerName="maas-api"
Nov 03 01:56:15.585424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.584080    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="721dd14b-55de-4342-a676-e06e3d8ae2e3" containerName="maas-api"
Nov 03 01:56:15.589056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.589028    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.638709 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.638683    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f95773-openshift-default-dockercfg-hwbnw\""
Nov 03 01:56:15.752426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.749027    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16676061-80e9-435c-923b-e4e9b6d128d0-istio-podinfo\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.752426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.749097    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16676061-80e9-435c-923b-e4e9b6d128d0-istiod-ca-cert\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.752426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.749147    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-data\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.752426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.749172    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-istio-token\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.752426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.749210    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87fkn\" (UniqueName: \"kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-kube-api-access-87fkn\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.752426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.749272    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-certs\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.752426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.749382    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-credential-socket\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.752426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.749424    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-envoy\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.752426 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.749471    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-socket\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.868443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.865045    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16676061-80e9-435c-923b-e4e9b6d128d0-istiod-ca-cert\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.868443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.865130    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-data\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.868443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.865157    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-istio-token\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.868443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.865193    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87fkn\" (UniqueName: \"kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-kube-api-access-87fkn\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.868443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.865268    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-certs\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.868443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.865377    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-credential-socket\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.868443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.865415    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-envoy\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.868443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.865464    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-socket\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.868443 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.865549    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16676061-80e9-435c-923b-e4e9b6d128d0-istio-podinfo\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.871423 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.871396    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16676061-80e9-435c-923b-e4e9b6d128d0-istiod-ca-cert\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.872160 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.872139    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-data\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.889384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.882921    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-socket\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.889384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.887305    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-certs\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.889384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.887671    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-credential-socket\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.889384 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.888713    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16676061-80e9-435c-923b-e4e9b6d128d0-istio-podinfo\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.890071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.890046    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-envoy\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.929141 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.929119    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-istio-token\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:15.929361 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:15.929324    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87fkn\" (UniqueName: \"kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-kube-api-access-87fkn\") pod \"e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") " pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:16.212421 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:16.212387    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:16.274252 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:16.273804    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 03 01:56:16.274252 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:16.273876    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8" podUID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 03 01:56:17.268423 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:17.268394    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:56:17.269536 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:17.269516    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:56:19.228442 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.228411    5508 generic.go:356] "Generic (PLEG): container finished" podID="063d1af5-8267-423a-be4b-a46a27c2e527" containerID="5dc32cabaac82471c5a28b2d74e4cddfd8febbaf07c86cee2081d8fb29578fcc" exitCode=0
Nov 03 01:56:19.228822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.228485    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv" event={"ID":"063d1af5-8267-423a-be4b-a46a27c2e527","Type":"ContainerDied","Data":"5dc32cabaac82471c5a28b2d74e4cddfd8febbaf07c86cee2081d8fb29578fcc"}
Nov 03 01:56:19.228822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.228541    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv" event={"ID":"063d1af5-8267-423a-be4b-a46a27c2e527","Type":"ContainerDied","Data":"b087583d56b209496f08454dd6c30d20ce6ae6ade9ab76ba353664211c6dc454"}
Nov 03 01:56:19.228822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.228560    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b087583d56b209496f08454dd6c30d20ce6ae6ade9ab76ba353664211c6dc454"
Nov 03 01:56:19.228822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.228572    5508 scope.go:122] "RemoveContainer" containerID="87f7c728154202051c6b1aa4e67e4b70baaa270305ded414b8430ad15ae076d5"
Nov 03 01:56:19.242869 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.242824    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:56:19.308552 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.308523    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-certs\") pod \"063d1af5-8267-423a-be4b-a46a27c2e527\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") "
Nov 03 01:56:19.308723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.308561    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-credential-socket\") pod \"063d1af5-8267-423a-be4b-a46a27c2e527\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") "
Nov 03 01:56:19.308723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.308583    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-socket\") pod \"063d1af5-8267-423a-be4b-a46a27c2e527\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") "
Nov 03 01:56:19.308723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.308628    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-envoy\") pod \"063d1af5-8267-423a-be4b-a46a27c2e527\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") "
Nov 03 01:56:19.308723 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.308667    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/063d1af5-8267-423a-be4b-a46a27c2e527-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063d1af5-8267-423a-be4b-a46a27c2e527-istio-podinfo\") pod \"063d1af5-8267-423a-be4b-a46a27c2e527\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") "
Nov 03 01:56:19.308952 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.308740    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-certs" pod "063d1af5-8267-423a-be4b-a46a27c2e527" (UID: "063d1af5-8267-423a-be4b-a46a27c2e527"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:56:19.308952 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.308758    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-kube-api-access-wqlr5\" (UniqueName: \"kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-kube-api-access-wqlr5\") pod \"063d1af5-8267-423a-be4b-a46a27c2e527\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") "
Nov 03 01:56:19.308952 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.308800    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-istio-token\" (UniqueName: \"kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-istio-token\") pod \"063d1af5-8267-423a-be4b-a46a27c2e527\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") "
Nov 03 01:56:19.308952 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.308867    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-data\") pod \"063d1af5-8267-423a-be4b-a46a27c2e527\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") "
Nov 03 01:56:19.309200 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.309163    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/063d1af5-8267-423a-be4b-a46a27c2e527-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063d1af5-8267-423a-be4b-a46a27c2e527-istiod-ca-cert\") pod \"063d1af5-8267-423a-be4b-a46a27c2e527\" (UID: \"063d1af5-8267-423a-be4b-a46a27c2e527\") "
Nov 03 01:56:19.309460 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.308880    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-credential-socket" pod "063d1af5-8267-423a-be4b-a46a27c2e527" (UID: "063d1af5-8267-423a-be4b-a46a27c2e527"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:56:19.309550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.308890    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-socket" pod "063d1af5-8267-423a-be4b-a46a27c2e527" (UID: "063d1af5-8267-423a-be4b-a46a27c2e527"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:56:19.309550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.309088    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-data" pod "063d1af5-8267-423a-be4b-a46a27c2e527" (UID: "063d1af5-8267-423a-be4b-a46a27c2e527"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:56:19.309550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.309472    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063d1af5-8267-423a-be4b-a46a27c2e527-istiod-ca-cert" pod "063d1af5-8267-423a-be4b-a46a27c2e527" (UID: "063d1af5-8267-423a-be4b-a46a27c2e527"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:56:19.309827 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.309808    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:19.309898 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.309833    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063d1af5-8267-423a-be4b-a46a27c2e527-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:19.309898 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.309846    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:19.309898 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.309859    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:19.309898 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.309872    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:19.311412 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.311369    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-envoy" pod "063d1af5-8267-423a-be4b-a46a27c2e527" (UID: "063d1af5-8267-423a-be4b-a46a27c2e527"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:56:19.311498 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.311450    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/063d1af5-8267-423a-be4b-a46a27c2e527-istio-podinfo" pod "063d1af5-8267-423a-be4b-a46a27c2e527" (UID: "063d1af5-8267-423a-be4b-a46a27c2e527"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:56:19.311498 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.311490    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-istio-token" pod "063d1af5-8267-423a-be4b-a46a27c2e527" (UID: "063d1af5-8267-423a-be4b-a46a27c2e527"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:56:19.311588 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.311515    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-kube-api-access-wqlr5" pod "063d1af5-8267-423a-be4b-a46a27c2e527" (UID: "063d1af5-8267-423a-be4b-a46a27c2e527"). InnerVolumeSpecName "kube-api-access-wqlr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:56:19.413489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.413458    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqlr5\" (UniqueName: \"kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-kube-api-access-wqlr5\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:19.413489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.413486    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063d1af5-8267-423a-be4b-a46a27c2e527-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:19.413489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.413496    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063d1af5-8267-423a-be4b-a46a27c2e527-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:19.413726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:19.413504    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063d1af5-8267-423a-be4b-a46a27c2e527-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:56:20.235650 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:20.235621    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"
Nov 03 01:56:20.484056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:20.484024    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"]
Nov 03 01:56:20.488615 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:20.488553    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0e0d61-openshift-default-6d9474b6f5-6rxhv"]
Nov 03 01:56:20.694319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:20.694286    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="063d1af5-8267-423a-be4b-a46a27c2e527" path="/var/lib/kubelet/pods/063d1af5-8267-423a-be4b-a46a27c2e527/volumes"
Nov 03 01:56:29.854165 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.854120    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"]
Nov 03 01:56:29.854828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.854660    5508 cpu_manager.go:414] "containerMap: removing container" podUID="063d1af5-8267-423a-be4b-a46a27c2e527" containerName="istio-proxy"
Nov 03 01:56:29.854828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.854673    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="063d1af5-8267-423a-be4b-a46a27c2e527" containerName="istio-proxy"
Nov 03 01:56:29.854828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.854782    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="063d1af5-8267-423a-be4b-a46a27c2e527" containerName="istio-proxy"
Nov 03 01:56:29.854828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.854795    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="063d1af5-8267-423a-be4b-a46a27c2e527" containerName="istio-proxy"
Nov 03 01:56:29.859020 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.858997    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:29.907936 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.907902    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e2db44-openshift-default-dockercfg-bqnnz\""
Nov 03 01:56:29.928414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.928378    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-envoy\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:29.928568 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.928420    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-data\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:29.928568 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.928525    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc85e883-f97a-417c-b58a-8535213343c0-istiod-ca-cert\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:29.928568 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.928564    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-istio-token\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:29.928699 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.928676    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9ns7\" (UniqueName: \"kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-kube-api-access-m9ns7\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:29.928747 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.928724    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-credential-socket\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:29.928787 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.928745    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-socket\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:29.928889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.928872    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-certs\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:29.928939 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:29.928920    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc85e883-f97a-417c-b58a-8535213343c0-istio-podinfo\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.029860 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.029826    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-envoy\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.030041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.029866    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-data\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.030041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.029902    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc85e883-f97a-417c-b58a-8535213343c0-istiod-ca-cert\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.030041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.029928    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-istio-token\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.030041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.030001    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9ns7\" (UniqueName: \"kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-kube-api-access-m9ns7\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.030275 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.030048    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-credential-socket\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.030275 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.030073    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-socket\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.030275 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.030123    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-certs\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.030275 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.030214    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc85e883-f97a-417c-b58a-8535213343c0-istio-podinfo\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.030953 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.030916    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-data\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.031071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.030981    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-socket\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.031071 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.031042    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-credential-socket\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.031248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.031227    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc85e883-f97a-417c-b58a-8535213343c0-istiod-ca-cert\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.031917 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.031459    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-certs\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.032563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.032531    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-envoy\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.045167 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.045140    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc85e883-f97a-417c-b58a-8535213343c0-istio-podinfo\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.113212 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.113134    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9ns7\" (UniqueName: \"kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-kube-api-access-m9ns7\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.114237 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.114212    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-istio-token\") pod \"e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") " pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:30.172685 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:30.172651    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:31.869322 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:31.869284    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"]
Nov 03 01:56:31.870100 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:31.870080    5508 cpu_manager.go:414] "containerMap: removing container" podUID="063d1af5-8267-423a-be4b-a46a27c2e527" containerName="istio-proxy"
Nov 03 01:56:31.870162 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:31.870103    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="063d1af5-8267-423a-be4b-a46a27c2e527" containerName="istio-proxy"
Nov 03 01:56:31.873875 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:31.873853    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:31.922400 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:31.922374    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-706bdd-serving-cert\""
Nov 03 01:56:31.954952 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:31.954918    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:31.955120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:31.955004    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/019e9d63-5cc5-4a76-8079-b27929c4174d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:31.955120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:31.955031    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45p5q\" (UniqueName: \"kubernetes.io/projected/019e9d63-5cc5-4a76-8079-b27929c4174d-kube-api-access-45p5q\") pod \"maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:31.955241 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:31.955218    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-metrics-certs\") pod \"maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:32.056863 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:32.056832    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-metrics-certs\") pod \"maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:32.057038 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:32.056901    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:32.057038 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:32.057012    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/019e9d63-5cc5-4a76-8079-b27929c4174d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:32.057129 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:56:32.057061    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-serving-cert: secret "maas-api-e2e-praxis-a-706bdd-serving-cert" not found
Nov 03 01:56:32.057129 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:32.057122    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45p5q\" (UniqueName: \"kubernetes.io/projected/019e9d63-5cc5-4a76-8079-b27929c4174d-kube-api-access-45p5q\") pod \"maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:32.057226 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:56:32.057145    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-maas-api-tls podName:019e9d63-5cc5-4a76-8079-b27929c4174d nodeName:}" failed. No retries permitted until 2026-11-03 01:56:32.557123629 +0000 UTC m=+2271.982730073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-maas-api-tls") pod "maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng" (UID: "019e9d63-5cc5-4a76-8079-b27929c4174d") : secret "maas-api-e2e-praxis-a-706bdd-serving-cert" not found
Nov 03 01:56:32.057756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:32.057733    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/019e9d63-5cc5-4a76-8079-b27929c4174d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:32.059918 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:32.059897    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-metrics-certs\") pod \"maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:32.124149 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:32.123904    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45p5q\" (UniqueName: \"kubernetes.io/projected/019e9d63-5cc5-4a76-8079-b27929c4174d-kube-api-access-45p5q\") pod \"maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:32.564205 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:32.564163    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:32.566763 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:32.566728    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:32.791696 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:32.791660    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:35.912646 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:35.912608    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"]
Nov 03 01:56:35.968107 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:35.968023    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"]
Nov 03 01:56:35.971563 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:35.971522    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"]
Nov 03 01:56:35.975284 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:35.975244    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"]
Nov 03 01:56:36.013614 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.013499    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"]
Nov 03 01:56:36.020120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.020087    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"]
Nov 03 01:56:36.251807 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.251721    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"]
Nov 03 01:56:36.258924 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:56:36.258900    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d79e96_1752_414a_acc6_5b80b18381ab.slice/crio-addb02d1a086b0eacb6525d3ff80ab96a990383b0331779bae85372354f996a5 WatchSource:0}: Error finding container addb02d1a086b0eacb6525d3ff80ab96a990383b0331779bae85372354f996a5: Status 404 returned error can't find the container with id addb02d1a086b0eacb6525d3ff80ab96a990383b0331779bae85372354f996a5
Nov 03 01:56:36.308941 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.308915    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"]
Nov 03 01:56:36.311479 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.311457    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"]
Nov 03 01:56:36.316075 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:56:36.316035    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc85e883_f97a_417c_b58a_8535213343c0.slice/crio-2c776356943d4f326c85d314c1e767480e8c4dc5a7bb3e6e313a737cf0f586bc WatchSource:0}: Error finding container 2c776356943d4f326c85d314c1e767480e8c4dc5a7bb3e6e313a737cf0f586bc: Status 404 returned error can't find the container with id 2c776356943d4f326c85d314c1e767480e8c4dc5a7bb3e6e313a737cf0f586bc
Nov 03 01:56:36.317839 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:56:36.317803    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21141eda_543d_4fa6_9404_8289591a86d6.slice/crio-d3683a0b3bcfc1a1ae77fcbc82ff1e61e359e4159a475f51621a2606e901033b WatchSource:0}: Error finding container d3683a0b3bcfc1a1ae77fcbc82ff1e61e359e4159a475f51621a2606e901033b: Status 404 returned error can't find the container with id d3683a0b3bcfc1a1ae77fcbc82ff1e61e359e4159a475f51621a2606e901033b
Nov 03 01:56:36.321478 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.321447    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:36.321573 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.321541    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:36.321618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.321572    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:36.321662 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.321635    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:36.321726 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.321708    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:36.321766 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.321756    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:36.322053 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.321993    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"]
Nov 03 01:56:36.326244 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.326223    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"]
Nov 03 01:56:36.335550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.335516    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"]
Nov 03 01:56:36.336879 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:56:36.336778    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e2845e_0c51_428b_a854_a5a681d43341.slice/crio-7b4d0a4fd714c36b7b1b998683bb3ece3a71aa55df5ed80aa86b9313f758cdc6 WatchSource:0}: Error finding container 7b4d0a4fd714c36b7b1b998683bb3ece3a71aa55df5ed80aa86b9313f758cdc6: Status 404 returned error can't find the container with id 7b4d0a4fd714c36b7b1b998683bb3ece3a71aa55df5ed80aa86b9313f758cdc6
Nov 03 01:56:36.339902 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:56:36.339878    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16676061_80e9_435c_923b_e4e9b6d128d0.slice/crio-a462cd7db3a62145729f6849194b90d65c3d097e1abe665147a9f34f4bb166a7 WatchSource:0}: Error finding container a462cd7db3a62145729f6849194b90d65c3d097e1abe665147a9f34f4bb166a7: Status 404 returned error can't find the container with id a462cd7db3a62145729f6849194b90d65c3d097e1abe665147a9f34f4bb166a7
Nov 03 01:56:36.345225 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:56:36.345164    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019e9d63_5cc5_4a76_8079_b27929c4174d.slice/crio-999b9308ee17ddcd70b0aa9cf6f0465df009615209e2de504e7ed25f8db967d7 WatchSource:0}: Error finding container 999b9308ee17ddcd70b0aa9cf6f0465df009615209e2de504e7ed25f8db967d7: Status 404 returned error can't find the container with id 999b9308ee17ddcd70b0aa9cf6f0465df009615209e2de504e7ed25f8db967d7
Nov 03 01:56:36.346107 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.346072    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:36.346310 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.346289    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:36.346509 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.346490    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:36.348261 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.348218    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6" event={"ID":"cc85e883-f97a-417c-b58a-8535213343c0","Type":"ContainerStarted","Data":"2c776356943d4f326c85d314c1e767480e8c4dc5a7bb3e6e313a737cf0f586bc"}
Nov 03 01:56:36.353541 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.353468    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv" event={"ID":"56d79e96-1752-414a-acc6-5b80b18381ab","Type":"ContainerStarted","Data":"addb02d1a086b0eacb6525d3ff80ab96a990383b0331779bae85372354f996a5"}
Nov 03 01:56:36.361480 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:36.361409    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" event={"ID":"21141eda-543d-4fa6-9404-8289591a86d6","Type":"ContainerStarted","Data":"d3683a0b3bcfc1a1ae77fcbc82ff1e61e359e4159a475f51621a2606e901033b"}
Nov 03 01:56:37.370293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.370257    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6" event={"ID":"cc85e883-f97a-417c-b58a-8535213343c0","Type":"ContainerStarted","Data":"e640d927b58acd93605e7b8f5b596db74f6d33f7847ab8864bc0a1dfd763b4eb"}
Nov 03 01:56:37.372113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.372090    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl" event={"ID":"66e2845e-0c51-428b-a854-a5a681d43341","Type":"ContainerStarted","Data":"9ef9133121869c6e069b4eb9666b94891f437515f6b16870279f3c87a670d7d7"}
Nov 03 01:56:37.372249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.372134    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl" event={"ID":"66e2845e-0c51-428b-a854-a5a681d43341","Type":"ContainerStarted","Data":"7b4d0a4fd714c36b7b1b998683bb3ece3a71aa55df5ed80aa86b9313f758cdc6"}
Nov 03 01:56:37.372249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.372162    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"
Nov 03 01:56:37.373925 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.373900    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv" event={"ID":"56d79e96-1752-414a-acc6-5b80b18381ab","Type":"ContainerStarted","Data":"de3d8a9a678057363009b9b9b2d0982e815430cafc06766ac04238588098ece0"}
Nov 03 01:56:37.374089 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.374069    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"
Nov 03 01:56:37.376828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.376800    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs" event={"ID":"16676061-80e9-435c-923b-e4e9b6d128d0","Type":"ContainerStarted","Data":"f4cb768843505ccc95c1f799d4d650c4dae09efa3e4a1fc5c9d6004938fc9550"}
Nov 03 01:56:37.376951 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.376850    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs" event={"ID":"16676061-80e9-435c-923b-e4e9b6d128d0","Type":"ContainerStarted","Data":"a462cd7db3a62145729f6849194b90d65c3d097e1abe665147a9f34f4bb166a7"}
Nov 03 01:56:37.378912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.378875    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" event={"ID":"21141eda-543d-4fa6-9404-8289591a86d6","Type":"ContainerStarted","Data":"20d24d8809523e1585b5c44693111bf0f78d4d726805c142d4cfc3b23b0d17db"}
Nov 03 01:56:37.381618 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.381555    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng" event={"ID":"019e9d63-5cc5-4a76-8079-b27929c4174d","Type":"ContainerStarted","Data":"999b9308ee17ddcd70b0aa9cf6f0465df009615209e2de504e7ed25f8db967d7"}
Nov 03 01:56:37.484305 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.484249    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6" podStartSLOduration=8.484232663 podStartE2EDuration="8.484232663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:37.483554424 +0000 UTC m=+2276.909160914" watchObservedRunningTime="2026-11-03 01:56:37.484232663 +0000 UTC m=+2276.909839128"
Nov 03 01:56:37.610458 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.608785    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" podStartSLOduration=32.60876573 podStartE2EDuration="32.60876573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:37.608153874 +0000 UTC m=+2277.033760363" watchObservedRunningTime="2026-11-03 01:56:37.60876573 +0000 UTC m=+2277.034372199"
Nov 03 01:56:37.741372 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.724544    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv" podStartSLOduration=37.72451459 podStartE2EDuration="37.72451459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:37.723080616 +0000 UTC m=+2277.148687080" watchObservedRunningTime="2026-11-03 01:56:37.72451459 +0000 UTC m=+2277.150121054"
Nov 03 01:56:37.832178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.832140    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:37.838073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.838049    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:37.839374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.839300    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs" podStartSLOduration=22.839282771 podStartE2EDuration="22.839282771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:37.83549806 +0000 UTC m=+2277.261104542" watchObservedRunningTime="2026-11-03 01:56:37.839282771 +0000 UTC m=+2277.264889237"
Nov 03 01:56:37.970173 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:37.970114    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl" podStartSLOduration=38.970093243 podStartE2EDuration="38.970093243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:37.955708932 +0000 UTC m=+2277.381315398" watchObservedRunningTime="2026-11-03 01:56:37.970093243 +0000 UTC m=+2277.395699708"
Nov 03 01:56:38.173129 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:38.173096    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:38.178249 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:38.178222    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:38.213441 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:38.213417    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:38.218551 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:38.218531    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:38.392924 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:38.392830    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng" event={"ID":"019e9d63-5cc5-4a76-8079-b27929c4174d","Type":"ContainerStarted","Data":"ea963a7e3662fd3393c2eafe7b5af97c335ae8c100c2f5da9b28f0ee91f896ff"}
Nov 03 01:56:38.392924 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:38.392911    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:38.393502 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:38.393304    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:38.393502 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:38.393361    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:38.393690 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:38.393670    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:38.394518 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:38.394495    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:56:38.394649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:38.394556    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:56:38.394649 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:38.394594    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:56:38.868609 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:38.868552    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng" podStartSLOduration=6.896613461 podStartE2EDuration="7.868532568s" totalImagesPullingTime="971.919107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:38.867664008 +0000 UTC m=+2278.293270491" watchObservedRunningTime="2026-11-03 01:56:38.868532568 +0000 UTC m=+2278.294139033"
Nov 03 01:56:43.874044 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:56:43.873968    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:56:44.261651 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.261618    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"]
Nov 03 01:56:44.267075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.267049    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"
Nov 03 01:56:44.318633 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.318606    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"]
Nov 03 01:56:44.362078 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.362037    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e2db44\""
Nov 03 01:56:44.362254 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.362119    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e2db44-dockercfg-4vlsb\""
Nov 03 01:56:44.373719 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.373685    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c241cdce-5e2e-4a75-8b72-d628bc68606f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc\" (UID: \"c241cdce-5e2e-4a75-8b72-d628bc68606f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"
Nov 03 01:56:44.374018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.374001    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vpdg\" (UniqueName: \"kubernetes.io/projected/c241cdce-5e2e-4a75-8b72-d628bc68606f-kube-api-access-9vpdg\") pod \"payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc\" (UID: \"c241cdce-5e2e-4a75-8b72-d628bc68606f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"
Nov 03 01:56:44.405647 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.405619    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 01:56:44.475529 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.475491    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c241cdce-5e2e-4a75-8b72-d628bc68606f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc\" (UID: \"c241cdce-5e2e-4a75-8b72-d628bc68606f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"
Nov 03 01:56:44.475733 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.475709    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vpdg\" (UniqueName: \"kubernetes.io/projected/c241cdce-5e2e-4a75-8b72-d628bc68606f-kube-api-access-9vpdg\") pod \"payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc\" (UID: \"c241cdce-5e2e-4a75-8b72-d628bc68606f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"
Nov 03 01:56:44.476272 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.476246    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c241cdce-5e2e-4a75-8b72-d628bc68606f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc\" (UID: \"c241cdce-5e2e-4a75-8b72-d628bc68606f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"
Nov 03 01:56:44.533250 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.533175    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vpdg\" (UniqueName: \"kubernetes.io/projected/c241cdce-5e2e-4a75-8b72-d628bc68606f-kube-api-access-9vpdg\") pod \"payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc\" (UID: \"c241cdce-5e2e-4a75-8b72-d628bc68606f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"
Nov 03 01:56:44.580908 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.580867    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"
Nov 03 01:56:44.965897 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:44.965860    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"]
Nov 03 01:56:44.973567 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:56:44.973534    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc241cdce_5e2e_4a75_8b72_d628bc68606f.slice/crio-f26dd451307aa00ea9c3985310f506a424a0cc976adda8c46c17eb211c476ff9 WatchSource:0}: Error finding container f26dd451307aa00ea9c3985310f506a424a0cc976adda8c46c17eb211c476ff9: Status 404 returned error can't find the container with id f26dd451307aa00ea9c3985310f506a424a0cc976adda8c46c17eb211c476ff9
Nov 03 01:56:45.443584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:45.443542    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc" event={"ID":"c241cdce-5e2e-4a75-8b72-d628bc68606f","Type":"ContainerStarted","Data":"50999c4f891b0647e8b450cd92192b4fbeb22085cbc192ebd1cef5f2184f7d57"}
Nov 03 01:56:45.443788 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:45.443602    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc" event={"ID":"c241cdce-5e2e-4a75-8b72-d628bc68606f","Type":"ContainerStarted","Data":"f26dd451307aa00ea9c3985310f506a424a0cc976adda8c46c17eb211c476ff9"}
Nov 03 01:56:45.443788 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:45.443761    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"
Nov 03 01:56:45.553658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:45.553597    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc" podStartSLOduration=1.553580808 podStartE2EDuration="1.553580808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:45.551707414 +0000 UTC m=+2284.977313904" watchObservedRunningTime="2026-11-03 01:56:45.553580808 +0000 UTC m=+2284.979187271"
Nov 03 01:56:48.100995 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:48.100954    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:56:48.403516 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:48.403489    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"
Nov 03 01:56:48.410457 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:48.409309    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"
Nov 03 01:56:51.085564 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:51.085526    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"]
Nov 03 01:56:51.096857 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:51.096828    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"
Nov 03 01:56:51.119617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:51.119594    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq\" (UID: \"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"
Nov 03 01:56:51.119753 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:51.119627    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9zdh\" (UniqueName: \"kubernetes.io/projected/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-kube-api-access-d9zdh\") pod \"payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq\" (UID: \"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"
Nov 03 01:56:51.152716 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:51.152666    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"]
Nov 03 01:56:51.258658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:51.239908    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f95773-dockercfg-d6885\""
Nov 03 01:56:51.258658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:51.239977    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq\" (UID: \"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"
Nov 03 01:56:51.258658 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:51.240027    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9zdh\" (UniqueName: \"kubernetes.io/projected/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-kube-api-access-d9zdh\") pod \"payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq\" (UID: \"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"
Nov 03 01:56:51.316459 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:51.312804    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9zdh\" (UniqueName: \"kubernetes.io/projected/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-kube-api-access-d9zdh\") pod \"payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq\" (UID: \"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"
Nov 03 01:56:51.320865 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:51.320839    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f95773\""
Nov 03 01:56:51.329364 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:51.328757    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq\" (UID: \"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"
Nov 03 01:56:51.411201 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:51.411173    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"
Nov 03 01:56:51.818090 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:51.818015    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"]
Nov 03 01:56:51.824482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:56:51.824454    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ca69c3_b0c1_443c_a27f_e3c3c9b45b1d.slice/crio-b3c48ff45af5b7134bb96d7baee0f6300dd444afcf9d2d0792c33b87fea22ec2 WatchSource:0}: Error finding container b3c48ff45af5b7134bb96d7baee0f6300dd444afcf9d2d0792c33b87fea22ec2: Status 404 returned error can't find the container with id b3c48ff45af5b7134bb96d7baee0f6300dd444afcf9d2d0792c33b87fea22ec2
Nov 03 01:56:52.516414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:52.516381    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq" event={"ID":"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d","Type":"ContainerStarted","Data":"e093e526b49ace20ee9491381c66a0420030d4800cd1d13de9a680d656615281"}
Nov 03 01:56:52.516881 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:52.516429    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq" event={"ID":"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d","Type":"ContainerStarted","Data":"b3c48ff45af5b7134bb96d7baee0f6300dd444afcf9d2d0792c33b87fea22ec2"}
Nov 03 01:56:52.516881 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:52.516514    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"
Nov 03 01:56:52.636492 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:52.636441    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq" podStartSLOduration=1.6364272469999999 podStartE2EDuration="1.636427247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:56:52.633809509 +0000 UTC m=+2292.059415973" watchObservedRunningTime="2026-11-03 01:56:52.636427247 +0000 UTC m=+2292.062033710"
Nov 03 01:56:56.453734 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:56.453701    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"
Nov 03 01:56:58.570887 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:58.570857    5508 generic.go:356] "Generic (PLEG): container finished" podID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerID="c5d098335a95cbc91d7edfe1f95d405e5255285c0b215761616242e1d0ae5c39" exitCode=0
Nov 03 01:56:58.571402 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:58.570927    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5" event={"ID":"9762f99e-d6cd-4369-bc2b-e11dbf56a32f","Type":"ContainerDied","Data":"c5d098335a95cbc91d7edfe1f95d405e5255285c0b215761616242e1d0ae5c39"}
Nov 03 01:56:58.571402 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:58.571279    5508 scope.go:122] "RemoveContainer" containerID="c5d098335a95cbc91d7edfe1f95d405e5255285c0b215761616242e1d0ae5c39"
Nov 03 01:56:58.572759 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:58.572734    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:58.572822 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:58.572803    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:58.572862 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:58.572836    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:56:59.582519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:59.582490    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5" event={"ID":"9762f99e-d6cd-4369-bc2b-e11dbf56a32f","Type":"ContainerStarted","Data":"a14d2d85f02c7238787043fc298f857f2bd7dd7054144b0c693befd599b08510"}
Nov 03 01:56:59.805518 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:59.805481    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:56:59.805687 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:59.805667    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 03 01:56:59.805751 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:56:59.805710    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5" podUID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 03 01:57:00.806018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:00.805984    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 03 01:57:00.806427 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:00.806032    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5" podUID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 03 01:57:01.805801 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:01.805757    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 03 01:57:01.806019 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:01.805816    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5" podUID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 03 01:57:02.805951 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:02.805919    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 03 01:57:02.806464 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:02.805967    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5" podUID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 03 01:57:03.525622 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:03.525590    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"
Nov 03 01:57:03.805766 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:03.805672    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Nov 03 01:57:03.805952 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:03.805748    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5" podUID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Nov 03 01:57:04.811286 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:04.811257    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:57:04.811713 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:04.811677    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:57:04.812287 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:04.812266    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 01:57:13.811727 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:57:13.811693    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:57:27.104058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:27.104029    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:57:35.471722 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.471689    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"]
Nov 03 01:57:35.475828 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.475809    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:35.501124 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.501099    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:35.501261 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.501139    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmm2b\" (UniqueName: \"kubernetes.io/projected/ebea57f5-714b-454d-843c-5cc7d546abcd-kube-api-access-lmm2b\") pod \"maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:35.501261 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.501184    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebea57f5-714b-454d-843c-5cc7d546abcd-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:35.501371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.501271    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-metrics-certs\") pod \"maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:35.524215 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.524195    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-e2db44-serving-cert\""
Nov 03 01:57:35.573056 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.573029    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"]
Nov 03 01:57:35.602608 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.602574    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:35.602769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.602624    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmm2b\" (UniqueName: \"kubernetes.io/projected/ebea57f5-714b-454d-843c-5cc7d546abcd-kube-api-access-lmm2b\") pod \"maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:35.602769 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:57:35.602736    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-serving-cert: secret "maas-api-e2e-praxis-b-e2db44-serving-cert" not found
Nov 03 01:57:35.602870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.602800    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebea57f5-714b-454d-843c-5cc7d546abcd-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:35.602870 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:57:35.602821    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-maas-api-tls podName:ebea57f5-714b-454d-843c-5cc7d546abcd nodeName:}" failed. No retries permitted until 2026-11-03 01:57:36.102804871 +0000 UTC m=+2335.528411319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-maas-api-tls") pod "maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk" (UID: "ebea57f5-714b-454d-843c-5cc7d546abcd") : secret "maas-api-e2e-praxis-b-e2db44-serving-cert" not found
Nov 03 01:57:35.602963 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.602949    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-metrics-certs\") pod \"maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:35.603518 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.603500    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebea57f5-714b-454d-843c-5cc7d546abcd-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:35.605424 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.605406    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-metrics-certs\") pod \"maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:35.660382 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:35.660329    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmm2b\" (UniqueName: \"kubernetes.io/projected/ebea57f5-714b-454d-843c-5cc7d546abcd-kube-api-access-lmm2b\") pod \"maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:36.123185 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:36.123154    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:36.125562 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:36.125533    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:36.388964 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:36.388885    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:36.759652 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:36.759625    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"]
Nov 03 01:57:36.768106 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:57:36.768078    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebea57f5_714b_454d_843c_5cc7d546abcd.slice/crio-5781204e0411fe7540656e667cd0c6a65112b55b2affa896a7ebe410eeb24af3 WatchSource:0}: Error finding container 5781204e0411fe7540656e667cd0c6a65112b55b2affa896a7ebe410eeb24af3: Status 404 returned error can't find the container with id 5781204e0411fe7540656e667cd0c6a65112b55b2affa896a7ebe410eeb24af3
Nov 03 01:57:36.835463 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:36.835434    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk" event={"ID":"ebea57f5-714b-454d-843c-5cc7d546abcd","Type":"ContainerStarted","Data":"5781204e0411fe7540656e667cd0c6a65112b55b2affa896a7ebe410eeb24af3"}
Nov 03 01:57:38.850173 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:38.850141    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk" event={"ID":"ebea57f5-714b-454d-843c-5cc7d546abcd","Type":"ContainerStarted","Data":"bd2bd39e59f1b6ac95ac6bb54a4503de594067f2c76546288ba68deb7e827c66"}
Nov 03 01:57:38.850584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:38.850267    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:57:38.961064 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:38.961004    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk" podStartSLOduration=2.962563645 podStartE2EDuration="3.960987316s" totalImagesPullingTime="998.423671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:57:38.960374515 +0000 UTC m=+2338.385980995" watchObservedRunningTime="2026-11-03 01:57:38.960987316 +0000 UTC m=+2338.386593781"
Nov 03 01:57:43.811732 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:57:43.811693    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:57:44.863693 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:57:44.863659    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 01:58:09.686594 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:09.686400    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:58:13.927372 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:58:13.926948    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:58:22.226095 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:22.226058    5508 generic.go:356] "Generic (PLEG): container finished" podID="21141eda-543d-4fa6-9404-8289591a86d6" containerID="20d24d8809523e1585b5c44693111bf0f78d4d726805c142d4cfc3b23b0d17db" exitCode=0
Nov 03 01:58:22.241484 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:22.226124    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" event={"ID":"21141eda-543d-4fa6-9404-8289591a86d6","Type":"ContainerDied","Data":"20d24d8809523e1585b5c44693111bf0f78d4d726805c142d4cfc3b23b0d17db"}
Nov 03 01:58:22.241484 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:22.226569    5508 scope.go:122] "RemoveContainer" containerID="20d24d8809523e1585b5c44693111bf0f78d4d726805c142d4cfc3b23b0d17db"
Nov 03 01:58:22.241484 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:22.228066    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:58:22.241484 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:22.228128    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:58:22.241484 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:22.228158    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:58:23.233663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.233625    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" event={"ID":"21141eda-543d-4fa6-9404-8289591a86d6","Type":"ContainerStarted","Data":"d59d15404f51cd9ec64c3090fef2ce359a10cffcf47fa78e3845f6d700d9047a"}
Nov 03 01:58:23.235178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.235151    5508 generic.go:356] "Generic (PLEG): container finished" podID="cc85e883-f97a-417c-b58a-8535213343c0" containerID="e640d927b58acd93605e7b8f5b596db74f6d33f7847ab8864bc0a1dfd763b4eb" exitCode=0
Nov 03 01:58:23.235318 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.235225    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6" event={"ID":"cc85e883-f97a-417c-b58a-8535213343c0","Type":"ContainerDied","Data":"e640d927b58acd93605e7b8f5b596db74f6d33f7847ab8864bc0a1dfd763b4eb"}
Nov 03 01:58:23.235630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.235611    5508 scope.go:122] "RemoveContainer" containerID="e640d927b58acd93605e7b8f5b596db74f6d33f7847ab8864bc0a1dfd763b4eb"
Nov 03 01:58:23.236867 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.236845    5508 generic.go:356] "Generic (PLEG): container finished" podID="16676061-80e9-435c-923b-e4e9b6d128d0" containerID="f4cb768843505ccc95c1f799d4d650c4dae09efa3e4a1fc5c9d6004938fc9550" exitCode=0
Nov 03 01:58:23.236974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.236892    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs" event={"ID":"16676061-80e9-435c-923b-e4e9b6d128d0","Type":"ContainerDied","Data":"f4cb768843505ccc95c1f799d4d650c4dae09efa3e4a1fc5c9d6004938fc9550"}
Nov 03 01:58:23.237107 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.237082    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:58:23.237176 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.237163    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:58:23.237217 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.237175    5508 scope.go:122] "RemoveContainer" containerID="f4cb768843505ccc95c1f799d4d650c4dae09efa3e4a1fc5c9d6004938fc9550"
Nov 03 01:58:23.237217 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.237196    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:58:23.238588 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.238562    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:58:23.238700 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.238635    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:58:23.238769 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.238704    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:58:23.399370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.396529    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:58:23.399370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.396773    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:58:23.399370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.398967    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:58:23.832557 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.832522    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:58:23.833786 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.833757    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 01:58:23.833948 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:23.833813    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 01:58:24.247373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:24.246763    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6" event={"ID":"cc85e883-f97a-417c-b58a-8535213343c0","Type":"ContainerStarted","Data":"6a284344597eb5cba19515d035dc41167afeb16e432f1869344ca53d638431d3"}
Nov 03 01:58:24.260373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:24.257089    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs" event={"ID":"16676061-80e9-435c-923b-e4e9b6d128d0","Type":"ContainerStarted","Data":"ec63eefc74f9569aca9843bf09cd4075f82d37d1b89d48b8324cbc760261d554"}
Nov 03 01:58:24.839371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:24.836835    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 01:58:24.839371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:24.836965    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 01:58:25.173605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:25.173570    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:58:25.192371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:25.177468    5508 patch_prober.go:29] interesting pod/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 01:58:25.192371 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:25.177539    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6" podUID="cc85e883-f97a-417c-b58a-8535213343c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 01:58:25.219386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:25.217733    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:58:25.219386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:25.217886    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 01:58:25.219386 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:25.217930    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs" podUID="16676061-80e9-435c-923b-e4e9b6d128d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 01:58:25.836374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:25.834765    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 01:58:25.836374 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:25.834827    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 01:58:26.174367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:26.173781    5508 patch_prober.go:29] interesting pod/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 01:58:26.174367 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:26.173861    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6" podUID="cc85e883-f97a-417c-b58a-8535213343c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 01:58:26.216373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:26.215738    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 01:58:26.216373 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:26.215814    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs" podUID="16676061-80e9-435c-923b-e4e9b6d128d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 01:58:26.835472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:26.833798    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 01:58:26.835472 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:26.833874    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 01:58:27.237489 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:27.237451    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 01:58:27.242084 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:27.241439    5508 patch_prober.go:29] interesting pod/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 01:58:27.242799 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:27.242325    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6" podUID="cc85e883-f97a-417c-b58a-8535213343c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 01:58:27.243003 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:27.242038    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs" podUID="16676061-80e9-435c-923b-e4e9b6d128d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 01:58:27.837376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:27.836744    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 01:58:27.837376 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:27.836819    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 01:58:28.173806 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:28.173766    5508 patch_prober.go:29] interesting pod/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 01:58:28.174113 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:28.174083    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6" podUID="cc85e883-f97a-417c-b58a-8535213343c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 01:58:28.222771 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:28.222733    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 01:58:28.223069 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:28.223040    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs" podUID="16676061-80e9-435c-923b-e4e9b6d128d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 01:58:28.832861 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:28.832829    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 01:58:28.833398 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:28.832897    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 01:58:29.175783 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:29.175743    5508 patch_prober.go:29] interesting pod/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 01:58:29.176075 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:29.176043    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6" podUID="cc85e883-f97a-417c-b58a-8535213343c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 01:58:29.212902 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:29.212863    5508 patch_prober.go:29] interesting pod/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Nov 03 01:58:29.213182 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:29.213157    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs" podUID="16676061-80e9-435c-923b-e4e9b6d128d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Nov 03 01:58:29.838288 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:29.838253    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:58:29.839417 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:29.839397    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 01:58:30.178651 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:30.178617    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:58:30.179038 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:30.179006    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:58:30.179974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:30.179953    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 01:58:30.218242 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:30.218213    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:58:30.218610 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:30.218588    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:58:30.219451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:30.219426    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 01:58:41.163946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:41.163923    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:58:41.200295 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:58:41.200274    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 01:58:43.811029 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:58:43.810994    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:59:00.934362 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:00.934319    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"]
Nov 03 01:59:00.990328 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:00.990293    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"]
Nov 03 01:59:05.486234 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.486194    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"]
Nov 03 01:59:05.739914 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.739837    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:59:05.859596 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.859566    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"]
Nov 03 01:59:05.864021 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.863998    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:05.916299 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.916268    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-db583d-openshift-default-dockercfg-tkbrd\""
Nov 03 01:59:05.926175 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.926146    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"]
Nov 03 01:59:05.935082 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.935045    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7" podUID="805c5346-d73c-4d36-9f28-8b030b76e4fe" containerName="payload-processing" containerID="cri-o://d6cfd015007c588c3d5df1b700b708d402af92ca60f9dd4c0e4b5eaae2c7d0c3" gracePeriod=25
Nov 03 01:59:05.961396 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.961371    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-socket\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:05.961508 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.961402    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-envoy\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:05.961561 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.961504    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-token\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:05.961625 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.961609    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-credential-socket\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:05.961664 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.961642    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-data\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:05.961717 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.961706    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8psh\" (UniqueName: \"kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-kube-api-access-w8psh\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:05.961763 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.961752    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23107beb-9060-480d-9f1b-6ba175e4d4ef-istiod-ca-cert\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:05.961800 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.961781    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-certs\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:05.961854 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.961843    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-podinfo\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:05.991826 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:05.991718    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p" podUID="7ea32c0f-32ac-40b3-9af9-e19ba93a708b" containerName="payload-pre-processing" containerID="cri-o://25dab1846bc443cd2e5251d805248dec5aebf5506d60ad852264f07b4ff4b1f9" gracePeriod=25
Nov 03 01:59:06.077580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.070137    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-token\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.077580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.070219    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-credential-socket\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.077580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.070255    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-data\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.077580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.070302    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8psh\" (UniqueName: \"kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-kube-api-access-w8psh\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.077580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.070380    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23107beb-9060-480d-9f1b-6ba175e4d4ef-istiod-ca-cert\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.077580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.070425    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-certs\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.077580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.070469    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-podinfo\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.077580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.070533    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-socket\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.077580 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.070556    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-envoy\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.089315 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.089105    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-credential-socket\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.089519 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.089434    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-data\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.090413 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.090371    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23107beb-9060-480d-9f1b-6ba175e4d4ef-istiod-ca-cert\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.094679 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.090640    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-certs\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.099120 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.099057    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-podinfo\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.101253 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.099935    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-socket\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.102850 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.102769    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-envoy\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.146151 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.146113    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-token\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.147630 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.147610    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8psh\" (UniqueName: \"kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-kube-api-access-w8psh\") pod \"e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") " pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.185380 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.184803    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:06.289303 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.287453    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"
Nov 03 01:59:06.343429 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.342337    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"
Nov 03 01:59:06.386404 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.386364    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805c5346-d73c-4d36-9f28-8b030b76e4fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805c5346-d73c-4d36-9f28-8b030b76e4fe-plugins-config-volume\") pod \"805c5346-d73c-4d36-9f28-8b030b76e4fe\" (UID: \"805c5346-d73c-4d36-9f28-8b030b76e4fe\") "
Nov 03 01:59:06.386583 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.386521    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805c5346-d73c-4d36-9f28-8b030b76e4fe-kube-api-access-zzbc8\" (UniqueName: \"kubernetes.io/projected/805c5346-d73c-4d36-9f28-8b030b76e4fe-kube-api-access-zzbc8\") pod \"805c5346-d73c-4d36-9f28-8b030b76e4fe\" (UID: \"805c5346-d73c-4d36-9f28-8b030b76e4fe\") "
Nov 03 01:59:06.386657 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.386633    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-plugins-config-volume\") pod \"7ea32c0f-32ac-40b3-9af9-e19ba93a708b\" (UID: \"7ea32c0f-32ac-40b3-9af9-e19ba93a708b\") "
Nov 03 01:59:06.386761 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.386743    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-kube-api-access-jwdfx\" (UniqueName: \"kubernetes.io/projected/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-kube-api-access-jwdfx\") pod \"7ea32c0f-32ac-40b3-9af9-e19ba93a708b\" (UID: \"7ea32c0f-32ac-40b3-9af9-e19ba93a708b\") "
Nov 03 01:59:06.386843 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.386753    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805c5346-d73c-4d36-9f28-8b030b76e4fe-plugins-config-volume" pod "805c5346-d73c-4d36-9f28-8b030b76e4fe" (UID: "805c5346-d73c-4d36-9f28-8b030b76e4fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:59:06.387019 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.386998    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-plugins-config-volume" pod "7ea32c0f-32ac-40b3-9af9-e19ba93a708b" (UID: "7ea32c0f-32ac-40b3-9af9-e19ba93a708b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:59:06.387264 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.387248    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805c5346-d73c-4d36-9f28-8b030b76e4fe-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:06.387334 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.387271    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:06.389416 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.389394    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805c5346-d73c-4d36-9f28-8b030b76e4fe-kube-api-access-zzbc8" pod "805c5346-d73c-4d36-9f28-8b030b76e4fe" (UID: "805c5346-d73c-4d36-9f28-8b030b76e4fe"). InnerVolumeSpecName "kube-api-access-zzbc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:59:06.389540 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.389423    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-kube-api-access-jwdfx" pod "7ea32c0f-32ac-40b3-9af9-e19ba93a708b" (UID: "7ea32c0f-32ac-40b3-9af9-e19ba93a708b"). InnerVolumeSpecName "kube-api-access-jwdfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:59:06.488230 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.488198    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzbc8\" (UniqueName: \"kubernetes.io/projected/805c5346-d73c-4d36-9f28-8b030b76e4fe-kube-api-access-zzbc8\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:06.488230 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.488229    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwdfx\" (UniqueName: \"kubernetes.io/projected/7ea32c0f-32ac-40b3-9af9-e19ba93a708b-kube-api-access-jwdfx\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:06.625190 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.625096    5508 generic.go:356] "Generic (PLEG): container finished" podID="805c5346-d73c-4d36-9f28-8b030b76e4fe" containerID="d6cfd015007c588c3d5df1b700b708d402af92ca60f9dd4c0e4b5eaae2c7d0c3" exitCode=0
Nov 03 01:59:06.625394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.625201    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7" event={"ID":"805c5346-d73c-4d36-9f28-8b030b76e4fe","Type":"ContainerDied","Data":"d6cfd015007c588c3d5df1b700b708d402af92ca60f9dd4c0e4b5eaae2c7d0c3"}
Nov 03 01:59:06.625394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.625264    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"
Nov 03 01:59:06.625523 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.625287    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7" event={"ID":"805c5346-d73c-4d36-9f28-8b030b76e4fe","Type":"ContainerDied","Data":"e95562dd05a5a459f5cf0876e5593da461bd2871a67eef9c1ba9cf27f1bb35b9"}
Nov 03 01:59:06.625523 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.625443    5508 scope.go:122] "RemoveContainer" containerID="d6cfd015007c588c3d5df1b700b708d402af92ca60f9dd4c0e4b5eaae2c7d0c3"
Nov 03 01:59:06.626912 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.626888    5508 generic.go:356] "Generic (PLEG): container finished" podID="7ea32c0f-32ac-40b3-9af9-e19ba93a708b" containerID="25dab1846bc443cd2e5251d805248dec5aebf5506d60ad852264f07b4ff4b1f9" exitCode=0
Nov 03 01:59:06.627031 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.626955    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"
Nov 03 01:59:06.627031 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.626969    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p" event={"ID":"7ea32c0f-32ac-40b3-9af9-e19ba93a708b","Type":"ContainerDied","Data":"25dab1846bc443cd2e5251d805248dec5aebf5506d60ad852264f07b4ff4b1f9"}
Nov 03 01:59:06.627031 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.627021    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p" event={"ID":"7ea32c0f-32ac-40b3-9af9-e19ba93a708b","Type":"ContainerDied","Data":"e0214949658893e1e93bb41a822980ccae35e727a9bcfef68a1c0ef1537ccb60"}
Nov 03 01:59:06.639229 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.639213    5508 scope.go:122] "RemoveContainer" containerID="d6cfd015007c588c3d5df1b700b708d402af92ca60f9dd4c0e4b5eaae2c7d0c3"
Nov 03 01:59:06.639638 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:59:06.639609    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6cfd015007c588c3d5df1b700b708d402af92ca60f9dd4c0e4b5eaae2c7d0c3\": container with ID starting with d6cfd015007c588c3d5df1b700b708d402af92ca60f9dd4c0e4b5eaae2c7d0c3 not found: ID does not exist" containerID="d6cfd015007c588c3d5df1b700b708d402af92ca60f9dd4c0e4b5eaae2c7d0c3"
Nov 03 01:59:06.639750 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.639648    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cfd015007c588c3d5df1b700b708d402af92ca60f9dd4c0e4b5eaae2c7d0c3"} err="failed to get container status \"d6cfd015007c588c3d5df1b700b708d402af92ca60f9dd4c0e4b5eaae2c7d0c3\": rpc error: code = NotFound desc = could not find container \"d6cfd015007c588c3d5df1b700b708d402af92ca60f9dd4c0e4b5eaae2c7d0c3\": container with ID starting with d6cfd015007c588c3d5df1b700b708d402af92ca60f9dd4c0e4b5eaae2c7d0c3 not found: ID does not exist"
Nov 03 01:59:06.639750 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.639678    5508 scope.go:122] "RemoveContainer" containerID="25dab1846bc443cd2e5251d805248dec5aebf5506d60ad852264f07b4ff4b1f9"
Nov 03 01:59:06.649296 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.649254    5508 scope.go:122] "RemoveContainer" containerID="25dab1846bc443cd2e5251d805248dec5aebf5506d60ad852264f07b4ff4b1f9"
Nov 03 01:59:06.649627 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:59:06.649583    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25dab1846bc443cd2e5251d805248dec5aebf5506d60ad852264f07b4ff4b1f9\": container with ID starting with 25dab1846bc443cd2e5251d805248dec5aebf5506d60ad852264f07b4ff4b1f9 not found: ID does not exist" containerID="25dab1846bc443cd2e5251d805248dec5aebf5506d60ad852264f07b4ff4b1f9"
Nov 03 01:59:06.649731 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.649625    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25dab1846bc443cd2e5251d805248dec5aebf5506d60ad852264f07b4ff4b1f9"} err="failed to get container status \"25dab1846bc443cd2e5251d805248dec5aebf5506d60ad852264f07b4ff4b1f9\": rpc error: code = NotFound desc = could not find container \"25dab1846bc443cd2e5251d805248dec5aebf5506d60ad852264f07b4ff4b1f9\": container with ID starting with 25dab1846bc443cd2e5251d805248dec5aebf5506d60ad852264f07b4ff4b1f9 not found: ID does not exist"
Nov 03 01:59:06.678080 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.678057    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"]
Nov 03 01:59:06.685398 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:59:06.685328    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23107beb_9060_480d_9f1b_6ba175e4d4ef.slice/crio-4cc9eafe3a23dc60be407d7b3859e34a6b3ab04bd3510e3a9f08bb4812ed1936 WatchSource:0}: Error finding container 4cc9eafe3a23dc60be407d7b3859e34a6b3ab04bd3510e3a9f08bb4812ed1936: Status 404 returned error can't find the container with id 4cc9eafe3a23dc60be407d7b3859e34a6b3ab04bd3510e3a9f08bb4812ed1936
Nov 03 01:59:06.687740 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.687713    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:59:06.687814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.687777    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:59:06.687814 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.687806    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 01:59:06.784456 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.784411    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"]
Nov 03 01:59:06.788178 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.788151    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7"]
Nov 03 01:59:06.918369 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.911434    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"]
Nov 03 01:59:06.919235 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:06.919202    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p"]
Nov 03 01:59:07.272541 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:07.272464    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d9025-7dc8d47559-pcb7p" podUID="7ea32c0f-32ac-40b3-9af9-e19ba93a708b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.196:50052\" within 1s: context deadline exceeded"
Nov 03 01:59:07.272690 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:59:07.272547    5508 logging.go:55] [core] [Channel #4304 SubChannel #4305]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.196:50052", ServerName: "10.133.0.196:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.196:50052: operation was canceled"
Nov 03 01:59:07.273482 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:07.273458    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d9025-7b76d954d9-hvhv7" podUID="805c5346-d73c-4d36-9f28-8b030b76e4fe" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.197:50052\" within 1s: context deadline exceeded"
Nov 03 01:59:07.273559 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:59:07.273519    5508 logging.go:55] [core] [Channel #4306 SubChannel #4307]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.197:50052", ServerName: "10.133.0.197:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.197:50052: operation was canceled"
Nov 03 01:59:07.637590 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:07.637504    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5" event={"ID":"23107beb-9060-480d-9f1b-6ba175e4d4ef","Type":"ContainerStarted","Data":"e63bb301ce12690968e0a956756241bfced2c4e100d55aa61091b1901b3aaeb0"}
Nov 03 01:59:07.637590 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:07.637571    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5" event={"ID":"23107beb-9060-480d-9f1b-6ba175e4d4ef","Type":"ContainerStarted","Data":"4cc9eafe3a23dc60be407d7b3859e34a6b3ab04bd3510e3a9f08bb4812ed1936"}
Nov 03 01:59:07.753587 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:07.753544    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5" podStartSLOduration=2.753531907 podStartE2EDuration="2.753531907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:07.750897029 +0000 UTC m=+2427.176503493" watchObservedRunningTime="2026-11-03 01:59:07.753531907 +0000 UTC m=+2427.179138371"
Nov 03 01:59:08.186014 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:08.185985    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:08.190958 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:08.190937    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:08.643584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:08.643508    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:08.644329 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:08.644312    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 01:59:08.693435 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:08.693406    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea32c0f-32ac-40b3-9af9-e19ba93a708b" path="/var/lib/kubelet/pods/7ea32c0f-32ac-40b3-9af9-e19ba93a708b/volumes"
Nov 03 01:59:08.693791 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:08.693776    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805c5346-d73c-4d36-9f28-8b030b76e4fe" path="/var/lib/kubelet/pods/805c5346-d73c-4d36-9f28-8b030b76e4fe/volumes"
Nov 03 01:59:09.498040 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.498002    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"]
Nov 03 01:59:09.498559 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.498535    5508 cpu_manager.go:414] "containerMap: removing container" podUID="805c5346-d73c-4d36-9f28-8b030b76e4fe" containerName="payload-processing"
Nov 03 01:59:09.498559 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.498552    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805c5346-d73c-4d36-9f28-8b030b76e4fe" containerName="payload-processing"
Nov 03 01:59:09.498559 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.498562    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7ea32c0f-32ac-40b3-9af9-e19ba93a708b" containerName="payload-pre-processing"
Nov 03 01:59:09.498559 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.498567    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea32c0f-32ac-40b3-9af9-e19ba93a708b" containerName="payload-pre-processing"
Nov 03 01:59:09.498865 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.498659    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="805c5346-d73c-4d36-9f28-8b030b76e4fe" containerName="payload-processing"
Nov 03 01:59:09.498865 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.498668    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea32c0f-32ac-40b3-9af9-e19ba93a708b" containerName="payload-pre-processing"
Nov 03 01:59:09.503464 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.503443    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"
Nov 03 01:59:09.552387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.552337    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e0907d\""
Nov 03 01:59:09.609310 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.609195    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e0907d-dockercfg-fjqjs\""
Nov 03 01:59:09.621442 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.621411    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"]
Nov 03 01:59:09.644626 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.644588    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2zp7\" (UniqueName: \"kubernetes.io/projected/867f1e43-883e-44ba-83ed-a3e19a552af0-kube-api-access-r2zp7\") pod \"payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld\" (UID: \"867f1e43-883e-44ba-83ed-a3e19a552af0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"
Nov 03 01:59:09.645107 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.644732    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867f1e43-883e-44ba-83ed-a3e19a552af0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld\" (UID: \"867f1e43-883e-44ba-83ed-a3e19a552af0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"
Nov 03 01:59:09.747398 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.745890    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2zp7\" (UniqueName: \"kubernetes.io/projected/867f1e43-883e-44ba-83ed-a3e19a552af0-kube-api-access-r2zp7\") pod \"payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld\" (UID: \"867f1e43-883e-44ba-83ed-a3e19a552af0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"
Nov 03 01:59:09.747398 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.746051    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867f1e43-883e-44ba-83ed-a3e19a552af0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld\" (UID: \"867f1e43-883e-44ba-83ed-a3e19a552af0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"
Nov 03 01:59:09.747398 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.746988    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867f1e43-883e-44ba-83ed-a3e19a552af0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld\" (UID: \"867f1e43-883e-44ba-83ed-a3e19a552af0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"
Nov 03 01:59:09.802890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.802822    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2zp7\" (UniqueName: \"kubernetes.io/projected/867f1e43-883e-44ba-83ed-a3e19a552af0-kube-api-access-r2zp7\") pod \"payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld\" (UID: \"867f1e43-883e-44ba-83ed-a3e19a552af0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"
Nov 03 01:59:09.816889 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:09.816868    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"
Nov 03 01:59:10.195125 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.195095    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"]
Nov 03 01:59:10.202482 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:59:10.202456    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867f1e43_883e_44ba_83ed_a3e19a552af0.slice/crio-0d5ba3c3a54b4c9235b7b19ceac33ed0d35cd3a37536081f3416f7a29103471f WatchSource:0}: Error finding container 0d5ba3c3a54b4c9235b7b19ceac33ed0d35cd3a37536081f3416f7a29103471f: Status 404 returned error can't find the container with id 0d5ba3c3a54b4c9235b7b19ceac33ed0d35cd3a37536081f3416f7a29103471f
Nov 03 01:59:10.487414 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.487298    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq" podUID="59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d" containerName="payload-pre-processing" containerID="cri-o://e093e526b49ace20ee9491381c66a0420030d4800cd1d13de9a680d656615281" gracePeriod=25
Nov 03 01:59:10.691236 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.691198    5508 generic.go:356] "Generic (PLEG): container finished" podID="59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d" containerID="e093e526b49ace20ee9491381c66a0420030d4800cd1d13de9a680d656615281" exitCode=0
Nov 03 01:59:10.699870 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.699838    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq" event={"ID":"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d","Type":"ContainerDied","Data":"e093e526b49ace20ee9491381c66a0420030d4800cd1d13de9a680d656615281"}
Nov 03 01:59:10.700018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.699899    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"
Nov 03 01:59:10.700018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.699924    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld" event={"ID":"867f1e43-883e-44ba-83ed-a3e19a552af0","Type":"ContainerStarted","Data":"b9ada2034edfe7deb5f4793fc9e6780fec55e35d8349c0bd0bc2bcfc3c68b83b"}
Nov 03 01:59:10.700018 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.699945    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld" event={"ID":"867f1e43-883e-44ba-83ed-a3e19a552af0","Type":"ContainerStarted","Data":"0d5ba3c3a54b4c9235b7b19ceac33ed0d35cd3a37536081f3416f7a29103471f"}
Nov 03 01:59:10.730663 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.730642    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"
Nov 03 01:59:10.804884 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.804804    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld" podStartSLOduration=1.804791674 podStartE2EDuration="1.804791674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:10.803003583 +0000 UTC m=+2430.228610065" watchObservedRunningTime="2026-11-03 01:59:10.804791674 +0000 UTC m=+2430.230398138"
Nov 03 01:59:10.859590 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.859568    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-kube-api-access-d9zdh\" (UniqueName: \"kubernetes.io/projected/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-kube-api-access-d9zdh\") pod \"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d\" (UID: \"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d\") "
Nov 03 01:59:10.859699 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.859683    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-plugins-config-volume\") pod \"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d\" (UID: \"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d\") "
Nov 03 01:59:10.860090 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.860066    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-plugins-config-volume" pod "59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d" (UID: "59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:59:10.861756 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.861733    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-kube-api-access-d9zdh" pod "59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d" (UID: "59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d"). InnerVolumeSpecName "kube-api-access-d9zdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:59:10.961257 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.961221    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:10.961257 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:10.961254    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9zdh\" (UniqueName: \"kubernetes.io/projected/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d-kube-api-access-d9zdh\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:11.703550 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:11.703524    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"
Nov 03 01:59:11.703947 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:11.703567    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq" event={"ID":"59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d","Type":"ContainerDied","Data":"b3c48ff45af5b7134bb96d7baee0f6300dd444afcf9d2d0792c33b87fea22ec2"}
Nov 03 01:59:11.703947 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:11.703648    5508 scope.go:122] "RemoveContainer" containerID="e093e526b49ace20ee9491381c66a0420030d4800cd1d13de9a680d656615281"
Nov 03 01:59:11.865293 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:11.865262    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"]
Nov 03 01:59:11.869324 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:11.869294    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-68f64b4c67-x4xpq"]
Nov 03 01:59:12.693604 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:12.693569    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d" path="/var/lib/kubelet/pods/59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d/volumes"
Nov 03 01:59:13.812455 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:59:13.812420    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:59:15.023840 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:15.023807    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv_f9f60d09-e74d-432e-84ef-b74af7c18842/payload-processing/0.log"
Nov 03 01:59:15.811605 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:15.811581    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv_56d79e96-1752-414a-acc6-5b80b18381ab/payload-processing/0.log"
Nov 03 01:59:16.057248 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.057182    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv_56d79e96-1752-414a-acc6-5b80b18381ab/payload-processing/0.log"
Nov 03 01:59:16.491207 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.491174    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"]
Nov 03 01:59:16.491749 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.491735    5508 cpu_manager.go:414] "containerMap: removing container" podUID="59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d" containerName="payload-pre-processing"
Nov 03 01:59:16.491811 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.491750    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d" containerName="payload-pre-processing"
Nov 03 01:59:16.491880 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.491870    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ca69c3-b0c1-443c-a27f-e3c3c9b45b1d" containerName="payload-pre-processing"
Nov 03 01:59:16.496890 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.496870    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"
Nov 03 01:59:16.552398 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.549685    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-db583d-dockercfg-8jklj\""
Nov 03 01:59:16.611149 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.611089    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-db583d\""
Nov 03 01:59:16.628470 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.628437    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"]
Nov 03 01:59:16.642058 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.630165    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58766af5-ad89-4a97-aeef-5cb49c27ba2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm\" (UID: \"58766af5-ad89-4a97-aeef-5cb49c27ba2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"
Nov 03 01:59:16.642498 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.642470    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndmws\" (UniqueName: \"kubernetes.io/projected/58766af5-ad89-4a97-aeef-5cb49c27ba2f-kube-api-access-ndmws\") pod \"payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm\" (UID: \"58766af5-ad89-4a97-aeef-5cb49c27ba2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"
Nov 03 01:59:16.744206 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.744108    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndmws\" (UniqueName: \"kubernetes.io/projected/58766af5-ad89-4a97-aeef-5cb49c27ba2f-kube-api-access-ndmws\") pod \"payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm\" (UID: \"58766af5-ad89-4a97-aeef-5cb49c27ba2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"
Nov 03 01:59:16.744385 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.744247    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58766af5-ad89-4a97-aeef-5cb49c27ba2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm\" (UID: \"58766af5-ad89-4a97-aeef-5cb49c27ba2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"
Nov 03 01:59:16.744942 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.744922    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58766af5-ad89-4a97-aeef-5cb49c27ba2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm\" (UID: \"58766af5-ad89-4a97-aeef-5cb49c27ba2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"
Nov 03 01:59:16.805461 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.805431    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndmws\" (UniqueName: \"kubernetes.io/projected/58766af5-ad89-4a97-aeef-5cb49c27ba2f-kube-api-access-ndmws\") pod \"payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm\" (UID: \"58766af5-ad89-4a97-aeef-5cb49c27ba2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"
Nov 03 01:59:16.809974 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:16.809949    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"
Nov 03 01:59:17.199451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:17.199419    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"]
Nov 03 01:59:17.206145 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:59:17.206117    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58766af5_ad89_4a97_aeef_5cb49c27ba2f.slice/crio-83867521fa50ae8b19551186c8af77f740d1eaed347585fc5de1046562f8504c WatchSource:0}: Error finding container 83867521fa50ae8b19551186c8af77f740d1eaed347585fc5de1046562f8504c: Status 404 returned error can't find the container with id 83867521fa50ae8b19551186c8af77f740d1eaed347585fc5de1046562f8504c
Nov 03 01:59:17.747749 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:17.747714    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm" event={"ID":"58766af5-ad89-4a97-aeef-5cb49c27ba2f","Type":"ContainerStarted","Data":"d70fe736a58b6425a00c75b0225e5541d4fb656edcccb0689a29838a6df20045"}
Nov 03 01:59:17.747946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:17.747774    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm" event={"ID":"58766af5-ad89-4a97-aeef-5cb49c27ba2f","Type":"ContainerStarted","Data":"83867521fa50ae8b19551186c8af77f740d1eaed347585fc5de1046562f8504c"}
Nov 03 01:59:17.747946 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:17.747799    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"
Nov 03 01:59:17.856192 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:17.856138    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm" podStartSLOduration=1.8561234 podStartE2EDuration="1.8561234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:59:17.854712194 +0000 UTC m=+2437.280318677" watchObservedRunningTime="2026-11-03 01:59:17.8561234 +0000 UTC m=+2437.281729901"
Nov 03 01:59:21.706767 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:21.706736    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"
Nov 03 01:59:28.757238 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:28.757201    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"
Nov 03 01:59:32.992562 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:32.992520    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"]
Nov 03 01:59:32.993451 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:32.992793    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg" podUID="624b2b40-ab2d-4d9a-8899-b65b04b6d59c" containerName="maas-api" containerID="cri-o://07500cf82d484f8d20f5e5cdbe663f74d52a0d66263d2c9324bf2d655925492f" gracePeriod=30
Nov 03 01:59:33.300537 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.300499    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:59:33.347761 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.347124    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-kube-api-access-jt8q4\" (UniqueName: \"kubernetes.io/projected/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-kube-api-access-jt8q4\") pod \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") "
Nov 03 01:59:33.369584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.364971    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-metrics-certs\") pod \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") "
Nov 03 01:59:33.369584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.365136    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-openshift-service-ca\") pod \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") "
Nov 03 01:59:33.369584 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.365181    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-maas-api-tls\") pod \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\" (UID: \"624b2b40-ab2d-4d9a-8899-b65b04b6d59c\") "
Nov 03 01:59:33.392599 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.377517    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-openshift-service-ca" pod "624b2b40-ab2d-4d9a-8899-b65b04b6d59c" (UID: "624b2b40-ab2d-4d9a-8899-b65b04b6d59c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:59:33.394738 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.394687    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-kube-api-access-jt8q4" pod "624b2b40-ab2d-4d9a-8899-b65b04b6d59c" (UID: "624b2b40-ab2d-4d9a-8899-b65b04b6d59c"). InnerVolumeSpecName "kube-api-access-jt8q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:59:33.402909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.402873    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-maas-api-tls" pod "624b2b40-ab2d-4d9a-8899-b65b04b6d59c" (UID: "624b2b40-ab2d-4d9a-8899-b65b04b6d59c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:59:33.404073 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.404042    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-metrics-certs" pod "624b2b40-ab2d-4d9a-8899-b65b04b6d59c" (UID: "624b2b40-ab2d-4d9a-8899-b65b04b6d59c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:59:33.473394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.471085    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:33.473394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.471124    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:33.473394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.471140    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt8q4\" (UniqueName: \"kubernetes.io/projected/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-kube-api-access-jt8q4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:33.473394 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.471155    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/624b2b40-ab2d-4d9a-8899-b65b04b6d59c-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:33.870166 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.870101    5508 generic.go:356] "Generic (PLEG): container finished" podID="624b2b40-ab2d-4d9a-8899-b65b04b6d59c" containerID="07500cf82d484f8d20f5e5cdbe663f74d52a0d66263d2c9324bf2d655925492f" exitCode=0
Nov 03 01:59:33.870425 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.870367    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg" event={"ID":"624b2b40-ab2d-4d9a-8899-b65b04b6d59c","Type":"ContainerDied","Data":"07500cf82d484f8d20f5e5cdbe663f74d52a0d66263d2c9324bf2d655925492f"}
Nov 03 01:59:33.870503 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.870477    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg" event={"ID":"624b2b40-ab2d-4d9a-8899-b65b04b6d59c","Type":"ContainerDied","Data":"618d839d13518c9416dbada4aa02434763f0cc478acfcf4f62566d39668ee1f8"}
Nov 03 01:59:33.870558 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.870522    5508 scope.go:122] "RemoveContainer" containerID="07500cf82d484f8d20f5e5cdbe663f74d52a0d66263d2c9324bf2d655925492f"
Nov 03 01:59:33.870659 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.870627    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"
Nov 03 01:59:33.890408 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.887868    5508 scope.go:122] "RemoveContainer" containerID="07500cf82d484f8d20f5e5cdbe663f74d52a0d66263d2c9324bf2d655925492f"
Nov 03 01:59:33.890408 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:59:33.888334    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07500cf82d484f8d20f5e5cdbe663f74d52a0d66263d2c9324bf2d655925492f\": container with ID starting with 07500cf82d484f8d20f5e5cdbe663f74d52a0d66263d2c9324bf2d655925492f not found: ID does not exist" containerID="07500cf82d484f8d20f5e5cdbe663f74d52a0d66263d2c9324bf2d655925492f"
Nov 03 01:59:33.890408 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:33.888405    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07500cf82d484f8d20f5e5cdbe663f74d52a0d66263d2c9324bf2d655925492f"} err="failed to get container status \"07500cf82d484f8d20f5e5cdbe663f74d52a0d66263d2c9324bf2d655925492f\": rpc error: code = NotFound desc = could not find container \"07500cf82d484f8d20f5e5cdbe663f74d52a0d66263d2c9324bf2d655925492f\": container with ID starting with 07500cf82d484f8d20f5e5cdbe663f74d52a0d66263d2c9324bf2d655925492f not found: ID does not exist"
Nov 03 01:59:34.031041 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:34.031006    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"]
Nov 03 01:59:34.075928 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:34.075891    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d9025-9b8845d97-hkvhg"]
Nov 03 01:59:34.692840 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:34.692805    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="624b2b40-ab2d-4d9a-8899-b65b04b6d59c" path="/var/lib/kubelet/pods/624b2b40-ab2d-4d9a-8899-b65b04b6d59c/volumes"
Nov 03 01:59:39.218585 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:39.218555    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"]
Nov 03 01:59:39.218981 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:39.218762    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8" podUID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerName="istio-proxy" containerID="cri-o://165aab56cfe7c46f2fe10558a030a43e20f486f08dfacf844c41c465ea521bb4" gracePeriod=30
Nov 03 01:59:43.811039 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:59:43.811005    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 01:59:44.481415 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.481337    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:59:44.616677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.616647    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-envoy\") pod \"3aa8e146-3961-4111-9712-e1323fdf7a08\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") "
Nov 03 01:59:44.616677 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.616678    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-podinfo\") pod \"3aa8e146-3961-4111-9712-e1323fdf7a08\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") "
Nov 03 01:59:44.616903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.616717    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-socket\") pod \"3aa8e146-3961-4111-9712-e1323fdf7a08\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") "
Nov 03 01:59:44.616903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.616762    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa8e146-3961-4111-9712-e1323fdf7a08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa8e146-3961-4111-9712-e1323fdf7a08-istiod-ca-cert\") pod \"3aa8e146-3961-4111-9712-e1323fdf7a08\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") "
Nov 03 01:59:44.616903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.616821    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-kube-api-access-gxgzh\" (UniqueName: \"kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-kube-api-access-gxgzh\") pod \"3aa8e146-3961-4111-9712-e1323fdf7a08\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") "
Nov 03 01:59:44.616903 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.616865    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-data\") pod \"3aa8e146-3961-4111-9712-e1323fdf7a08\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") "
Nov 03 01:59:44.617188 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.616957    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-certs\") pod \"3aa8e146-3961-4111-9712-e1323fdf7a08\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") "
Nov 03 01:59:44.617188 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.617035    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-credential-socket\") pod \"3aa8e146-3961-4111-9712-e1323fdf7a08\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") "
Nov 03 01:59:44.617188 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.617077    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-socket" pod "3aa8e146-3961-4111-9712-e1323fdf7a08" (UID: "3aa8e146-3961-4111-9712-e1323fdf7a08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:59:44.617188 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.617081    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-token\" (UniqueName: \"kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-token\") pod \"3aa8e146-3961-4111-9712-e1323fdf7a08\" (UID: \"3aa8e146-3961-4111-9712-e1323fdf7a08\") "
Nov 03 01:59:44.617420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.617205    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-data" pod "3aa8e146-3961-4111-9712-e1323fdf7a08" (UID: "3aa8e146-3961-4111-9712-e1323fdf7a08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:59:44.617420 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.617257    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa8e146-3961-4111-9712-e1323fdf7a08-istiod-ca-cert" pod "3aa8e146-3961-4111-9712-e1323fdf7a08" (UID: "3aa8e146-3961-4111-9712-e1323fdf7a08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:59:44.617515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.617418    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-certs" pod "3aa8e146-3961-4111-9712-e1323fdf7a08" (UID: "3aa8e146-3961-4111-9712-e1323fdf7a08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:59:44.617515 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.617440    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-credential-socket" pod "3aa8e146-3961-4111-9712-e1323fdf7a08" (UID: "3aa8e146-3961-4111-9712-e1323fdf7a08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:59:44.617676 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.617655    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:44.617721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.617683    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aa8e146-3961-4111-9712-e1323fdf7a08-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:44.617721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.617699    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:44.617721 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.617712    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:44.617817 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.617727    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:44.619453 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.619424    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-kube-api-access-gxgzh" pod "3aa8e146-3961-4111-9712-e1323fdf7a08" (UID: "3aa8e146-3961-4111-9712-e1323fdf7a08"). InnerVolumeSpecName "kube-api-access-gxgzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:59:44.619582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.619466    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-podinfo" pod "3aa8e146-3961-4111-9712-e1323fdf7a08" (UID: "3aa8e146-3961-4111-9712-e1323fdf7a08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:59:44.619582 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.619551    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-token" pod "3aa8e146-3961-4111-9712-e1323fdf7a08" (UID: "3aa8e146-3961-4111-9712-e1323fdf7a08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:59:44.620915 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.620893    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-envoy" pod "3aa8e146-3961-4111-9712-e1323fdf7a08" (UID: "3aa8e146-3961-4111-9712-e1323fdf7a08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:59:44.719319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.719294    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:44.719319 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.719319    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:44.719485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.719329    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aa8e146-3961-4111-9712-e1323fdf7a08-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:44.719485 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.719338    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxgzh\" (UniqueName: \"kubernetes.io/projected/3aa8e146-3961-4111-9712-e1323fdf7a08-kube-api-access-gxgzh\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:44.829387 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.829291    5508 scope.go:122] "RemoveContainer" containerID="5dc32cabaac82471c5a28b2d74e4cddfd8febbaf07c86cee2081d8fb29578fcc"
Nov 03 01:59:44.942297 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.942262    5508 generic.go:356] "Generic (PLEG): container finished" podID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerID="165aab56cfe7c46f2fe10558a030a43e20f486f08dfacf844c41c465ea521bb4" exitCode=0
Nov 03 01:59:44.942437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.942333    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"
Nov 03 01:59:44.942437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.942341    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8" event={"ID":"3aa8e146-3961-4111-9712-e1323fdf7a08","Type":"ContainerDied","Data":"165aab56cfe7c46f2fe10558a030a43e20f486f08dfacf844c41c465ea521bb4"}
Nov 03 01:59:44.942437 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.942420    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8" event={"ID":"3aa8e146-3961-4111-9712-e1323fdf7a08","Type":"ContainerDied","Data":"98a66d403475022d25c4a68ce536a044d3cfeb5c3d59161b9f982205358bc139"}
Nov 03 01:59:44.942570 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.942449    5508 scope.go:122] "RemoveContainer" containerID="165aab56cfe7c46f2fe10558a030a43e20f486f08dfacf844c41c465ea521bb4"
Nov 03 01:59:44.952419 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.952399    5508 scope.go:122] "RemoveContainer" containerID="ca902b641aaf8e5779c64a63d6db5f3814de24e9941cf528112d99006b101b8c"
Nov 03 01:59:44.961091 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.961076    5508 scope.go:122] "RemoveContainer" containerID="165aab56cfe7c46f2fe10558a030a43e20f486f08dfacf844c41c465ea521bb4"
Nov 03 01:59:44.961417 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:59:44.961396    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165aab56cfe7c46f2fe10558a030a43e20f486f08dfacf844c41c465ea521bb4\": container with ID starting with 165aab56cfe7c46f2fe10558a030a43e20f486f08dfacf844c41c465ea521bb4 not found: ID does not exist" containerID="165aab56cfe7c46f2fe10558a030a43e20f486f08dfacf844c41c465ea521bb4"
Nov 03 01:59:44.961486 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.961426    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165aab56cfe7c46f2fe10558a030a43e20f486f08dfacf844c41c465ea521bb4"} err="failed to get container status \"165aab56cfe7c46f2fe10558a030a43e20f486f08dfacf844c41c465ea521bb4\": rpc error: code = NotFound desc = could not find container \"165aab56cfe7c46f2fe10558a030a43e20f486f08dfacf844c41c465ea521bb4\": container with ID starting with 165aab56cfe7c46f2fe10558a030a43e20f486f08dfacf844c41c465ea521bb4 not found: ID does not exist"
Nov 03 01:59:44.961486 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.961444    5508 scope.go:122] "RemoveContainer" containerID="ca902b641aaf8e5779c64a63d6db5f3814de24e9941cf528112d99006b101b8c"
Nov 03 01:59:44.961688 ip-10-0-140-255 kubenswrapper[5508]: E1103 01:59:44.961674    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca902b641aaf8e5779c64a63d6db5f3814de24e9941cf528112d99006b101b8c\": container with ID starting with ca902b641aaf8e5779c64a63d6db5f3814de24e9941cf528112d99006b101b8c not found: ID does not exist" containerID="ca902b641aaf8e5779c64a63d6db5f3814de24e9941cf528112d99006b101b8c"
Nov 03 01:59:44.961744 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:44.961694    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca902b641aaf8e5779c64a63d6db5f3814de24e9941cf528112d99006b101b8c"} err="failed to get container status \"ca902b641aaf8e5779c64a63d6db5f3814de24e9941cf528112d99006b101b8c\": rpc error: code = NotFound desc = could not find container \"ca902b641aaf8e5779c64a63d6db5f3814de24e9941cf528112d99006b101b8c\": container with ID starting with ca902b641aaf8e5779c64a63d6db5f3814de24e9941cf528112d99006b101b8c not found: ID does not exist"
Nov 03 01:59:45.114049 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:45.113969    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"]
Nov 03 01:59:45.117813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:45.117788    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d9025-openshift-default-694c486b69-8nns8"]
Nov 03 01:59:46.695921 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:46.695882    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa8e146-3961-4111-9712-e1323fdf7a08" path="/var/lib/kubelet/pods/3aa8e146-3961-4111-9712-e1323fdf7a08/volumes"
Nov 03 01:59:51.477692 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:51.477661    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 01:59:52.831711 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:52.831682    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"]
Nov 03 01:59:57.832284 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:57.832234    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc" podUID="c241cdce-5e2e-4a75-8b72-d628bc68606f" containerName="payload-processing" containerID="cri-o://50999c4f891b0647e8b450cd92192b4fbeb22085cbc192ebd1cef5f2184f7d57" gracePeriod=25
Nov 03 01:59:58.027669 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.027638    5508 generic.go:356] "Generic (PLEG): container finished" podID="c241cdce-5e2e-4a75-8b72-d628bc68606f" containerID="50999c4f891b0647e8b450cd92192b4fbeb22085cbc192ebd1cef5f2184f7d57" exitCode=0
Nov 03 01:59:58.027813 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.027713    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc" event={"ID":"c241cdce-5e2e-4a75-8b72-d628bc68606f","Type":"ContainerDied","Data":"50999c4f891b0647e8b450cd92192b4fbeb22085cbc192ebd1cef5f2184f7d57"}
Nov 03 01:59:58.112787 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.112755    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"
Nov 03 01:59:58.170952 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.170918    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c241cdce-5e2e-4a75-8b72-d628bc68606f-kube-api-access-9vpdg\" (UniqueName: \"kubernetes.io/projected/c241cdce-5e2e-4a75-8b72-d628bc68606f-kube-api-access-9vpdg\") pod \"c241cdce-5e2e-4a75-8b72-d628bc68606f\" (UID: \"c241cdce-5e2e-4a75-8b72-d628bc68606f\") "
Nov 03 01:59:58.171133 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.171075    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c241cdce-5e2e-4a75-8b72-d628bc68606f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c241cdce-5e2e-4a75-8b72-d628bc68606f-plugins-config-volume\") pod \"c241cdce-5e2e-4a75-8b72-d628bc68606f\" (UID: \"c241cdce-5e2e-4a75-8b72-d628bc68606f\") "
Nov 03 01:59:58.171533 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.171507    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c241cdce-5e2e-4a75-8b72-d628bc68606f-plugins-config-volume" pod "c241cdce-5e2e-4a75-8b72-d628bc68606f" (UID: "c241cdce-5e2e-4a75-8b72-d628bc68606f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:59:58.172985 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.172963    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c241cdce-5e2e-4a75-8b72-d628bc68606f-kube-api-access-9vpdg" pod "c241cdce-5e2e-4a75-8b72-d628bc68606f" (UID: "c241cdce-5e2e-4a75-8b72-d628bc68606f"). InnerVolumeSpecName "kube-api-access-9vpdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:59:58.272201 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.272165    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c241cdce-5e2e-4a75-8b72-d628bc68606f-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:58.272201 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.272195    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vpdg\" (UniqueName: \"kubernetes.io/projected/c241cdce-5e2e-4a75-8b72-d628bc68606f-kube-api-access-9vpdg\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 01:59:58.529705 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.529672    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"]
Nov 03 01:59:58.530167 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.530154    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c241cdce-5e2e-4a75-8b72-d628bc68606f" containerName="payload-processing"
Nov 03 01:59:58.530216 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.530168    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c241cdce-5e2e-4a75-8b72-d628bc68606f" containerName="payload-processing"
Nov 03 01:59:58.530216 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.530186    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerName="istio-proxy"
Nov 03 01:59:58.530216 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.530191    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerName="istio-proxy"
Nov 03 01:59:58.530216 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.530214    5508 cpu_manager.go:414] "containerMap: removing container" podUID="624b2b40-ab2d-4d9a-8899-b65b04b6d59c" containerName="maas-api"
Nov 03 01:59:58.530370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.530219    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624b2b40-ab2d-4d9a-8899-b65b04b6d59c" containerName="maas-api"
Nov 03 01:59:58.530370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.530235    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerName="istio-proxy"
Nov 03 01:59:58.530370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.530239    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerName="istio-proxy"
Nov 03 01:59:58.530370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.530314    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerName="istio-proxy"
Nov 03 01:59:58.530370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.530329    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="624b2b40-ab2d-4d9a-8899-b65b04b6d59c" containerName="maas-api"
Nov 03 01:59:58.530370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.530342    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa8e146-3961-4111-9712-e1323fdf7a08" containerName="istio-proxy"
Nov 03 01:59:58.530370 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.530360    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c241cdce-5e2e-4a75-8b72-d628bc68606f" containerName="payload-processing"
Nov 03 01:59:58.533933 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.533909    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"
Nov 03 01:59:58.575136 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.575102    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa188f32-tnfs4\" (UID: \"f6628479-581a-41b5-8aac-8a1cf766f7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"
Nov 03 01:59:58.575314 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.575169    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qqh4\" (UniqueName: \"kubernetes.io/projected/f6628479-581a-41b5-8aac-8a1cf766f7f1-kube-api-access-5qqh4\") pod \"maas-api-revoke-sub-aa188f32-tnfs4\" (UID: \"f6628479-581a-41b5-8aac-8a1cf766f7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"
Nov 03 01:59:58.575408 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.575320    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa188f32-tnfs4\" (UID: \"f6628479-581a-41b5-8aac-8a1cf766f7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"
Nov 03 01:59:58.582709 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.582685    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:59:58.582909 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.582889    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 01:59:58.632221 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.632183    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"]
Nov 03 01:59:58.676192 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.676165    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa188f32-tnfs4\" (UID: \"f6628479-581a-41b5-8aac-8a1cf766f7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"
Nov 03 01:59:58.676390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.676234    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa188f32-tnfs4\" (UID: \"f6628479-581a-41b5-8aac-8a1cf766f7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"
Nov 03 01:59:58.676390 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.676278    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qqh4\" (UniqueName: \"kubernetes.io/projected/f6628479-581a-41b5-8aac-8a1cf766f7f1-kube-api-access-5qqh4\") pod \"maas-api-revoke-sub-aa188f32-tnfs4\" (UID: \"f6628479-581a-41b5-8aac-8a1cf766f7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"
Nov 03 01:59:58.676985 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.676965    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa188f32-tnfs4\" (UID: \"f6628479-581a-41b5-8aac-8a1cf766f7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"
Nov 03 01:59:58.678623 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.678598    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa188f32-tnfs4\" (UID: \"f6628479-581a-41b5-8aac-8a1cf766f7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"
Nov 03 01:59:58.739562 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.739497    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qqh4\" (UniqueName: \"kubernetes.io/projected/f6628479-581a-41b5-8aac-8a1cf766f7f1-kube-api-access-5qqh4\") pod \"maas-api-revoke-sub-aa188f32-tnfs4\" (UID: \"f6628479-581a-41b5-8aac-8a1cf766f7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"
Nov 03 01:59:58.847467 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:58.847395    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"
Nov 03 01:59:59.039498 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:59.039472    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"
Nov 03 01:59:59.039669 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:59.039465    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc" event={"ID":"c241cdce-5e2e-4a75-8b72-d628bc68606f","Type":"ContainerDied","Data":"f26dd451307aa00ea9c3985310f506a424a0cc976adda8c46c17eb211c476ff9"}
Nov 03 01:59:59.039669 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:59.039599    5508 scope.go:122] "RemoveContainer" containerID="50999c4f891b0647e8b450cd92192b4fbeb22085cbc192ebd1cef5f2184f7d57"
Nov 03 01:59:59.193617 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:59.193582    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"]
Nov 03 01:59:59.196759 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:59.196735    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e2db44-84c87b6fc5-z8mvc"]
Nov 03 01:59:59.228588 ip-10-0-140-255 kubenswrapper[5508]: I1103 01:59:59.228567    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"]
Nov 03 01:59:59.281631 ip-10-0-140-255 kubenswrapper[5508]: W1103 01:59:59.281604    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6628479_581a_41b5_8aac_8a1cf766f7f1.slice/crio-d03a0236f4983fcd7f1cb23fc778e5516c2bf9cfcee08c83da820356de0811c0 WatchSource:0}: Error finding container d03a0236f4983fcd7f1cb23fc778e5516c2bf9cfcee08c83da820356de0811c0: Status 404 returned error can't find the container with id d03a0236f4983fcd7f1cb23fc778e5516c2bf9cfcee08c83da820356de0811c0
Nov 03 02:00:00.048497 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.048456    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4" event={"ID":"f6628479-581a-41b5-8aac-8a1cf766f7f1","Type":"ContainerStarted","Data":"eff5a106f93938e332da14618c2430132630418fd4cb904e7101baa6fd9b33a0"}
Nov 03 02:00:00.048955 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.048519    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4" event={"ID":"f6628479-581a-41b5-8aac-8a1cf766f7f1","Type":"ContainerStarted","Data":"d03a0236f4983fcd7f1cb23fc778e5516c2bf9cfcee08c83da820356de0811c0"}
Nov 03 02:00:00.157684 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.157656    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"]
Nov 03 02:00:00.164496 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.164474    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"]
Nov 03 02:00:00.164642 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.164619    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"
Nov 03 02:00:00.168271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.168255    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"
Nov 03 02:00:00.199549 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.199529    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r\" (UID: \"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"
Nov 03 02:00:00.199683 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.199561    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d426eea-4c34-46ea-98b6-3235a0cbacff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd\" (UID: \"0d426eea-4c34-46ea-98b6-3235a0cbacff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"
Nov 03 02:00:00.199683 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.199587    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcv95\" (UniqueName: \"kubernetes.io/projected/0d426eea-4c34-46ea-98b6-3235a0cbacff-kube-api-access-kcv95\") pod \"maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd\" (UID: \"0d426eea-4c34-46ea-98b6-3235a0cbacff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"
Nov 03 02:00:00.199683 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.199642    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566bb\" (UniqueName: \"kubernetes.io/projected/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-kube-api-access-566bb\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r\" (UID: \"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"
Nov 03 02:00:00.204709 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.204687    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"]
Nov 03 02:00:00.206460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.206415    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4" podStartSLOduration=2.206399822 podStartE2EDuration="2.206399822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:00.158315516 +0000 UTC m=+2479.583921982" watchObservedRunningTime="2026-11-03 02:00:00.206399822 +0000 UTC m=+2479.632006288"
Nov 03 02:00:00.209218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.209198    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"
Nov 03 02:00:00.210814 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.210795    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"]
Nov 03 02:00:00.214662 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.214643    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"]
Nov 03 02:00:00.214769 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.214743    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"
Nov 03 02:00:00.218246 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.218229    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"]
Nov 03 02:00:00.218379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.218333    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"
Nov 03 02:00:00.221880 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.221861    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"]
Nov 03 02:00:00.221990 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.221936    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"
Nov 03 02:00:00.260726 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.260698    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"]
Nov 03 02:00:00.273915 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.273890    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"]
Nov 03 02:00:00.301198 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.301134    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm\" (UID: \"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"
Nov 03 02:00:00.301198 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.301167    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212d6a7a-fe8d-492f-b509-7044ba779aeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f\" (UID: \"212d6a7a-fe8d-492f-b509-7044ba779aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"
Nov 03 02:00:00.301394 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.301274    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnpn9\" (UniqueName: \"kubernetes.io/projected/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-kube-api-access-wnpn9\") pod \"maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm\" (UID: \"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"
Nov 03 02:00:00.301394 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.301338    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqxqt\" (UniqueName: \"kubernetes.io/projected/fbfcd995-8f45-423f-b566-350c1270a59d-kube-api-access-dqxqt\") pod \"maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh\" (UID: \"fbfcd995-8f45-423f-b566-350c1270a59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"
Nov 03 02:00:00.301513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.301399    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48146943-2c15-4c2c-be1f-b7d340ec713c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2\" (UID: \"48146943-2c15-4c2c-be1f-b7d340ec713c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"
Nov 03 02:00:00.301513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.301442    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r\" (UID: \"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"
Nov 03 02:00:00.301593 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.301537    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d426eea-4c34-46ea-98b6-3235a0cbacff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd\" (UID: \"0d426eea-4c34-46ea-98b6-3235a0cbacff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"
Nov 03 02:00:00.301593 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.301562    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcv95\" (UniqueName: \"kubernetes.io/projected/0d426eea-4c34-46ea-98b6-3235a0cbacff-kube-api-access-kcv95\") pod \"maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd\" (UID: \"0d426eea-4c34-46ea-98b6-3235a0cbacff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"
Nov 03 02:00:00.301686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.301594    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfcd995-8f45-423f-b566-350c1270a59d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh\" (UID: \"fbfcd995-8f45-423f-b566-350c1270a59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"
Nov 03 02:00:00.301686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.301627    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566bb\" (UniqueName: \"kubernetes.io/projected/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-kube-api-access-566bb\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r\" (UID: \"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"
Nov 03 02:00:00.301686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.301678    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5r7f\" (UniqueName: \"kubernetes.io/projected/48146943-2c15-4c2c-be1f-b7d340ec713c-kube-api-access-q5r7f\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2\" (UID: \"48146943-2c15-4c2c-be1f-b7d340ec713c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"
Nov 03 02:00:00.301847 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.301754    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79spv\" (UniqueName: \"kubernetes.io/projected/212d6a7a-fe8d-492f-b509-7044ba779aeb-kube-api-access-79spv\") pod \"maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f\" (UID: \"212d6a7a-fe8d-492f-b509-7044ba779aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"
Nov 03 02:00:00.302476 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.302147    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r\" (UID: \"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"
Nov 03 02:00:00.302476 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.302215    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d426eea-4c34-46ea-98b6-3235a0cbacff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd\" (UID: \"0d426eea-4c34-46ea-98b6-3235a0cbacff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"
Nov 03 02:00:00.318312 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.318287    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"]
Nov 03 02:00:00.322718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.322692    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"]
Nov 03 02:00:00.330372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.326203    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"]
Nov 03 02:00:00.359144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.359108    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566bb\" (UniqueName: \"kubernetes.io/projected/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-kube-api-access-566bb\") pod \"maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r\" (UID: \"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"
Nov 03 02:00:00.359258 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.359150    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcv95\" (UniqueName: \"kubernetes.io/projected/0d426eea-4c34-46ea-98b6-3235a0cbacff-kube-api-access-kcv95\") pod \"maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd\" (UID: \"0d426eea-4c34-46ea-98b6-3235a0cbacff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"
Nov 03 02:00:00.403029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.403000    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48146943-2c15-4c2c-be1f-b7d340ec713c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2\" (UID: \"48146943-2c15-4c2c-be1f-b7d340ec713c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"
Nov 03 02:00:00.403159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.403064    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfcd995-8f45-423f-b566-350c1270a59d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh\" (UID: \"fbfcd995-8f45-423f-b566-350c1270a59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"
Nov 03 02:00:00.403230 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.403210    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5r7f\" (UniqueName: \"kubernetes.io/projected/48146943-2c15-4c2c-be1f-b7d340ec713c-kube-api-access-q5r7f\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2\" (UID: \"48146943-2c15-4c2c-be1f-b7d340ec713c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"
Nov 03 02:00:00.403304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.403290    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79spv\" (UniqueName: \"kubernetes.io/projected/212d6a7a-fe8d-492f-b509-7044ba779aeb-kube-api-access-79spv\") pod \"maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f\" (UID: \"212d6a7a-fe8d-492f-b509-7044ba779aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"
Nov 03 02:00:00.403379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.403319    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm\" (UID: \"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"
Nov 03 02:00:00.403379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.403342    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212d6a7a-fe8d-492f-b509-7044ba779aeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f\" (UID: \"212d6a7a-fe8d-492f-b509-7044ba779aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"
Nov 03 02:00:00.403494 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.403415    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnpn9\" (UniqueName: \"kubernetes.io/projected/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-kube-api-access-wnpn9\") pod \"maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm\" (UID: \"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"
Nov 03 02:00:00.403494 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.403480    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqxqt\" (UniqueName: \"kubernetes.io/projected/fbfcd995-8f45-423f-b566-350c1270a59d-kube-api-access-dqxqt\") pod \"maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh\" (UID: \"fbfcd995-8f45-423f-b566-350c1270a59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"
Nov 03 02:00:00.403751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.403731    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48146943-2c15-4c2c-be1f-b7d340ec713c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2\" (UID: \"48146943-2c15-4c2c-be1f-b7d340ec713c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"
Nov 03 02:00:00.404033 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.404015    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212d6a7a-fe8d-492f-b509-7044ba779aeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f\" (UID: \"212d6a7a-fe8d-492f-b509-7044ba779aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"
Nov 03 02:00:00.404074 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.404057    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm\" (UID: \"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"
Nov 03 02:00:00.404074 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.404064    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfcd995-8f45-423f-b566-350c1270a59d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh\" (UID: \"fbfcd995-8f45-423f-b566-350c1270a59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"
Nov 03 02:00:00.460858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.460830    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79spv\" (UniqueName: \"kubernetes.io/projected/212d6a7a-fe8d-492f-b509-7044ba779aeb-kube-api-access-79spv\") pod \"maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f\" (UID: \"212d6a7a-fe8d-492f-b509-7044ba779aeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"
Nov 03 02:00:00.460972 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.460905    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnpn9\" (UniqueName: \"kubernetes.io/projected/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-kube-api-access-wnpn9\") pod \"maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm\" (UID: \"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"
Nov 03 02:00:00.461149 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.461126    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqxqt\" (UniqueName: \"kubernetes.io/projected/fbfcd995-8f45-423f-b566-350c1270a59d-kube-api-access-dqxqt\") pod \"maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh\" (UID: \"fbfcd995-8f45-423f-b566-350c1270a59d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"
Nov 03 02:00:00.461328 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.461314    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5r7f\" (UniqueName: \"kubernetes.io/projected/48146943-2c15-4c2c-be1f-b7d340ec713c-kube-api-access-q5r7f\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2\" (UID: \"48146943-2c15-4c2c-be1f-b7d340ec713c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"
Nov 03 02:00:00.530070 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.530044    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"
Nov 03 02:00:00.536308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.536284    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"
Nov 03 02:00:00.543428 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.543408    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"
Nov 03 02:00:00.551958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.551909    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"
Nov 03 02:00:00.570567 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.570535    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"
Nov 03 02:00:00.592377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.592284    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"
Nov 03 02:00:00.709110 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.709071    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c241cdce-5e2e-4a75-8b72-d628bc68606f" path="/var/lib/kubelet/pods/c241cdce-5e2e-4a75-8b72-d628bc68606f/volumes"
Nov 03 02:00:00.920699 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.920672    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"]
Nov 03 02:00:00.927323 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:00:00.927299    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4328ed5a_ab0c_48e8_a9bd_846e3e63ec1f.slice/crio-8caaebe2783e8ab77f625c377be3ca526379f63f5d551a410711e0135b13c179 WatchSource:0}: Error finding container 8caaebe2783e8ab77f625c377be3ca526379f63f5d551a410711e0135b13c179: Status 404 returned error can't find the container with id 8caaebe2783e8ab77f625c377be3ca526379f63f5d551a410711e0135b13c179
Nov 03 02:00:00.971186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.971161    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"]
Nov 03 02:00:00.973935 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:00.973913    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"]
Nov 03 02:00:00.978132 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:00:00.978106    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d426eea_4c34_46ea_98b6_3235a0cbacff.slice/crio-fea7cbd486a76f6ca8463478408fbe0d24fc7c1a1fabe8d5ce15a01a5b2ae028 WatchSource:0}: Error finding container fea7cbd486a76f6ca8463478408fbe0d24fc7c1a1fabe8d5ce15a01a5b2ae028: Status 404 returned error can't find the container with id fea7cbd486a76f6ca8463478408fbe0d24fc7c1a1fabe8d5ce15a01a5b2ae028
Nov 03 02:00:00.980714 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:00:00.980634    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfcd995_8f45_423f_b566_350c1270a59d.slice/crio-3cd12b7f485c3164bb223abbe10436db836975a267b536d670225ce148188b56 WatchSource:0}: Error finding container 3cd12b7f485c3164bb223abbe10436db836975a267b536d670225ce148188b56: Status 404 returned error can't find the container with id 3cd12b7f485c3164bb223abbe10436db836975a267b536d670225ce148188b56
Nov 03 02:00:01.024581 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:01.024499    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"]
Nov 03 02:00:01.053757 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:01.053486    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"]
Nov 03 02:00:01.059130 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:01.059105    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"]
Nov 03 02:00:01.078719 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:00:01.076780    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212d6a7a_fe8d_492f_b509_7044ba779aeb.slice/crio-624039b3e47ef276f8c77359a7d6edcbf6ba5070abd76b6779378157b4766b91 WatchSource:0}: Error finding container 624039b3e47ef276f8c77359a7d6edcbf6ba5070abd76b6779378157b4766b91: Status 404 returned error can't find the container with id 624039b3e47ef276f8c77359a7d6edcbf6ba5070abd76b6779378157b4766b91
Nov 03 02:00:01.094815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:01.094755    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh" event={"ID":"fbfcd995-8f45-423f-b566-350c1270a59d","Type":"ContainerStarted","Data":"3cd12b7f485c3164bb223abbe10436db836975a267b536d670225ce148188b56"}
Nov 03 02:00:01.096204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:01.096177    5508 generic.go:356] "Generic (PLEG): container finished" podID="f6628479-581a-41b5-8aac-8a1cf766f7f1" containerID="eff5a106f93938e332da14618c2430132630418fd4cb904e7101baa6fd9b33a0" exitCode=0
Nov 03 02:00:01.096277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:01.096261    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4" event={"ID":"f6628479-581a-41b5-8aac-8a1cf766f7f1","Type":"ContainerDied","Data":"eff5a106f93938e332da14618c2430132630418fd4cb904e7101baa6fd9b33a0"}
Nov 03 02:00:01.097912 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:01.097887    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r" event={"ID":"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f","Type":"ContainerStarted","Data":"8caaebe2783e8ab77f625c377be3ca526379f63f5d551a410711e0135b13c179"}
Nov 03 02:00:01.099297 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:01.099278    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd" event={"ID":"0d426eea-4c34-46ea-98b6-3235a0cbacff","Type":"ContainerStarted","Data":"fea7cbd486a76f6ca8463478408fbe0d24fc7c1a1fabe8d5ce15a01a5b2ae028"}
Nov 03 02:00:02.106903 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.106875    5508 generic.go:356] "Generic (PLEG): container finished" podID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" containerID="54d68cef3ae4ca67894c0c821f96e345adcd066c1850368aa4bb31bcdac9cad6" exitCode=60
Nov 03 02:00:02.107308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.106960    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm" event={"ID":"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c","Type":"ContainerDied","Data":"54d68cef3ae4ca67894c0c821f96e345adcd066c1850368aa4bb31bcdac9cad6"}
Nov 03 02:00:02.107308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.107018    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm" event={"ID":"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c","Type":"ContainerStarted","Data":"069fc03e6678225f57f07af9b051e501b272a40b8711d872580a71fbb8523cc3"}
Nov 03 02:00:02.107308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.107232    5508 scope.go:122] "RemoveContainer" containerID="54d68cef3ae4ca67894c0c821f96e345adcd066c1850368aa4bb31bcdac9cad6"
Nov 03 02:00:02.108946 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.108854    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd" event={"ID":"0d426eea-4c34-46ea-98b6-3235a0cbacff","Type":"ContainerStarted","Data":"1a334835398d43749a8e44b9e22e5ee677678fd1619e83654c80b202d7e78cc0"}
Nov 03 02:00:02.110583 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.110563    5508 generic.go:356] "Generic (PLEG): container finished" podID="212d6a7a-fe8d-492f-b509-7044ba779aeb" containerID="637dc8fc15a84f7455c57d2d26d55ae45e46ee9820e9246922e8ed62182b3048" exitCode=60
Nov 03 02:00:02.110635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.110590    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f" event={"ID":"212d6a7a-fe8d-492f-b509-7044ba779aeb","Type":"ContainerDied","Data":"637dc8fc15a84f7455c57d2d26d55ae45e46ee9820e9246922e8ed62182b3048"}
Nov 03 02:00:02.110635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.110624    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f" event={"ID":"212d6a7a-fe8d-492f-b509-7044ba779aeb","Type":"ContainerStarted","Data":"624039b3e47ef276f8c77359a7d6edcbf6ba5070abd76b6779378157b4766b91"}
Nov 03 02:00:02.110842 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.110817    5508 scope.go:122] "RemoveContainer" containerID="637dc8fc15a84f7455c57d2d26d55ae45e46ee9820e9246922e8ed62182b3048"
Nov 03 02:00:02.112379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.112341    5508 generic.go:356] "Generic (PLEG): container finished" podID="48146943-2c15-4c2c-be1f-b7d340ec713c" containerID="12f26bbf8e3692114b32ed3a61001f8de0fc9e20a628d110c50af89800dd29e2" exitCode=60
Nov 03 02:00:02.112470 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.112387    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2" event={"ID":"48146943-2c15-4c2c-be1f-b7d340ec713c","Type":"ContainerDied","Data":"12f26bbf8e3692114b32ed3a61001f8de0fc9e20a628d110c50af89800dd29e2"}
Nov 03 02:00:02.112470 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.112428    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2" event={"ID":"48146943-2c15-4c2c-be1f-b7d340ec713c","Type":"ContainerStarted","Data":"a4fbbecc6ede02cc53cb8411a2bed260c892ecf9e01ab4814d4de85e3b80a6b6"}
Nov 03 02:00:02.112614 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.112602    5508 scope.go:122] "RemoveContainer" containerID="12f26bbf8e3692114b32ed3a61001f8de0fc9e20a628d110c50af89800dd29e2"
Nov 03 02:00:02.114022 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.114005    5508 generic.go:356] "Generic (PLEG): container finished" podID="fbfcd995-8f45-423f-b566-350c1270a59d" containerID="128cba637ca309c2e46a15c7331022aa005e127f372a32943cb2341f58903942" exitCode=60
Nov 03 02:00:02.114118 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.114085    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh" event={"ID":"fbfcd995-8f45-423f-b566-350c1270a59d","Type":"ContainerDied","Data":"128cba637ca309c2e46a15c7331022aa005e127f372a32943cb2341f58903942"}
Nov 03 02:00:02.114239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.114223    5508 scope.go:122] "RemoveContainer" containerID="128cba637ca309c2e46a15c7331022aa005e127f372a32943cb2341f58903942"
Nov 03 02:00:02.115606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.115585    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r" event={"ID":"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f","Type":"ContainerStarted","Data":"0e2cf339e4f99c20fb9ae775b42cabbe91633f4cc9530ca0bc4680282afd5637"}
Nov 03 02:00:02.345803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.345776    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"
Nov 03 02:00:02.477982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.477951    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-xks-ca\") pod \"f6628479-581a-41b5-8aac-8a1cf766f7f1\" (UID: \"f6628479-581a-41b5-8aac-8a1cf766f7f1\") "
Nov 03 02:00:02.478134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.478005    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-openshift-ca\") pod \"f6628479-581a-41b5-8aac-8a1cf766f7f1\" (UID: \"f6628479-581a-41b5-8aac-8a1cf766f7f1\") "
Nov 03 02:00:02.478134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.478060    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6628479-581a-41b5-8aac-8a1cf766f7f1-kube-api-access-5qqh4\" (UniqueName: \"kubernetes.io/projected/f6628479-581a-41b5-8aac-8a1cf766f7f1-kube-api-access-5qqh4\") pod \"f6628479-581a-41b5-8aac-8a1cf766f7f1\" (UID: \"f6628479-581a-41b5-8aac-8a1cf766f7f1\") "
Nov 03 02:00:02.478423 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.478403    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-openshift-ca" pod "f6628479-581a-41b5-8aac-8a1cf766f7f1" (UID: "f6628479-581a-41b5-8aac-8a1cf766f7f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:00:02.478552 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.478535    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:02.550292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.550205    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-xks-ca" pod "f6628479-581a-41b5-8aac-8a1cf766f7f1" (UID: "f6628479-581a-41b5-8aac-8a1cf766f7f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:00:02.550437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.550333    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6628479-581a-41b5-8aac-8a1cf766f7f1-kube-api-access-5qqh4" pod "f6628479-581a-41b5-8aac-8a1cf766f7f1" (UID: "f6628479-581a-41b5-8aac-8a1cf766f7f1"). InnerVolumeSpecName "kube-api-access-5qqh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:00:02.579534 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.579512    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6628479-581a-41b5-8aac-8a1cf766f7f1-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:02.579636 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.579535    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qqh4\" (UniqueName: \"kubernetes.io/projected/f6628479-581a-41b5-8aac-8a1cf766f7f1-kube-api-access-5qqh4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:02.649450 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.649395    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r" podStartSLOduration=2.649376223 podStartE2EDuration="2.649376223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:02.646455911 +0000 UTC m=+2482.072062385" watchObservedRunningTime="2026-11-03 02:00:02.649376223 +0000 UTC m=+2482.074982686"
Nov 03 02:00:02.753926 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:02.753869    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd" podStartSLOduration=2.753854068 podStartE2EDuration="2.753854068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:02.753384967 +0000 UTC m=+2482.178991429" watchObservedRunningTime="2026-11-03 02:00:02.753854068 +0000 UTC m=+2482.179460532"
Nov 03 02:00:03.124521 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.124440    5508 generic.go:356] "Generic (PLEG): container finished" podID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" containerID="0e2cf339e4f99c20fb9ae775b42cabbe91633f4cc9530ca0bc4680282afd5637" exitCode=60
Nov 03 02:00:03.125000 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.124514    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r" event={"ID":"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f","Type":"ContainerDied","Data":"0e2cf339e4f99c20fb9ae775b42cabbe91633f4cc9530ca0bc4680282afd5637"}
Nov 03 02:00:03.125000 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.124795    5508 scope.go:122] "RemoveContainer" containerID="0e2cf339e4f99c20fb9ae775b42cabbe91633f4cc9530ca0bc4680282afd5637"
Nov 03 02:00:03.126562 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.126536    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm" event={"ID":"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c","Type":"ContainerStarted","Data":"c754cacf64dd27758359c8cefcf6ebffa4538feeb85d476c1b19c927e152cb19"}
Nov 03 02:00:03.128128 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.128111    5508 generic.go:356] "Generic (PLEG): container finished" podID="0d426eea-4c34-46ea-98b6-3235a0cbacff" containerID="1a334835398d43749a8e44b9e22e5ee677678fd1619e83654c80b202d7e78cc0" exitCode=60
Nov 03 02:00:03.128204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.128186    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd" event={"ID":"0d426eea-4c34-46ea-98b6-3235a0cbacff","Type":"ContainerDied","Data":"1a334835398d43749a8e44b9e22e5ee677678fd1619e83654c80b202d7e78cc0"}
Nov 03 02:00:03.128380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.128339    5508 scope.go:122] "RemoveContainer" containerID="1a334835398d43749a8e44b9e22e5ee677678fd1619e83654c80b202d7e78cc0"
Nov 03 02:00:03.130112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.130089    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f" event={"ID":"212d6a7a-fe8d-492f-b509-7044ba779aeb","Type":"ContainerStarted","Data":"1c47d63c787fe98e082530d2f909fa4dc2da6925306a4669b11fb61474383b2e"}
Nov 03 02:00:03.131812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.131794    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2" event={"ID":"48146943-2c15-4c2c-be1f-b7d340ec713c","Type":"ContainerStarted","Data":"4ab679cb3dc66f4f1b313dbef1517c85a60b661aeb2036bb9159f4b62c5c2cb5"}
Nov 03 02:00:03.133600 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.133578    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh" event={"ID":"fbfcd995-8f45-423f-b566-350c1270a59d","Type":"ContainerStarted","Data":"67eb6707eca38ec03cf2396e0837d9c7759f6df7d99605dbae51d5ae8291451b"}
Nov 03 02:00:03.134705 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.134683    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4" event={"ID":"f6628479-581a-41b5-8aac-8a1cf766f7f1","Type":"ContainerDied","Data":"d03a0236f4983fcd7f1cb23fc778e5516c2bf9cfcee08c83da820356de0811c0"}
Nov 03 02:00:03.134782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.134719    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d03a0236f4983fcd7f1cb23fc778e5516c2bf9cfcee08c83da820356de0811c0"
Nov 03 02:00:03.134782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.134726    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"
Nov 03 02:00:03.341235 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.341187    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f" podStartSLOduration=3.34117486 podStartE2EDuration="3.34117486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:03.338632838 +0000 UTC m=+2482.764239302" watchObservedRunningTime="2026-11-03 02:00:03.34117486 +0000 UTC m=+2482.766781371"
Nov 03 02:00:03.450091 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.450018    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2" podStartSLOduration=3.449999606 podStartE2EDuration="3.449999606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:03.447945705 +0000 UTC m=+2482.873552169" watchObservedRunningTime="2026-11-03 02:00:03.449999606 +0000 UTC m=+2482.875606072"
Nov 03 02:00:03.717221 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.717126    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm" podStartSLOduration=3.717112239 podStartE2EDuration="3.717112239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:03.714266433 +0000 UTC m=+2483.139872904" watchObservedRunningTime="2026-11-03 02:00:03.717112239 +0000 UTC m=+2483.142718702"
Nov 03 02:00:03.823610 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:03.823552    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh" podStartSLOduration=3.823535348 podStartE2EDuration="3.823535348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:03.82116146 +0000 UTC m=+2483.246767925" watchObservedRunningTime="2026-11-03 02:00:03.823535348 +0000 UTC m=+2483.249141813"
Nov 03 02:00:04.145595 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.145558    5508 generic.go:356] "Generic (PLEG): container finished" podID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" containerID="c754cacf64dd27758359c8cefcf6ebffa4538feeb85d476c1b19c927e152cb19" exitCode=60
Nov 03 02:00:04.146165 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.145601    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm" event={"ID":"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c","Type":"ContainerDied","Data":"c754cacf64dd27758359c8cefcf6ebffa4538feeb85d476c1b19c927e152cb19"}
Nov 03 02:00:04.146165 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.145693    5508 scope.go:122] "RemoveContainer" containerID="54d68cef3ae4ca67894c0c821f96e345adcd066c1850368aa4bb31bcdac9cad6"
Nov 03 02:00:04.146165 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.145952    5508 scope.go:122] "RemoveContainer" containerID="c754cacf64dd27758359c8cefcf6ebffa4538feeb85d476c1b19c927e152cb19"
Nov 03 02:00:04.146307 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:04.146263    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm_odh-ai-gateway-infra(fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm" podUID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c"
Nov 03 02:00:04.147817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.147798    5508 generic.go:356] "Generic (PLEG): container finished" podID="0d426eea-4c34-46ea-98b6-3235a0cbacff" containerID="a484b433ef7655a8adc32842c16b2f1cbeab935582c18737d6dd791e197f86cf" exitCode=60
Nov 03 02:00:04.147918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.147878    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd" event={"ID":"0d426eea-4c34-46ea-98b6-3235a0cbacff","Type":"ContainerDied","Data":"a484b433ef7655a8adc32842c16b2f1cbeab935582c18737d6dd791e197f86cf"}
Nov 03 02:00:04.148153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.148138    5508 scope.go:122] "RemoveContainer" containerID="a484b433ef7655a8adc32842c16b2f1cbeab935582c18737d6dd791e197f86cf"
Nov 03 02:00:04.148424 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:04.148406    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd_odh-ai-gateway-infra(0d426eea-4c34-46ea-98b6-3235a0cbacff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd" podUID="0d426eea-4c34-46ea-98b6-3235a0cbacff"
Nov 03 02:00:04.150019 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.149993    5508 generic.go:356] "Generic (PLEG): container finished" podID="212d6a7a-fe8d-492f-b509-7044ba779aeb" containerID="1c47d63c787fe98e082530d2f909fa4dc2da6925306a4669b11fb61474383b2e" exitCode=60
Nov 03 02:00:04.150114 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.150071    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f" event={"ID":"212d6a7a-fe8d-492f-b509-7044ba779aeb","Type":"ContainerDied","Data":"1c47d63c787fe98e082530d2f909fa4dc2da6925306a4669b11fb61474383b2e"}
Nov 03 02:00:04.150334 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.150313    5508 scope.go:122] "RemoveContainer" containerID="1c47d63c787fe98e082530d2f909fa4dc2da6925306a4669b11fb61474383b2e"
Nov 03 02:00:04.150612 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:04.150587    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f_odh-ai-gateway-infra(212d6a7a-fe8d-492f-b509-7044ba779aeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f" podUID="212d6a7a-fe8d-492f-b509-7044ba779aeb"
Nov 03 02:00:04.154231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.153592    5508 generic.go:356] "Generic (PLEG): container finished" podID="48146943-2c15-4c2c-be1f-b7d340ec713c" containerID="4ab679cb3dc66f4f1b313dbef1517c85a60b661aeb2036bb9159f4b62c5c2cb5" exitCode=60
Nov 03 02:00:04.154231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.153678    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2" event={"ID":"48146943-2c15-4c2c-be1f-b7d340ec713c","Type":"ContainerDied","Data":"4ab679cb3dc66f4f1b313dbef1517c85a60b661aeb2036bb9159f4b62c5c2cb5"}
Nov 03 02:00:04.154231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.153948    5508 scope.go:122] "RemoveContainer" containerID="4ab679cb3dc66f4f1b313dbef1517c85a60b661aeb2036bb9159f4b62c5c2cb5"
Nov 03 02:00:04.154231 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:04.154187    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2_odh-ai-gateway-infra(48146943-2c15-4c2c-be1f-b7d340ec713c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2" podUID="48146943-2c15-4c2c-be1f-b7d340ec713c"
Nov 03 02:00:04.156643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.156620    5508 generic.go:356] "Generic (PLEG): container finished" podID="fbfcd995-8f45-423f-b566-350c1270a59d" containerID="67eb6707eca38ec03cf2396e0837d9c7759f6df7d99605dbae51d5ae8291451b" exitCode=60
Nov 03 02:00:04.156760 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.156701    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh" event={"ID":"fbfcd995-8f45-423f-b566-350c1270a59d","Type":"ContainerDied","Data":"67eb6707eca38ec03cf2396e0837d9c7759f6df7d99605dbae51d5ae8291451b"}
Nov 03 02:00:04.156898 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.156885    5508 scope.go:122] "RemoveContainer" containerID="67eb6707eca38ec03cf2396e0837d9c7759f6df7d99605dbae51d5ae8291451b"
Nov 03 02:00:04.157094 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:04.157078    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh_odh-ai-gateway-infra(fbfcd995-8f45-423f-b566-350c1270a59d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh" podUID="fbfcd995-8f45-423f-b566-350c1270a59d"
Nov 03 02:00:04.158764 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.158744    5508 generic.go:356] "Generic (PLEG): container finished" podID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" containerID="294f58ea6486e68fbcd434bf6106610b4482af858e0c475c4755757bcf368774" exitCode=60
Nov 03 02:00:04.158859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.158767    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r" event={"ID":"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f","Type":"ContainerDied","Data":"294f58ea6486e68fbcd434bf6106610b4482af858e0c475c4755757bcf368774"}
Nov 03 02:00:04.159055 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.159038    5508 scope.go:122] "RemoveContainer" containerID="294f58ea6486e68fbcd434bf6106610b4482af858e0c475c4755757bcf368774"
Nov 03 02:00:04.159285 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:04.159266    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r_odh-ai-gateway-infra(4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r" podUID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f"
Nov 03 02:00:04.199377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.199337    5508 scope.go:122] "RemoveContainer" containerID="1a334835398d43749a8e44b9e22e5ee677678fd1619e83654c80b202d7e78cc0"
Nov 03 02:00:04.212069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.212046    5508 scope.go:122] "RemoveContainer" containerID="637dc8fc15a84f7455c57d2d26d55ae45e46ee9820e9246922e8ed62182b3048"
Nov 03 02:00:04.224070 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.224040    5508 scope.go:122] "RemoveContainer" containerID="12f26bbf8e3692114b32ed3a61001f8de0fc9e20a628d110c50af89800dd29e2"
Nov 03 02:00:04.236234 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.236219    5508 scope.go:122] "RemoveContainer" containerID="128cba637ca309c2e46a15c7331022aa005e127f372a32943cb2341f58903942"
Nov 03 02:00:04.247949 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:04.247930    5508 scope.go:122] "RemoveContainer" containerID="0e2cf339e4f99c20fb9ae775b42cabbe91633f4cc9530ca0bc4680282afd5637"
Nov 03 02:00:05.167073 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:05.167046    5508 scope.go:122] "RemoveContainer" containerID="294f58ea6486e68fbcd434bf6106610b4482af858e0c475c4755757bcf368774"
Nov 03 02:00:05.167592 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:05.167388    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r_odh-ai-gateway-infra(4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r" podUID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f"
Nov 03 02:00:05.168807 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:05.168788    5508 scope.go:122] "RemoveContainer" containerID="c754cacf64dd27758359c8cefcf6ebffa4538feeb85d476c1b19c927e152cb19"
Nov 03 02:00:05.169090 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:05.169065    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm_odh-ai-gateway-infra(fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm" podUID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c"
Nov 03 02:00:05.170310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:05.170294    5508 scope.go:122] "RemoveContainer" containerID="a484b433ef7655a8adc32842c16b2f1cbeab935582c18737d6dd791e197f86cf"
Nov 03 02:00:05.170537 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:05.170513    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd_odh-ai-gateway-infra(0d426eea-4c34-46ea-98b6-3235a0cbacff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd" podUID="0d426eea-4c34-46ea-98b6-3235a0cbacff"
Nov 03 02:00:05.171980 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:05.171793    5508 scope.go:122] "RemoveContainer" containerID="1c47d63c787fe98e082530d2f909fa4dc2da6925306a4669b11fb61474383b2e"
Nov 03 02:00:05.172084 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:05.171996    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f_odh-ai-gateway-infra(212d6a7a-fe8d-492f-b509-7044ba779aeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f" podUID="212d6a7a-fe8d-492f-b509-7044ba779aeb"
Nov 03 02:00:05.173420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:05.173400    5508 scope.go:122] "RemoveContainer" containerID="4ab679cb3dc66f4f1b313dbef1517c85a60b661aeb2036bb9159f4b62c5c2cb5"
Nov 03 02:00:05.173638 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:05.173613    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2_odh-ai-gateway-infra(48146943-2c15-4c2c-be1f-b7d340ec713c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2" podUID="48146943-2c15-4c2c-be1f-b7d340ec713c"
Nov 03 02:00:05.174810 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:05.174795    5508 scope.go:122] "RemoveContainer" containerID="67eb6707eca38ec03cf2396e0837d9c7759f6df7d99605dbae51d5ae8291451b"
Nov 03 02:00:05.174967 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:05.174950    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh_odh-ai-gateway-infra(fbfcd995-8f45-423f-b566-350c1270a59d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh" podUID="fbfcd995-8f45-423f-b566-350c1270a59d"
Nov 03 02:00:12.448284 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.448249    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"]
Nov 03 02:00:12.449225 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.449065    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f6628479-581a-41b5-8aac-8a1cf766f7f1" containerName="revoke-keys"
Nov 03 02:00:12.449225 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.449089    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6628479-581a-41b5-8aac-8a1cf766f7f1" containerName="revoke-keys"
Nov 03 02:00:12.449476 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.449259    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6628479-581a-41b5-8aac-8a1cf766f7f1" containerName="revoke-keys"
Nov 03 02:00:12.454515 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.454493    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"
Nov 03 02:00:12.506125 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.506092    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f95773\""
Nov 03 02:00:12.552519 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.552484    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f95773-dockercfg-r6mtj\""
Nov 03 02:00:12.557375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.557328    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"]
Nov 03 02:00:12.614954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.614920    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9172158a-823e-413f-a32f-d0a4d39ef09b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw\" (UID: \"9172158a-823e-413f-a32f-d0a4d39ef09b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"
Nov 03 02:00:12.615143 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.614998    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f45tn\" (UniqueName: \"kubernetes.io/projected/9172158a-823e-413f-a32f-d0a4d39ef09b-kube-api-access-f45tn\") pod \"payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw\" (UID: \"9172158a-823e-413f-a32f-d0a4d39ef09b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"
Nov 03 02:00:12.716143 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.716064    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f45tn\" (UniqueName: \"kubernetes.io/projected/9172158a-823e-413f-a32f-d0a4d39ef09b-kube-api-access-f45tn\") pod \"payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw\" (UID: \"9172158a-823e-413f-a32f-d0a4d39ef09b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"
Nov 03 02:00:12.716315 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.716196    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9172158a-823e-413f-a32f-d0a4d39ef09b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw\" (UID: \"9172158a-823e-413f-a32f-d0a4d39ef09b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"
Nov 03 02:00:12.716830 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.716811    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9172158a-823e-413f-a32f-d0a4d39ef09b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw\" (UID: \"9172158a-823e-413f-a32f-d0a4d39ef09b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"
Nov 03 02:00:12.773410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:12.773380    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f45tn\" (UniqueName: \"kubernetes.io/projected/9172158a-823e-413f-a32f-d0a4d39ef09b-kube-api-access-f45tn\") pod \"payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw\" (UID: \"9172158a-823e-413f-a32f-d0a4d39ef09b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"
Nov 03 02:00:13.068121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:13.068035    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"
Nov 03 02:00:13.198417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:13.198376    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"]
Nov 03 02:00:13.204311 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:13.204286    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"
Nov 03 02:00:13.255981 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:13.255955    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"]
Nov 03 02:00:13.322204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:13.322115    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2w6b\" (UniqueName: \"kubernetes.io/projected/b310f9ce-a97a-4ec8-89eb-e9d266b93253-kube-api-access-z2w6b\") pod \"payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5\" (UID: \"b310f9ce-a97a-4ec8-89eb-e9d266b93253\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"
Nov 03 02:00:13.322204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:13.322181    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b310f9ce-a97a-4ec8-89eb-e9d266b93253-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5\" (UID: \"b310f9ce-a97a-4ec8-89eb-e9d266b93253\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"
Nov 03 02:00:13.424027 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:13.423991    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2w6b\" (UniqueName: \"kubernetes.io/projected/b310f9ce-a97a-4ec8-89eb-e9d266b93253-kube-api-access-z2w6b\") pod \"payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5\" (UID: \"b310f9ce-a97a-4ec8-89eb-e9d266b93253\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"
Nov 03 02:00:13.424244 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:13.424050    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b310f9ce-a97a-4ec8-89eb-e9d266b93253-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5\" (UID: \"b310f9ce-a97a-4ec8-89eb-e9d266b93253\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"
Nov 03 02:00:13.424780 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:13.424759    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b310f9ce-a97a-4ec8-89eb-e9d266b93253-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5\" (UID: \"b310f9ce-a97a-4ec8-89eb-e9d266b93253\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"
Nov 03 02:00:13.460848 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:13.460817    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"]
Nov 03 02:00:13.467603 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:00:13.467570    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9172158a_823e_413f_a32f_d0a4d39ef09b.slice/crio-0ab6ab058dafaa1886c7dab481792f840ef329f7142c09fbec45530cad9c336f WatchSource:0}: Error finding container 0ab6ab058dafaa1886c7dab481792f840ef329f7142c09fbec45530cad9c336f: Status 404 returned error can't find the container with id 0ab6ab058dafaa1886c7dab481792f840ef329f7142c09fbec45530cad9c336f
Nov 03 02:00:13.485463 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:13.485432    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2w6b\" (UniqueName: \"kubernetes.io/projected/b310f9ce-a97a-4ec8-89eb-e9d266b93253-kube-api-access-z2w6b\") pod \"payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5\" (UID: \"b310f9ce-a97a-4ec8-89eb-e9d266b93253\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"
Nov 03 02:00:13.518555 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:13.518527    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"
Nov 03 02:00:13.812464 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:13.812426    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:00:13.899395 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:13.899362    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"]
Nov 03 02:00:13.906581 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:00:13.906547    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb310f9ce_a97a_4ec8_89eb_e9d266b93253.slice/crio-9fd1e7319bd027ded8e3f7a685a677221ee96c0575360cbaa5c38754dd91de98 WatchSource:0}: Error finding container 9fd1e7319bd027ded8e3f7a685a677221ee96c0575360cbaa5c38754dd91de98: Status 404 returned error can't find the container with id 9fd1e7319bd027ded8e3f7a685a677221ee96c0575360cbaa5c38754dd91de98
Nov 03 02:00:14.223443 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:14.223362    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"]
Nov 03 02:00:14.286850 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:14.281880    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5" event={"ID":"b310f9ce-a97a-4ec8-89eb-e9d266b93253","Type":"ContainerStarted","Data":"b55cd6d406664ccc05e9ca568c7e9e082de21a7a3523392210acb2992c038efc"}
Nov 03 02:00:14.286850 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:14.281947    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5" event={"ID":"b310f9ce-a97a-4ec8-89eb-e9d266b93253","Type":"ContainerStarted","Data":"9fd1e7319bd027ded8e3f7a685a677221ee96c0575360cbaa5c38754dd91de98"}
Nov 03 02:00:14.286850 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:14.282771    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"
Nov 03 02:00:14.326661 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:14.326626    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw" event={"ID":"9172158a-823e-413f-a32f-d0a4d39ef09b","Type":"ContainerStarted","Data":"0bec3b21a46b0b123822146c16f3f95134b49a58af0f55f955823744ffb633fe"}
Nov 03 02:00:14.326933 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:14.326914    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw" event={"ID":"9172158a-823e-413f-a32f-d0a4d39ef09b","Type":"ContainerStarted","Data":"0ab6ab058dafaa1886c7dab481792f840ef329f7142c09fbec45530cad9c336f"}
Nov 03 02:00:14.327458 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:14.327432    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"
Nov 03 02:00:14.416484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:14.416454    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"]
Nov 03 02:00:14.464893 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:14.464825    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw" podStartSLOduration=2.464806367 podStartE2EDuration="2.464806367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:14.459435664 +0000 UTC m=+2493.885042157" watchObservedRunningTime="2026-11-03 02:00:14.464806367 +0000 UTC m=+2493.890412831"
Nov 03 02:00:14.572242 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:14.572131    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5" podStartSLOduration=1.572115771 podStartE2EDuration="1.572115771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:14.568764092 +0000 UTC m=+2493.994370556" watchObservedRunningTime="2026-11-03 02:00:14.572115771 +0000 UTC m=+2493.997722236"
Nov 03 02:00:14.695532 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:14.695501    5508 scope.go:122] "RemoveContainer" containerID="c754cacf64dd27758359c8cefcf6ebffa4538feeb85d476c1b19c927e152cb19"
Nov 03 02:00:15.335579 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:15.335498    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm" event={"ID":"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c","Type":"ContainerStarted","Data":"86f2fb94a7a8eeb609bf746687c35c8e9be15a8b481a888c5a6035661e0bdceb"}
Nov 03 02:00:15.687726 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:15.687696    5508 scope.go:122] "RemoveContainer" containerID="67eb6707eca38ec03cf2396e0837d9c7759f6df7d99605dbae51d5ae8291451b"
Nov 03 02:00:15.689008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:15.688985    5508 scope.go:122] "RemoveContainer" containerID="a484b433ef7655a8adc32842c16b2f1cbeab935582c18737d6dd791e197f86cf"
Nov 03 02:00:16.345771 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:16.345688    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd" event={"ID":"0d426eea-4c34-46ea-98b6-3235a0cbacff","Type":"ContainerStarted","Data":"2ee51b033a43e37e1b3b50d371b57192e6566107c893e7eab3db7c7a25b33981"}
Nov 03 02:00:16.347736 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:16.347703    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh" event={"ID":"fbfcd995-8f45-423f-b566-350c1270a59d","Type":"ContainerStarted","Data":"f4d45c77edd6924379eb2fa035e37f41cf486af3cbc9192544378a8783180354"}
Nov 03 02:00:16.349838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:16.349812    5508 generic.go:356] "Generic (PLEG): container finished" podID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" containerID="86f2fb94a7a8eeb609bf746687c35c8e9be15a8b481a888c5a6035661e0bdceb" exitCode=60
Nov 03 02:00:16.349958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:16.349873    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm" event={"ID":"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c","Type":"ContainerDied","Data":"86f2fb94a7a8eeb609bf746687c35c8e9be15a8b481a888c5a6035661e0bdceb"}
Nov 03 02:00:16.349958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:16.349924    5508 scope.go:122] "RemoveContainer" containerID="c754cacf64dd27758359c8cefcf6ebffa4538feeb85d476c1b19c927e152cb19"
Nov 03 02:00:16.350386 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:16.350370    5508 scope.go:122] "RemoveContainer" containerID="86f2fb94a7a8eeb609bf746687c35c8e9be15a8b481a888c5a6035661e0bdceb"
Nov 03 02:00:16.350661 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:16.350631    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm_odh-ai-gateway-infra(fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm" podUID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c"
Nov 03 02:00:16.456546 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:16.456516    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"]
Nov 03 02:00:16.694803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:16.694781    5508 scope.go:122] "RemoveContainer" containerID="1c47d63c787fe98e082530d2f909fa4dc2da6925306a4669b11fb61474383b2e"
Nov 03 02:00:16.695179 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:16.694937    5508 scope.go:122] "RemoveContainer" containerID="4ab679cb3dc66f4f1b313dbef1517c85a60b661aeb2036bb9159f4b62c5c2cb5"
Nov 03 02:00:17.383121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.383029    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f" event={"ID":"212d6a7a-fe8d-492f-b509-7044ba779aeb","Type":"ContainerStarted","Data":"2d464b5971bbcb89c8cc3a5fc79f0f17f0a92016b6c0068856e65bd4bf95e218"}
Nov 03 02:00:17.387151 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.386960    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2" event={"ID":"48146943-2c15-4c2c-be1f-b7d340ec713c","Type":"ContainerStarted","Data":"76067aa9c1f0ca166b9077ef0d5322a68b731cc21f2c5a8c40022a36762e6f0c"}
Nov 03 02:00:17.410670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.394431    5508 generic.go:356] "Generic (PLEG): container finished" podID="fbfcd995-8f45-423f-b566-350c1270a59d" containerID="f4d45c77edd6924379eb2fa035e37f41cf486af3cbc9192544378a8783180354" exitCode=60
Nov 03 02:00:17.410670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.394664    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh" event={"ID":"fbfcd995-8f45-423f-b566-350c1270a59d","Type":"ContainerDied","Data":"f4d45c77edd6924379eb2fa035e37f41cf486af3cbc9192544378a8783180354"}
Nov 03 02:00:17.410670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.394733    5508 scope.go:122] "RemoveContainer" containerID="67eb6707eca38ec03cf2396e0837d9c7759f6df7d99605dbae51d5ae8291451b"
Nov 03 02:00:17.410670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.399181    5508 scope.go:122] "RemoveContainer" containerID="f4d45c77edd6924379eb2fa035e37f41cf486af3cbc9192544378a8783180354"
Nov 03 02:00:17.410670 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:17.399500    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh_odh-ai-gateway-infra(fbfcd995-8f45-423f-b566-350c1270a59d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh" podUID="fbfcd995-8f45-423f-b566-350c1270a59d"
Nov 03 02:00:17.410670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.404704    5508 generic.go:356] "Generic (PLEG): container finished" podID="0d426eea-4c34-46ea-98b6-3235a0cbacff" containerID="2ee51b033a43e37e1b3b50d371b57192e6566107c893e7eab3db7c7a25b33981" exitCode=60
Nov 03 02:00:17.410670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.404736    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd" event={"ID":"0d426eea-4c34-46ea-98b6-3235a0cbacff","Type":"ContainerDied","Data":"2ee51b033a43e37e1b3b50d371b57192e6566107c893e7eab3db7c7a25b33981"}
Nov 03 02:00:17.410670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.405078    5508 scope.go:122] "RemoveContainer" containerID="2ee51b033a43e37e1b3b50d371b57192e6566107c893e7eab3db7c7a25b33981"
Nov 03 02:00:17.410670 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:17.405373    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd_odh-ai-gateway-infra(0d426eea-4c34-46ea-98b6-3235a0cbacff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd" podUID="0d426eea-4c34-46ea-98b6-3235a0cbacff"
Nov 03 02:00:17.413798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.413770    5508 scope.go:122] "RemoveContainer" containerID="a484b433ef7655a8adc32842c16b2f1cbeab935582c18737d6dd791e197f86cf"
Nov 03 02:00:17.469363 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.469318    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"]
Nov 03 02:00:17.562750 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.562725    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"
Nov 03 02:00:17.592799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.592773    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-maas-api-service-ca\") pod \"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c\" (UID: \"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c\") "
Nov 03 02:00:17.592962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.592855    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-kube-api-access-wnpn9\" (UniqueName: \"kubernetes.io/projected/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-kube-api-access-wnpn9\") pod \"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c\" (UID: \"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c\") "
Nov 03 02:00:17.593287 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.593244    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-maas-api-service-ca" pod "fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" (UID: "fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:00:17.593475 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.593457    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:17.649188 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.648877    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-kube-api-access-wnpn9" pod "fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" (UID: "fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c"). InnerVolumeSpecName "kube-api-access-wnpn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:00:17.680455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.680432    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"]
Nov 03 02:00:17.685108 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.685085    5508 scope.go:122] "RemoveContainer" containerID="294f58ea6486e68fbcd434bf6106610b4482af858e0c475c4755757bcf368774"
Nov 03 02:00:17.694754 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:17.694730    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnpn9\" (UniqueName: \"kubernetes.io/projected/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c-kube-api-access-wnpn9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:18.412063 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.412036    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"
Nov 03 02:00:18.412552 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.412074    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm" event={"ID":"fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c","Type":"ContainerDied","Data":"069fc03e6678225f57f07af9b051e501b272a40b8711d872580a71fbb8523cc3"}
Nov 03 02:00:18.412552 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.412147    5508 scope.go:122] "RemoveContainer" containerID="86f2fb94a7a8eeb609bf746687c35c8e9be15a8b481a888c5a6035661e0bdceb"
Nov 03 02:00:18.416697 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.416673    5508 generic.go:356] "Generic (PLEG): container finished" podID="212d6a7a-fe8d-492f-b509-7044ba779aeb" containerID="2d464b5971bbcb89c8cc3a5fc79f0f17f0a92016b6c0068856e65bd4bf95e218" exitCode=60
Nov 03 02:00:18.416804 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.416741    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f" event={"ID":"212d6a7a-fe8d-492f-b509-7044ba779aeb","Type":"ContainerDied","Data":"2d464b5971bbcb89c8cc3a5fc79f0f17f0a92016b6c0068856e65bd4bf95e218"}
Nov 03 02:00:18.417135 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.417116    5508 scope.go:122] "RemoveContainer" containerID="2d464b5971bbcb89c8cc3a5fc79f0f17f0a92016b6c0068856e65bd4bf95e218"
Nov 03 02:00:18.417451 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:18.417425    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f_odh-ai-gateway-infra(212d6a7a-fe8d-492f-b509-7044ba779aeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f" podUID="212d6a7a-fe8d-492f-b509-7044ba779aeb"
Nov 03 02:00:18.419245 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.419222    5508 generic.go:356] "Generic (PLEG): container finished" podID="48146943-2c15-4c2c-be1f-b7d340ec713c" containerID="76067aa9c1f0ca166b9077ef0d5322a68b731cc21f2c5a8c40022a36762e6f0c" exitCode=60
Nov 03 02:00:18.419395 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.419287    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2" event={"ID":"48146943-2c15-4c2c-be1f-b7d340ec713c","Type":"ContainerDied","Data":"76067aa9c1f0ca166b9077ef0d5322a68b731cc21f2c5a8c40022a36762e6f0c"}
Nov 03 02:00:18.419621 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.419604    5508 scope.go:122] "RemoveContainer" containerID="76067aa9c1f0ca166b9077ef0d5322a68b731cc21f2c5a8c40022a36762e6f0c"
Nov 03 02:00:18.419862 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:18.419841    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2_odh-ai-gateway-infra(48146943-2c15-4c2c-be1f-b7d340ec713c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2" podUID="48146943-2c15-4c2c-be1f-b7d340ec713c"
Nov 03 02:00:18.425018 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.425001    5508 scope.go:122] "RemoveContainer" containerID="1c47d63c787fe98e082530d2f909fa4dc2da6925306a4669b11fb61474383b2e"
Nov 03 02:00:18.425089 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.425019    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r" event={"ID":"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f","Type":"ContainerStarted","Data":"74b51b806a5286121cb0c0d121a2a9e664a7061be30badd446f0206e66c68fac"}
Nov 03 02:00:18.440868 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.439840    5508 scope.go:122] "RemoveContainer" containerID="4ab679cb3dc66f4f1b313dbef1517c85a60b661aeb2036bb9159f4b62c5c2cb5"
Nov 03 02:00:18.505742 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.505445    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"]
Nov 03 02:00:18.580602 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.580573    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"
Nov 03 02:00:18.595007 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.594987    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"
Nov 03 02:00:18.605444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.605427    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d426eea-4c34-46ea-98b6-3235a0cbacff-kube-api-access-kcv95\" (UniqueName: \"kubernetes.io/projected/0d426eea-4c34-46ea-98b6-3235a0cbacff-kube-api-access-kcv95\") pod \"0d426eea-4c34-46ea-98b6-3235a0cbacff\" (UID: \"0d426eea-4c34-46ea-98b6-3235a0cbacff\") "
Nov 03 02:00:18.605521 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.605491    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d426eea-4c34-46ea-98b6-3235a0cbacff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d426eea-4c34-46ea-98b6-3235a0cbacff-maas-api-service-ca\") pod \"0d426eea-4c34-46ea-98b6-3235a0cbacff\" (UID: \"0d426eea-4c34-46ea-98b6-3235a0cbacff\") "
Nov 03 02:00:18.605521 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.605512    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbfcd995-8f45-423f-b566-350c1270a59d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfcd995-8f45-423f-b566-350c1270a59d-maas-api-service-ca\") pod \"fbfcd995-8f45-423f-b566-350c1270a59d\" (UID: \"fbfcd995-8f45-423f-b566-350c1270a59d\") "
Nov 03 02:00:18.605597 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.605575    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfcd995-8f45-423f-b566-350c1270a59d-kube-api-access-dqxqt\" (UniqueName: \"kubernetes.io/projected/fbfcd995-8f45-423f-b566-350c1270a59d-kube-api-access-dqxqt\") pod \"fbfcd995-8f45-423f-b566-350c1270a59d\" (UID: \"fbfcd995-8f45-423f-b566-350c1270a59d\") "
Nov 03 02:00:18.605861 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.605834    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d426eea-4c34-46ea-98b6-3235a0cbacff-maas-api-service-ca" pod "0d426eea-4c34-46ea-98b6-3235a0cbacff" (UID: "0d426eea-4c34-46ea-98b6-3235a0cbacff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:00:18.605983 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.605880    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfcd995-8f45-423f-b566-350c1270a59d-maas-api-service-ca" pod "fbfcd995-8f45-423f-b566-350c1270a59d" (UID: "fbfcd995-8f45-423f-b566-350c1270a59d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:00:18.606075 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.606059    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d426eea-4c34-46ea-98b6-3235a0cbacff-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:18.606121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.606081    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbfcd995-8f45-423f-b566-350c1270a59d-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:18.607582 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.607564    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d426eea-4c34-46ea-98b6-3235a0cbacff-kube-api-access-kcv95" pod "0d426eea-4c34-46ea-98b6-3235a0cbacff" (UID: "0d426eea-4c34-46ea-98b6-3235a0cbacff"). InnerVolumeSpecName "kube-api-access-kcv95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:00:18.607675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.607595    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfcd995-8f45-423f-b566-350c1270a59d-kube-api-access-dqxqt" pod "fbfcd995-8f45-423f-b566-350c1270a59d" (UID: "fbfcd995-8f45-423f-b566-350c1270a59d"). InnerVolumeSpecName "kube-api-access-dqxqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:00:18.676826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.676771    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"]
Nov 03 02:00:18.680815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.680788    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-738acb-29894520-vq9cm"]
Nov 03 02:00:18.693155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.693137    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" path="/var/lib/kubelet/pods/fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c/volumes"
Nov 03 02:00:18.707585 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.707565    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcv95\" (UniqueName: \"kubernetes.io/projected/0d426eea-4c34-46ea-98b6-3235a0cbacff-kube-api-access-kcv95\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:18.707654 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.707586    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqxqt\" (UniqueName: \"kubernetes.io/projected/fbfcd995-8f45-423f-b566-350c1270a59d-kube-api-access-dqxqt\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:18.937218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:18.937140    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"]
Nov 03 02:00:19.061660 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.061623    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"]
Nov 03 02:00:19.062461 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062436    5508 cpu_manager.go:414] "containerMap: removing container" podUID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" containerName="cleanup"
Nov 03 02:00:19.062461 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062459    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" containerName="cleanup"
Nov 03 02:00:19.062657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062487    5508 cpu_manager.go:414] "containerMap: removing container" podUID="fbfcd995-8f45-423f-b566-350c1270a59d" containerName="cleanup"
Nov 03 02:00:19.062657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062495    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfcd995-8f45-423f-b566-350c1270a59d" containerName="cleanup"
Nov 03 02:00:19.062657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062516    5508 cpu_manager.go:414] "containerMap: removing container" podUID="fbfcd995-8f45-423f-b566-350c1270a59d" containerName="cleanup"
Nov 03 02:00:19.062657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062523    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfcd995-8f45-423f-b566-350c1270a59d" containerName="cleanup"
Nov 03 02:00:19.062657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062540    5508 cpu_manager.go:414] "containerMap: removing container" podUID="fbfcd995-8f45-423f-b566-350c1270a59d" containerName="cleanup"
Nov 03 02:00:19.062657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062546    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfcd995-8f45-423f-b566-350c1270a59d" containerName="cleanup"
Nov 03 02:00:19.062657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062560    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0d426eea-4c34-46ea-98b6-3235a0cbacff" containerName="cleanup"
Nov 03 02:00:19.062657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062566    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d426eea-4c34-46ea-98b6-3235a0cbacff" containerName="cleanup"
Nov 03 02:00:19.062657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062584    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0d426eea-4c34-46ea-98b6-3235a0cbacff" containerName="cleanup"
Nov 03 02:00:19.062657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062592    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d426eea-4c34-46ea-98b6-3235a0cbacff" containerName="cleanup"
Nov 03 02:00:19.062657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062603    5508 cpu_manager.go:414] "containerMap: removing container" podUID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" containerName="cleanup"
Nov 03 02:00:19.062657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062609    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" containerName="cleanup"
Nov 03 02:00:19.063029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062741    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" containerName="cleanup"
Nov 03 02:00:19.063029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062755    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfcd995-8f45-423f-b566-350c1270a59d" containerName="cleanup"
Nov 03 02:00:19.063029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062766    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfcd995-8f45-423f-b566-350c1270a59d" containerName="cleanup"
Nov 03 02:00:19.063029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062777    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfcd995-8f45-423f-b566-350c1270a59d" containerName="cleanup"
Nov 03 02:00:19.063029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062787    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d426eea-4c34-46ea-98b6-3235a0cbacff" containerName="cleanup"
Nov 03 02:00:19.063029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062795    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d426eea-4c34-46ea-98b6-3235a0cbacff" containerName="cleanup"
Nov 03 02:00:19.063029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.062802    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" containerName="cleanup"
Nov 03 02:00:19.067218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.067203    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.111698 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.111675    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.111846 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.111723    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-metrics-certs\") pod \"maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.111846 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.111818    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.111943 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.111869    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpwt9\" (UniqueName: \"kubernetes.io/projected/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-kube-api-access-wpwt9\") pod \"maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.115984 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.115966    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-e0907d-serving-cert\""
Nov 03 02:00:19.124437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.124412    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"]
Nov 03 02:00:19.213553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.213454    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.213553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.213525    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-metrics-certs\") pod \"maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.213840 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.213565    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.213840 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.213596    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpwt9\" (UniqueName: \"kubernetes.io/projected/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-kube-api-access-wpwt9\") pod \"maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.214555 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:19.214533    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-serving-cert: secret "maas-api-e2e-praxis-b-e0907d-serving-cert" not found
Nov 03 02:00:19.214770 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:19.214757    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-maas-api-tls podName:9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4 nodeName:}" failed. No retries permitted until 2026-11-03 02:00:19.714734623 +0000 UTC m=+2499.140341081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-maas-api-tls") pod "maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd" (UID: "9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4") : secret "maas-api-e2e-praxis-b-e0907d-serving-cert" not found
Nov 03 02:00:19.217731 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.217704    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-metrics-certs\") pod \"maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.218934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.218907    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.223695 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.223663    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n" podUID="9ed1fa2e-f822-4105-8b59-4da996a8ce65" containerName="payload-processing" containerID="cri-o://51abda21b40d16006dca86992b1cc604c5d50cb0760eb57aae612489e9d013f5" gracePeriod=25
Nov 03 02:00:19.270920 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.270897    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpwt9\" (UniqueName: \"kubernetes.io/projected/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-kube-api-access-wpwt9\") pod \"maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.416796 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.416729    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7" podUID="0ccb7d48-53d7-4fe6-aa77-c1675128fdc2" containerName="payload-pre-processing" containerID="cri-o://b8d24bc7c48ccd8819e3334e094936c865128c9acb8b35653e59481f76dbdb46" gracePeriod=25
Nov 03 02:00:19.436286 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.436258    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"
Nov 03 02:00:19.437607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.437072    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd" event={"ID":"0d426eea-4c34-46ea-98b6-3235a0cbacff","Type":"ContainerDied","Data":"fea7cbd486a76f6ca8463478408fbe0d24fc7c1a1fabe8d5ce15a01a5b2ae028"}
Nov 03 02:00:19.437607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.437137    5508 scope.go:122] "RemoveContainer" containerID="2ee51b033a43e37e1b3b50d371b57192e6566107c893e7eab3db7c7a25b33981"
Nov 03 02:00:19.462371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.460169    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh" event={"ID":"fbfcd995-8f45-423f-b566-350c1270a59d","Type":"ContainerDied","Data":"3cd12b7f485c3164bb223abbe10436db836975a267b536d670225ce148188b56"}
Nov 03 02:00:19.475459 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.462838    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"
Nov 03 02:00:19.483532 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.483300    5508 generic.go:356] "Generic (PLEG): container finished" podID="9ed1fa2e-f822-4105-8b59-4da996a8ce65" containerID="51abda21b40d16006dca86992b1cc604c5d50cb0760eb57aae612489e9d013f5" exitCode=0
Nov 03 02:00:19.483532 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.483422    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n" event={"ID":"9ed1fa2e-f822-4105-8b59-4da996a8ce65","Type":"ContainerDied","Data":"51abda21b40d16006dca86992b1cc604c5d50cb0760eb57aae612489e9d013f5"}
Nov 03 02:00:19.487457 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.487328    5508 generic.go:356] "Generic (PLEG): container finished" podID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" containerID="74b51b806a5286121cb0c0d121a2a9e664a7061be30badd446f0206e66c68fac" exitCode=60
Nov 03 02:00:19.487780 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.487740    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r" event={"ID":"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f","Type":"ContainerDied","Data":"74b51b806a5286121cb0c0d121a2a9e664a7061be30badd446f0206e66c68fac"}
Nov 03 02:00:19.489544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.489522    5508 scope.go:122] "RemoveContainer" containerID="74b51b806a5286121cb0c0d121a2a9e664a7061be30badd446f0206e66c68fac"
Nov 03 02:00:19.490064 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:19.490030    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r_odh-ai-gateway-infra(4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r" podUID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f"
Nov 03 02:00:19.534802 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.534783    5508 scope.go:122] "RemoveContainer" containerID="f4d45c77edd6924379eb2fa035e37f41cf486af3cbc9192544378a8783180354"
Nov 03 02:00:19.541820 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.541795    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"
Nov 03 02:00:19.548757 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.548737    5508 scope.go:122] "RemoveContainer" containerID="294f58ea6486e68fbcd434bf6106610b4482af858e0c475c4755757bcf368774"
Nov 03 02:00:19.619335 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.619304    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed1fa2e-f822-4105-8b59-4da996a8ce65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed1fa2e-f822-4105-8b59-4da996a8ce65-plugins-config-volume\") pod \"9ed1fa2e-f822-4105-8b59-4da996a8ce65\" (UID: \"9ed1fa2e-f822-4105-8b59-4da996a8ce65\") "
Nov 03 02:00:19.619958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.619375    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed1fa2e-f822-4105-8b59-4da996a8ce65-kube-api-access-h7gpl\" (UniqueName: \"kubernetes.io/projected/9ed1fa2e-f822-4105-8b59-4da996a8ce65-kube-api-access-h7gpl\") pod \"9ed1fa2e-f822-4105-8b59-4da996a8ce65\" (UID: \"9ed1fa2e-f822-4105-8b59-4da996a8ce65\") "
Nov 03 02:00:19.619958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.619747    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed1fa2e-f822-4105-8b59-4da996a8ce65-plugins-config-volume" pod "9ed1fa2e-f822-4105-8b59-4da996a8ce65" (UID: "9ed1fa2e-f822-4105-8b59-4da996a8ce65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:00:19.619958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.619880    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed1fa2e-f822-4105-8b59-4da996a8ce65-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:19.622069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.622044    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed1fa2e-f822-4105-8b59-4da996a8ce65-kube-api-access-h7gpl" pod "9ed1fa2e-f822-4105-8b59-4da996a8ce65" (UID: "9ed1fa2e-f822-4105-8b59-4da996a8ce65"). InnerVolumeSpecName "kube-api-access-h7gpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:00:19.679086 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.679056    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"
Nov 03 02:00:19.702056 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.702034    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"
Nov 03 02:00:19.709770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.709749    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"
Nov 03 02:00:19.720580 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.720559    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-kube-api-access-2psj7\" (UniqueName: \"kubernetes.io/projected/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-kube-api-access-2psj7\") pod \"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2\" (UID: \"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2\") "
Nov 03 02:00:19.720690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.720620    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212d6a7a-fe8d-492f-b509-7044ba779aeb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212d6a7a-fe8d-492f-b509-7044ba779aeb-maas-api-service-ca\") pod \"212d6a7a-fe8d-492f-b509-7044ba779aeb\" (UID: \"212d6a7a-fe8d-492f-b509-7044ba779aeb\") "
Nov 03 02:00:19.720755 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.720730    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212d6a7a-fe8d-492f-b509-7044ba779aeb-kube-api-access-79spv\" (UniqueName: \"kubernetes.io/projected/212d6a7a-fe8d-492f-b509-7044ba779aeb-kube-api-access-79spv\") pod \"212d6a7a-fe8d-492f-b509-7044ba779aeb\" (UID: \"212d6a7a-fe8d-492f-b509-7044ba779aeb\") "
Nov 03 02:00:19.720841 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.720823    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48146943-2c15-4c2c-be1f-b7d340ec713c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48146943-2c15-4c2c-be1f-b7d340ec713c-maas-api-service-ca\") pod \"48146943-2c15-4c2c-be1f-b7d340ec713c\" (UID: \"48146943-2c15-4c2c-be1f-b7d340ec713c\") "
Nov 03 02:00:19.720934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.720875    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-plugins-config-volume\") pod \"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2\" (UID: \"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2\") "
Nov 03 02:00:19.720934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.720893    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212d6a7a-fe8d-492f-b509-7044ba779aeb-maas-api-service-ca" pod "212d6a7a-fe8d-492f-b509-7044ba779aeb" (UID: "212d6a7a-fe8d-492f-b509-7044ba779aeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:00:19.720934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.720902    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48146943-2c15-4c2c-be1f-b7d340ec713c-kube-api-access-q5r7f\" (UniqueName: \"kubernetes.io/projected/48146943-2c15-4c2c-be1f-b7d340ec713c-kube-api-access-q5r7f\") pod \"48146943-2c15-4c2c-be1f-b7d340ec713c\" (UID: \"48146943-2c15-4c2c-be1f-b7d340ec713c\") "
Nov 03 02:00:19.721279 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.721253    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-plugins-config-volume" pod "0ccb7d48-53d7-4fe6-aa77-c1675128fdc2" (UID: "0ccb7d48-53d7-4fe6-aa77-c1675128fdc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:00:19.721564 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.721540    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48146943-2c15-4c2c-be1f-b7d340ec713c-maas-api-service-ca" pod "48146943-2c15-4c2c-be1f-b7d340ec713c" (UID: "48146943-2c15-4c2c-be1f-b7d340ec713c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:00:19.721664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.721646    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.721762 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.721744    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48146943-2c15-4c2c-be1f-b7d340ec713c-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:19.721825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.721767    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:19.721825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.721784    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7gpl\" (UniqueName: \"kubernetes.io/projected/9ed1fa2e-f822-4105-8b59-4da996a8ce65-kube-api-access-h7gpl\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:19.721825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.721798    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/212d6a7a-fe8d-492f-b509-7044ba779aeb-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:19.723166 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.723141    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212d6a7a-fe8d-492f-b509-7044ba779aeb-kube-api-access-79spv" pod "212d6a7a-fe8d-492f-b509-7044ba779aeb" (UID: "212d6a7a-fe8d-492f-b509-7044ba779aeb"). InnerVolumeSpecName "kube-api-access-79spv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:00:19.723343 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.723324    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-kube-api-access-2psj7" pod "0ccb7d48-53d7-4fe6-aa77-c1675128fdc2" (UID: "0ccb7d48-53d7-4fe6-aa77-c1675128fdc2"). InnerVolumeSpecName "kube-api-access-2psj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:00:19.723759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.723735    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48146943-2c15-4c2c-be1f-b7d340ec713c-kube-api-access-q5r7f" pod "48146943-2c15-4c2c-be1f-b7d340ec713c" (UID: "48146943-2c15-4c2c-be1f-b7d340ec713c"). InnerVolumeSpecName "kube-api-access-q5r7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:00:19.724017 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.723999    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:19.822774 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.822741    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5r7f\" (UniqueName: \"kubernetes.io/projected/48146943-2c15-4c2c-be1f-b7d340ec713c-kube-api-access-q5r7f\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:19.822774 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.822771    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2psj7\" (UniqueName: \"kubernetes.io/projected/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2-kube-api-access-2psj7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:19.822774 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.822780    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79spv\" (UniqueName: \"kubernetes.io/projected/212d6a7a-fe8d-492f-b509-7044ba779aeb-kube-api-access-79spv\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:19.975677 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.975644    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"]
Nov 03 02:00:19.979564 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.979536    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-70235a-29894520-d5xrd"]
Nov 03 02:00:19.983503 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:19.983447    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:20.083889 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.083855    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"]
Nov 03 02:00:20.088887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.088860    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f95773-29894520-64qzh"]
Nov 03 02:00:20.121129 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.121087    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"]
Nov 03 02:00:20.412332 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.412304    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"]
Nov 03 02:00:20.424975 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:00:20.424930    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1c45d1_f0d4_44a3_b99e_bcb07d09d2b4.slice/crio-365db1dd29b870d39f9a4a47274ce715bf9d8703fcfdbd1ffad49dad4fde88d0 WatchSource:0}: Error finding container 365db1dd29b870d39f9a4a47274ce715bf9d8703fcfdbd1ffad49dad4fde88d0: Status 404 returned error can't find the container with id 365db1dd29b870d39f9a4a47274ce715bf9d8703fcfdbd1ffad49dad4fde88d0
Nov 03 02:00:20.426925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.426902    5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:00:20.499381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.499323    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd" event={"ID":"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4","Type":"ContainerStarted","Data":"365db1dd29b870d39f9a4a47274ce715bf9d8703fcfdbd1ffad49dad4fde88d0"}
Nov 03 02:00:20.500635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.500617    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"
Nov 03 02:00:20.500741 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.500629    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2" event={"ID":"48146943-2c15-4c2c-be1f-b7d340ec713c","Type":"ContainerDied","Data":"a4fbbecc6ede02cc53cb8411a2bed260c892ecf9e01ab4814d4de85e3b80a6b6"}
Nov 03 02:00:20.500741 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.500684    5508 scope.go:122] "RemoveContainer" containerID="76067aa9c1f0ca166b9077ef0d5322a68b731cc21f2c5a8c40022a36762e6f0c"
Nov 03 02:00:20.503134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.503102    5508 generic.go:356] "Generic (PLEG): container finished" podID="0ccb7d48-53d7-4fe6-aa77-c1675128fdc2" containerID="b8d24bc7c48ccd8819e3334e094936c865128c9acb8b35653e59481f76dbdb46" exitCode=0
Nov 03 02:00:20.503252 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.503149    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"
Nov 03 02:00:20.503252 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.503179    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7" event={"ID":"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2","Type":"ContainerDied","Data":"b8d24bc7c48ccd8819e3334e094936c865128c9acb8b35653e59481f76dbdb46"}
Nov 03 02:00:20.503252 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.503223    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7" event={"ID":"0ccb7d48-53d7-4fe6-aa77-c1675128fdc2","Type":"ContainerDied","Data":"9d9564eaf8cecbf6148770fc40467929d9ce69f8b7798b09b5aeb7ebcff19cd7"}
Nov 03 02:00:20.504735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.504709    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"
Nov 03 02:00:20.504735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.504716    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n" event={"ID":"9ed1fa2e-f822-4105-8b59-4da996a8ce65","Type":"ContainerDied","Data":"d1688b7074d3a1f5f317bb227dcd8801ff43b74805ff9ff20a12f4c5ce762c0e"}
Nov 03 02:00:20.509465 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.509413    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f" event={"ID":"212d6a7a-fe8d-492f-b509-7044ba779aeb","Type":"ContainerDied","Data":"624039b3e47ef276f8c77359a7d6edcbf6ba5070abd76b6779378157b4766b91"}
Nov 03 02:00:20.509465 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.509433    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"
Nov 03 02:00:20.514017 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.513995    5508 scope.go:122] "RemoveContainer" containerID="b8d24bc7c48ccd8819e3334e094936c865128c9acb8b35653e59481f76dbdb46"
Nov 03 02:00:20.526628 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.526605    5508 scope.go:122] "RemoveContainer" containerID="b8d24bc7c48ccd8819e3334e094936c865128c9acb8b35653e59481f76dbdb46"
Nov 03 02:00:20.526973 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:20.526955    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d24bc7c48ccd8819e3334e094936c865128c9acb8b35653e59481f76dbdb46\": container with ID starting with b8d24bc7c48ccd8819e3334e094936c865128c9acb8b35653e59481f76dbdb46 not found: ID does not exist" containerID="b8d24bc7c48ccd8819e3334e094936c865128c9acb8b35653e59481f76dbdb46"
Nov 03 02:00:20.527049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.526983    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d24bc7c48ccd8819e3334e094936c865128c9acb8b35653e59481f76dbdb46"} err="failed to get container status \"b8d24bc7c48ccd8819e3334e094936c865128c9acb8b35653e59481f76dbdb46\": rpc error: code = NotFound desc = could not find container \"b8d24bc7c48ccd8819e3334e094936c865128c9acb8b35653e59481f76dbdb46\": container with ID starting with b8d24bc7c48ccd8819e3334e094936c865128c9acb8b35653e59481f76dbdb46 not found: ID does not exist"
Nov 03 02:00:20.527049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.527001    5508 scope.go:122] "RemoveContainer" containerID="51abda21b40d16006dca86992b1cc604c5d50cb0760eb57aae612489e9d013f5"
Nov 03 02:00:20.536891 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.536864    5508 scope.go:122] "RemoveContainer" containerID="2d464b5971bbcb89c8cc3a5fc79f0f17f0a92016b6c0068856e65bd4bf95e218"
Nov 03 02:00:20.625782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.625746    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"]
Nov 03 02:00:20.631168 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.631140    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-706bdd-85bd565fbc-wnlc7"]
Nov 03 02:00:20.643028 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.643004    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"
Nov 03 02:00:20.693314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.693231    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ccb7d48-53d7-4fe6-aa77-c1675128fdc2" path="/var/lib/kubelet/pods/0ccb7d48-53d7-4fe6-aa77-c1675128fdc2/volumes"
Nov 03 02:00:20.693679 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.693665    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d426eea-4c34-46ea-98b6-3235a0cbacff" path="/var/lib/kubelet/pods/0d426eea-4c34-46ea-98b6-3235a0cbacff/volumes"
Nov 03 02:00:20.694027 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.694014    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbfcd995-8f45-423f-b566-350c1270a59d" path="/var/lib/kubelet/pods/fbfcd995-8f45-423f-b566-350c1270a59d/volumes"
Nov 03 02:00:20.733373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.733328    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-maas-api-service-ca\") pod \"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f\" (UID: \"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f\") "
Nov 03 02:00:20.733528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.733514    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-kube-api-access-566bb\" (UniqueName: \"kubernetes.io/projected/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-kube-api-access-566bb\") pod \"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f\" (UID: \"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f\") "
Nov 03 02:00:20.733636 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.733615    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-maas-api-service-ca" pod "4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" (UID: "4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:00:20.734027 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.733989    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:20.736052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.736033    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-kube-api-access-566bb" pod "4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" (UID: "4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f"). InnerVolumeSpecName "kube-api-access-566bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:00:20.781042 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.781007    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"]
Nov 03 02:00:20.786680 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.786652    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894520-qfcd2"]
Nov 03 02:00:20.834690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.834664    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-566bb\" (UniqueName: \"kubernetes.io/projected/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f-kube-api-access-566bb\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:20.997281 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:20.997202    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"]
Nov 03 02:00:21.001934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:21.001906    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-706bdd-5d9fdb4549-qrv2n"]
Nov 03 02:00:21.103498 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:21.103455    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"]
Nov 03 02:00:21.107155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:21.107128    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-b177c9-29894520-cps4f"]
Nov 03 02:00:21.521235 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:21.521200    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r" event={"ID":"4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f","Type":"ContainerDied","Data":"8caaebe2783e8ab77f625c377be3ca526379f63f5d551a410711e0135b13c179"}
Nov 03 02:00:21.521235 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:21.521226    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"
Nov 03 02:00:21.521733 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:21.521278    5508 scope.go:122] "RemoveContainer" containerID="74b51b806a5286121cb0c0d121a2a9e664a7061be30badd446f0206e66c68fac"
Nov 03 02:00:21.676777 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:21.676747    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"]
Nov 03 02:00:21.681384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:21.681361    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-0bf34a-29894520-bmf4r"]
Nov 03 02:00:22.530018 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:22.529961    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd" event={"ID":"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4","Type":"ContainerStarted","Data":"1e61dfc904622cc0cf35d3b0fca7f82cb5aeaab35fa4b483d5555ebfbd26c7f9"}
Nov 03 02:00:22.530475 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:22.530236    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:22.641559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:22.641502    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd" podStartSLOduration=2.592320834 podStartE2EDuration="3.641484581s" totalImagesPullingTime="1.049163747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:22.639712837 +0000 UTC m=+2502.065319320" watchObservedRunningTime="2026-11-03 02:00:22.641484581 +0000 UTC m=+2502.067091067"
Nov 03 02:00:22.692487 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:22.692454    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212d6a7a-fe8d-492f-b509-7044ba779aeb" path="/var/lib/kubelet/pods/212d6a7a-fe8d-492f-b509-7044ba779aeb/volumes"
Nov 03 02:00:22.692876 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:22.692863    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" path="/var/lib/kubelet/pods/4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f/volumes"
Nov 03 02:00:22.693253 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:22.693233    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48146943-2c15-4c2c-be1f-b7d340ec713c" path="/var/lib/kubelet/pods/48146943-2c15-4c2c-be1f-b7d340ec713c/volumes"
Nov 03 02:00:22.693920 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:22.693905    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed1fa2e-f822-4105-8b59-4da996a8ce65" path="/var/lib/kubelet/pods/9ed1fa2e-f822-4105-8b59-4da996a8ce65/volumes"
Nov 03 02:00:25.337852 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:25.337826    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"
Nov 03 02:00:26.352630 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:26.352601    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"
Nov 03 02:00:28.542593 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:28.542563    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:00:30.884029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.883997    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"]
Nov 03 02:00:30.884538 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884508    5508 cpu_manager.go:414] "containerMap: removing container" podUID="48146943-2c15-4c2c-be1f-b7d340ec713c" containerName="cleanup"
Nov 03 02:00:30.884538 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884519    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48146943-2c15-4c2c-be1f-b7d340ec713c" containerName="cleanup"
Nov 03 02:00:30.884538 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884535    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9ed1fa2e-f822-4105-8b59-4da996a8ce65" containerName="payload-processing"
Nov 03 02:00:30.884538 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884540    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed1fa2e-f822-4105-8b59-4da996a8ce65" containerName="payload-processing"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884551    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0ccb7d48-53d7-4fe6-aa77-c1675128fdc2" containerName="payload-pre-processing"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884557    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ccb7d48-53d7-4fe6-aa77-c1675128fdc2" containerName="payload-pre-processing"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884566    5508 cpu_manager.go:414] "containerMap: removing container" podUID="212d6a7a-fe8d-492f-b509-7044ba779aeb" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884571    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212d6a7a-fe8d-492f-b509-7044ba779aeb" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884577    5508 cpu_manager.go:414] "containerMap: removing container" podUID="48146943-2c15-4c2c-be1f-b7d340ec713c" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884582    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48146943-2c15-4c2c-be1f-b7d340ec713c" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884590    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884594    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884604    5508 cpu_manager.go:414] "containerMap: removing container" podUID="48146943-2c15-4c2c-be1f-b7d340ec713c" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884607    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48146943-2c15-4c2c-be1f-b7d340ec713c" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884635    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0d426eea-4c34-46ea-98b6-3235a0cbacff" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884639    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d426eea-4c34-46ea-98b6-3235a0cbacff" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884646    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884650    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884655    5508 cpu_manager.go:414] "containerMap: removing container" podUID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884659    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884666    5508 cpu_manager.go:414] "containerMap: removing container" podUID="212d6a7a-fe8d-492f-b509-7044ba779aeb" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884670    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212d6a7a-fe8d-492f-b509-7044ba779aeb" containerName="cleanup"
Nov 03 02:00:30.884749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884759    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="48146943-2c15-4c2c-be1f-b7d340ec713c" containerName="cleanup"
Nov 03 02:00:30.885656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884769    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="212d6a7a-fe8d-492f-b509-7044ba779aeb" containerName="cleanup"
Nov 03 02:00:30.885656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884777    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" containerName="cleanup"
Nov 03 02:00:30.885656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884783    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d426eea-4c34-46ea-98b6-3235a0cbacff" containerName="cleanup"
Nov 03 02:00:30.885656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884795    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed1fa2e-f822-4105-8b59-4da996a8ce65" containerName="payload-processing"
Nov 03 02:00:30.885656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884805    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ccb7d48-53d7-4fe6-aa77-c1675128fdc2" containerName="payload-pre-processing"
Nov 03 02:00:30.885656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884812    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="212d6a7a-fe8d-492f-b509-7044ba779aeb" containerName="cleanup"
Nov 03 02:00:30.885656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884818    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="48146943-2c15-4c2c-be1f-b7d340ec713c" containerName="cleanup"
Nov 03 02:00:30.885656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884822    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa34f9b7-d0ee-4eed-ab87-6d9d856c9f7c" containerName="cleanup"
Nov 03 02:00:30.885656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.884827    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" containerName="cleanup"
Nov 03 02:00:30.889901 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.889880    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:30.938846 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.938825    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-db583d-serving-cert\""
Nov 03 02:00:30.968859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.968832    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gbh4\" (UniqueName: \"kubernetes.io/projected/c61eafee-cfd7-41d6-825e-1e358e76ddc2-kube-api-access-9gbh4\") pod \"maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:30.969041 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.968864    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-metrics-certs\") pod \"maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:30.969041 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.968987    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:30.969041 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.969031    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61eafee-cfd7-41d6-825e-1e358e76ddc2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:30.988163 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:30.988131    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"]
Nov 03 02:00:31.070551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:31.070466    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61eafee-cfd7-41d6-825e-1e358e76ddc2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:31.070692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:31.070581    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gbh4\" (UniqueName: \"kubernetes.io/projected/c61eafee-cfd7-41d6-825e-1e358e76ddc2-kube-api-access-9gbh4\") pod \"maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:31.070692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:31.070603    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-metrics-certs\") pod \"maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:31.070692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:31.070671    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:31.070823 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:31.070789    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-serving-cert: secret "maas-api-e2e-ipp-b-db583d-serving-cert" not found
Nov 03 02:00:31.070891 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:31.070877    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-maas-api-tls podName:c61eafee-cfd7-41d6-825e-1e358e76ddc2 nodeName:}" failed. No retries permitted until 2026-11-03 02:00:31.570856763 +0000 UTC m=+2510.996463215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-maas-api-tls") pod "maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b" (UID: "c61eafee-cfd7-41d6-825e-1e358e76ddc2") : secret "maas-api-e2e-ipp-b-db583d-serving-cert" not found
Nov 03 02:00:31.071236 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:31.071206    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61eafee-cfd7-41d6-825e-1e358e76ddc2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:31.073003 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:31.072978    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-metrics-certs\") pod \"maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:31.127828 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:31.127799    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gbh4\" (UniqueName: \"kubernetes.io/projected/c61eafee-cfd7-41d6-825e-1e358e76ddc2-kube-api-access-9gbh4\") pod \"maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:31.584315 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:31.584285    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:31.586791 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:31.586765    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:31.808376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:31.808331    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:32.192206 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:32.192170    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"]
Nov 03 02:00:32.199204 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:00:32.199173    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61eafee_cfd7_41d6_825e_1e358e76ddc2.slice/crio-97280fce7de49bb60f6dd60c260ceb19baa64eb833986523ae6daeb372652912 WatchSource:0}: Error finding container 97280fce7de49bb60f6dd60c260ceb19baa64eb833986523ae6daeb372652912: Status 404 returned error can't find the container with id 97280fce7de49bb60f6dd60c260ceb19baa64eb833986523ae6daeb372652912
Nov 03 02:00:32.598838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:32.598755    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b" event={"ID":"c61eafee-cfd7-41d6-825e-1e358e76ddc2","Type":"ContainerStarted","Data":"97280fce7de49bb60f6dd60c260ceb19baa64eb833986523ae6daeb372652912"}
Nov 03 02:00:33.605796 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:33.605704    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b" event={"ID":"c61eafee-cfd7-41d6-825e-1e358e76ddc2","Type":"ContainerStarted","Data":"6d0bfad6aecac1c32cc12adf11fd0a50418b01e605a78834b5688a0353bdffa7"}
Nov 03 02:00:33.606199 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:33.605838    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:33.717816 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:33.717762    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b" podStartSLOduration=2.658772824 podStartE2EDuration="3.717748368s" totalImagesPullingTime="1.058975544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:00:33.71533394 +0000 UTC m=+2513.140940404" watchObservedRunningTime="2026-11-03 02:00:33.717748368 +0000 UTC m=+2513.143354833"
Nov 03 02:00:35.660756 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:35.660722    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:00:39.627888 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:39.627858    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:00:43.811575 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:43.811533    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:00:47.885616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.885539    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"]
Nov 03 02:00:47.886151 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.886134    5508 cpu_manager.go:414] "containerMap: removing container" podUID="212d6a7a-fe8d-492f-b509-7044ba779aeb" containerName="cleanup"
Nov 03 02:00:47.886151 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.886151    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212d6a7a-fe8d-492f-b509-7044ba779aeb" containerName="cleanup"
Nov 03 02:00:47.886569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.886175    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" containerName="cleanup"
Nov 03 02:00:47.886569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.886181    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" containerName="cleanup"
Nov 03 02:00:47.886569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.886253    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="212d6a7a-fe8d-492f-b509-7044ba779aeb" containerName="cleanup"
Nov 03 02:00:47.886569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.886263    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4328ed5a-ab0c-48e8-a9bd-846e3e63ec1f" containerName="cleanup"
Nov 03 02:00:47.886569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.886277    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="48146943-2c15-4c2c-be1f-b7d340ec713c" containerName="cleanup"
Nov 03 02:00:47.891367 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.891326    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"
Nov 03 02:00:47.940303 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.940272    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:00:47.986636 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.986601    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:00:47.989613 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.989589    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"]
Nov 03 02:00:47.996140 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.996121    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r\" (UID: \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"
Nov 03 02:00:47.996291 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.996272    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5l46\" (UniqueName: \"kubernetes.io/projected/b5ad1411-1fa4-4f96-9133-aec8efe8b530-kube-api-access-k5l46\") pod \"maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r\" (UID: \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"
Nov 03 02:00:47.996370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:47.996338    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r\" (UID: \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"
Nov 03 02:00:48.097648 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:48.097608    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5l46\" (UniqueName: \"kubernetes.io/projected/b5ad1411-1fa4-4f96-9133-aec8efe8b530-kube-api-access-k5l46\") pod \"maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r\" (UID: \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"
Nov 03 02:00:48.097843 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:48.097705    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r\" (UID: \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"
Nov 03 02:00:48.097843 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:48.097787    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r\" (UID: \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"
Nov 03 02:00:48.098478 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:48.098452    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r\" (UID: \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"
Nov 03 02:00:48.099982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:48.099956    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r\" (UID: \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"
Nov 03 02:00:48.155459 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:48.155429    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5l46\" (UniqueName: \"kubernetes.io/projected/b5ad1411-1fa4-4f96-9133-aec8efe8b530-kube-api-access-k5l46\") pod \"maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r\" (UID: \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"
Nov 03 02:00:48.205437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:48.205402    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"
Nov 03 02:00:48.578066 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:48.577992    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"]
Nov 03 02:00:48.589864 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:00:48.589823    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ad1411_1fa4_4f96_9133_aec8efe8b530.slice/crio-0c1611d34b3d823197eaef679792b0beac677a2b907b6f66b3edc087c0fac698 WatchSource:0}: Error finding container 0c1611d34b3d823197eaef679792b0beac677a2b907b6f66b3edc087c0fac698: Status 404 returned error can't find the container with id 0c1611d34b3d823197eaef679792b0beac677a2b907b6f66b3edc087c0fac698
Nov 03 02:00:48.714805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:48.714762    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r" event={"ID":"b5ad1411-1fa4-4f96-9133-aec8efe8b530","Type":"ContainerStarted","Data":"0c1611d34b3d823197eaef679792b0beac677a2b907b6f66b3edc087c0fac698"}
Nov 03 02:00:49.723074 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:49.723038    5508 generic.go:356] "Generic (PLEG): container finished" podID="b5ad1411-1fa4-4f96-9133-aec8efe8b530" containerID="28d08abb496cbd445b9a809f54fabddf0fb716ffc9ab4bf91152a9658be7158c" exitCode=0
Nov 03 02:00:49.723488 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:49.723096    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r" event={"ID":"b5ad1411-1fa4-4f96-9133-aec8efe8b530","Type":"ContainerDied","Data":"28d08abb496cbd445b9a809f54fabddf0fb716ffc9ab4bf91152a9658be7158c"}
Nov 03 02:00:50.887091 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:50.887061    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"
Nov 03 02:00:51.033301 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:51.033216    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-openshift-ca\") pod \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\" (UID: \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\") "
Nov 03 02:00:51.033473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:51.033408    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ad1411-1fa4-4f96-9133-aec8efe8b530-kube-api-access-k5l46\" (UniqueName: \"kubernetes.io/projected/b5ad1411-1fa4-4f96-9133-aec8efe8b530-kube-api-access-k5l46\") pod \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\" (UID: \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\") "
Nov 03 02:00:51.033473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:51.033450    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-xks-ca\") pod \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\" (UID: \"b5ad1411-1fa4-4f96-9133-aec8efe8b530\") "
Nov 03 02:00:51.033604 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:51.033563    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-openshift-ca" pod "b5ad1411-1fa4-4f96-9133-aec8efe8b530" (UID: "b5ad1411-1fa4-4f96-9133-aec8efe8b530"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:00:51.034015 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:51.033989    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:51.035567 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:51.035539    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ad1411-1fa4-4f96-9133-aec8efe8b530-kube-api-access-k5l46" pod "b5ad1411-1fa4-4f96-9133-aec8efe8b530" (UID: "b5ad1411-1fa4-4f96-9133-aec8efe8b530"). InnerVolumeSpecName "kube-api-access-k5l46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:00:51.035675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:51.035625    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-xks-ca" pod "b5ad1411-1fa4-4f96-9133-aec8efe8b530" (UID: "b5ad1411-1fa4-4f96-9133-aec8efe8b530"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:00:51.134820 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:51.134782    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5l46\" (UniqueName: \"kubernetes.io/projected/b5ad1411-1fa4-4f96-9133-aec8efe8b530-kube-api-access-k5l46\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:51.134820 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:51.134814    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5ad1411-1fa4-4f96-9133-aec8efe8b530-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:51.737014 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:51.736984    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"
Nov 03 02:00:51.737213 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:51.737018    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r" event={"ID":"b5ad1411-1fa4-4f96-9133-aec8efe8b530","Type":"ContainerDied","Data":"0c1611d34b3d823197eaef679792b0beac677a2b907b6f66b3edc087c0fac698"}
Nov 03 02:00:51.737213 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:51.737065    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c1611d34b3d823197eaef679792b0beac677a2b907b6f66b3edc087c0fac698"
Nov 03 02:00:57.186838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:57.186803    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"]
Nov 03 02:00:57.187330 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:57.187049    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx" podUID="9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" containerName="istio-proxy" containerID="cri-o://06212159354a4f6e7f334edb6d69fef2240eb27e429495059907fcc3aeba1ff6" gracePeriod=30
Nov 03 02:00:59.467944 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.467906    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"]
Nov 03 02:00:59.468372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.468174    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng" podUID="019e9d63-5cc5-4a76-8079-b27929c4174d" containerName="maas-api" containerID="cri-o://ea963a7e3662fd3393c2eafe7b5af97c335ae8c100c2f5da9b28f0ee91f896ff" gracePeriod=30
Nov 03 02:00:59.760643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.760610    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 02:00:59.793376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.793324    5508 generic.go:356] "Generic (PLEG): container finished" podID="019e9d63-5cc5-4a76-8079-b27929c4174d" containerID="ea963a7e3662fd3393c2eafe7b5af97c335ae8c100c2f5da9b28f0ee91f896ff" exitCode=0
Nov 03 02:00:59.793376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.793381    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"
Nov 03 02:00:59.793581 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.793415    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng" event={"ID":"019e9d63-5cc5-4a76-8079-b27929c4174d","Type":"ContainerDied","Data":"ea963a7e3662fd3393c2eafe7b5af97c335ae8c100c2f5da9b28f0ee91f896ff"}
Nov 03 02:00:59.793581 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.793480    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng" event={"ID":"019e9d63-5cc5-4a76-8079-b27929c4174d","Type":"ContainerDied","Data":"999b9308ee17ddcd70b0aa9cf6f0465df009615209e2de504e7ed25f8db967d7"}
Nov 03 02:00:59.793581 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.793510    5508 scope.go:122] "RemoveContainer" containerID="ea963a7e3662fd3393c2eafe7b5af97c335ae8c100c2f5da9b28f0ee91f896ff"
Nov 03 02:00:59.802990 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.802973    5508 scope.go:122] "RemoveContainer" containerID="ea963a7e3662fd3393c2eafe7b5af97c335ae8c100c2f5da9b28f0ee91f896ff"
Nov 03 02:00:59.803275 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:00:59.803256    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea963a7e3662fd3393c2eafe7b5af97c335ae8c100c2f5da9b28f0ee91f896ff\": container with ID starting with ea963a7e3662fd3393c2eafe7b5af97c335ae8c100c2f5da9b28f0ee91f896ff not found: ID does not exist" containerID="ea963a7e3662fd3393c2eafe7b5af97c335ae8c100c2f5da9b28f0ee91f896ff"
Nov 03 02:00:59.803387 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.803284    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea963a7e3662fd3393c2eafe7b5af97c335ae8c100c2f5da9b28f0ee91f896ff"} err="failed to get container status \"ea963a7e3662fd3393c2eafe7b5af97c335ae8c100c2f5da9b28f0ee91f896ff\": rpc error: code = NotFound desc = could not find container \"ea963a7e3662fd3393c2eafe7b5af97c335ae8c100c2f5da9b28f0ee91f896ff\": container with ID starting with ea963a7e3662fd3393c2eafe7b5af97c335ae8c100c2f5da9b28f0ee91f896ff not found: ID does not exist"
Nov 03 02:00:59.941134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.941096    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-metrics-certs\") pod \"019e9d63-5cc5-4a76-8079-b27929c4174d\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") "
Nov 03 02:00:59.941325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.941206    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-maas-api-tls\") pod \"019e9d63-5cc5-4a76-8079-b27929c4174d\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") "
Nov 03 02:00:59.941432 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.941414    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/019e9d63-5cc5-4a76-8079-b27929c4174d-kube-api-access-45p5q\" (UniqueName: \"kubernetes.io/projected/019e9d63-5cc5-4a76-8079-b27929c4174d-kube-api-access-45p5q\") pod \"019e9d63-5cc5-4a76-8079-b27929c4174d\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") "
Nov 03 02:00:59.941488 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.941450    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/019e9d63-5cc5-4a76-8079-b27929c4174d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/019e9d63-5cc5-4a76-8079-b27929c4174d-openshift-service-ca\") pod \"019e9d63-5cc5-4a76-8079-b27929c4174d\" (UID: \"019e9d63-5cc5-4a76-8079-b27929c4174d\") "
Nov 03 02:00:59.941864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.941829    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/019e9d63-5cc5-4a76-8079-b27929c4174d-openshift-service-ca" pod "019e9d63-5cc5-4a76-8079-b27929c4174d" (UID: "019e9d63-5cc5-4a76-8079-b27929c4174d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:00:59.941994 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.941915    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/019e9d63-5cc5-4a76-8079-b27929c4174d-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:00:59.943344 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.943314    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-metrics-certs" pod "019e9d63-5cc5-4a76-8079-b27929c4174d" (UID: "019e9d63-5cc5-4a76-8079-b27929c4174d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:00:59.943344 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.943374    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-maas-api-tls" pod "019e9d63-5cc5-4a76-8079-b27929c4174d" (UID: "019e9d63-5cc5-4a76-8079-b27929c4174d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:00:59.943551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:00:59.943416    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019e9d63-5cc5-4a76-8079-b27929c4174d-kube-api-access-45p5q" pod "019e9d63-5cc5-4a76-8079-b27929c4174d" (UID: "019e9d63-5cc5-4a76-8079-b27929c4174d"). InnerVolumeSpecName "kube-api-access-45p5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:00.043239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:00.043148    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:00.043239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:00.043188    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45p5q\" (UniqueName: \"kubernetes.io/projected/019e9d63-5cc5-4a76-8079-b27929c4174d-kube-api-access-45p5q\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:00.043239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:00.043202    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/019e9d63-5cc5-4a76-8079-b27929c4174d-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:00.208409 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:00.208368    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"]
Nov 03 02:01:00.212058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:00.212029    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-706bdd-6574c9955f-dd9ng"]
Nov 03 02:01:00.694028 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:00.693995    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="019e9d63-5cc5-4a76-8079-b27929c4174d" path="/var/lib/kubelet/pods/019e9d63-5cc5-4a76-8079-b27929c4174d/volumes"
Nov 03 02:01:00.999469 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:00.999370    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"]
Nov 03 02:01:00.999650 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:00.999609    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv" podUID="f9f60d09-e74d-432e-84ef-b74af7c18842" containerName="payload-processing" containerID="cri-o://e0bc29baca70abff81f4ce0127576b10bec3d5830c7b229dbe61e16fe145f351" gracePeriod=30
Nov 03 02:01:01.047357 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.047310    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"]
Nov 03 02:01:01.047566 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.047543    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc" podUID="f93be790-6233-4f77-b8c5-5f289079cd50" containerName="payload-pre-processing" containerID="cri-o://feb361101af9cce6df95c418560ab3b90fd9a87dd13925b97b3aff330d7c1b1a" gracePeriod=30
Nov 03 02:01:01.337528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.330414    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"
Nov 03 02:01:01.349320 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.349300    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"
Nov 03 02:01:01.461681 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.461640    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f60d09-e74d-432e-84ef-b74af7c18842-kube-api-access-wwntw\" (UniqueName: \"kubernetes.io/projected/f9f60d09-e74d-432e-84ef-b74af7c18842-kube-api-access-wwntw\") pod \"f9f60d09-e74d-432e-84ef-b74af7c18842\" (UID: \"f9f60d09-e74d-432e-84ef-b74af7c18842\") "
Nov 03 02:01:01.461899 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.461705    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93be790-6233-4f77-b8c5-5f289079cd50-kube-api-access-5df5j\" (UniqueName: \"kubernetes.io/projected/f93be790-6233-4f77-b8c5-5f289079cd50-kube-api-access-5df5j\") pod \"f93be790-6233-4f77-b8c5-5f289079cd50\" (UID: \"f93be790-6233-4f77-b8c5-5f289079cd50\") "
Nov 03 02:01:01.461979 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.461932    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9f60d09-e74d-432e-84ef-b74af7c18842-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f60d09-e74d-432e-84ef-b74af7c18842-plugins-config-volume\") pod \"f9f60d09-e74d-432e-84ef-b74af7c18842\" (UID: \"f9f60d09-e74d-432e-84ef-b74af7c18842\") "
Nov 03 02:01:01.462408 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.462333    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f60d09-e74d-432e-84ef-b74af7c18842-plugins-config-volume" pod "f9f60d09-e74d-432e-84ef-b74af7c18842" (UID: "f9f60d09-e74d-432e-84ef-b74af7c18842"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:01.462545 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.462499    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f93be790-6233-4f77-b8c5-5f289079cd50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93be790-6233-4f77-b8c5-5f289079cd50-plugins-config-volume\") pod \"f93be790-6233-4f77-b8c5-5f289079cd50\" (UID: \"f93be790-6233-4f77-b8c5-5f289079cd50\") "
Nov 03 02:01:01.462931 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.462909    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93be790-6233-4f77-b8c5-5f289079cd50-plugins-config-volume" pod "f93be790-6233-4f77-b8c5-5f289079cd50" (UID: "f93be790-6233-4f77-b8c5-5f289079cd50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:01.463445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.463415    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f60d09-e74d-432e-84ef-b74af7c18842-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:01.463445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.463442    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93be790-6233-4f77-b8c5-5f289079cd50-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:01.463990 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.463964    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f60d09-e74d-432e-84ef-b74af7c18842-kube-api-access-wwntw" pod "f9f60d09-e74d-432e-84ef-b74af7c18842" (UID: "f9f60d09-e74d-432e-84ef-b74af7c18842"). InnerVolumeSpecName "kube-api-access-wwntw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:01.464134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.464115    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93be790-6233-4f77-b8c5-5f289079cd50-kube-api-access-5df5j" pod "f93be790-6233-4f77-b8c5-5f289079cd50" (UID: "f93be790-6233-4f77-b8c5-5f289079cd50"). InnerVolumeSpecName "kube-api-access-5df5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:01.573839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.573362    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwntw\" (UniqueName: \"kubernetes.io/projected/f9f60d09-e74d-432e-84ef-b74af7c18842-kube-api-access-wwntw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:01.573839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.573402    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5df5j\" (UniqueName: \"kubernetes.io/projected/f93be790-6233-4f77-b8c5-5f289079cd50-kube-api-access-5df5j\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:01.703422 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.703385    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"]
Nov 03 02:01:01.703896 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.703588    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk" podUID="ebea57f5-714b-454d-843c-5cc7d546abcd" containerName="maas-api" containerID="cri-o://bd2bd39e59f1b6ac95ac6bb54a4503de594067f2c76546288ba68deb7e827c66" gracePeriod=30
Nov 03 02:01:01.820632 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.820603    5508 generic.go:356] "Generic (PLEG): container finished" podID="f9f60d09-e74d-432e-84ef-b74af7c18842" containerID="e0bc29baca70abff81f4ce0127576b10bec3d5830c7b229dbe61e16fe145f351" exitCode=0
Nov 03 02:01:01.820798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.820651    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"
Nov 03 02:01:01.820798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.820682    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv" event={"ID":"f9f60d09-e74d-432e-84ef-b74af7c18842","Type":"ContainerDied","Data":"e0bc29baca70abff81f4ce0127576b10bec3d5830c7b229dbe61e16fe145f351"}
Nov 03 02:01:01.820798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.820737    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv" event={"ID":"f9f60d09-e74d-432e-84ef-b74af7c18842","Type":"ContainerDied","Data":"b9482f254fe384b58aece619416e19b30f156bc80b99da533f8a739852f76c6c"}
Nov 03 02:01:01.820798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.820759    5508 scope.go:122] "RemoveContainer" containerID="e0bc29baca70abff81f4ce0127576b10bec3d5830c7b229dbe61e16fe145f351"
Nov 03 02:01:01.822256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.822233    5508 generic.go:356] "Generic (PLEG): container finished" podID="f93be790-6233-4f77-b8c5-5f289079cd50" containerID="feb361101af9cce6df95c418560ab3b90fd9a87dd13925b97b3aff330d7c1b1a" exitCode=0
Nov 03 02:01:01.822455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.822270    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"
Nov 03 02:01:01.822455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.822275    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc" event={"ID":"f93be790-6233-4f77-b8c5-5f289079cd50","Type":"ContainerDied","Data":"feb361101af9cce6df95c418560ab3b90fd9a87dd13925b97b3aff330d7c1b1a"}
Nov 03 02:01:01.822455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.822315    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc" event={"ID":"f93be790-6233-4f77-b8c5-5f289079cd50","Type":"ContainerDied","Data":"7fd0f77e8b8a316051a8d29ec4128e908c452f0b0f3329c9fceb5ca63cf49fab"}
Nov 03 02:01:01.832480 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.832445    5508 scope.go:122] "RemoveContainer" containerID="e0bc29baca70abff81f4ce0127576b10bec3d5830c7b229dbe61e16fe145f351"
Nov 03 02:01:01.832805 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:01:01.832780    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0bc29baca70abff81f4ce0127576b10bec3d5830c7b229dbe61e16fe145f351\": container with ID starting with e0bc29baca70abff81f4ce0127576b10bec3d5830c7b229dbe61e16fe145f351 not found: ID does not exist" containerID="e0bc29baca70abff81f4ce0127576b10bec3d5830c7b229dbe61e16fe145f351"
Nov 03 02:01:01.832911 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.832813    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0bc29baca70abff81f4ce0127576b10bec3d5830c7b229dbe61e16fe145f351"} err="failed to get container status \"e0bc29baca70abff81f4ce0127576b10bec3d5830c7b229dbe61e16fe145f351\": rpc error: code = NotFound desc = could not find container \"e0bc29baca70abff81f4ce0127576b10bec3d5830c7b229dbe61e16fe145f351\": container with ID starting with e0bc29baca70abff81f4ce0127576b10bec3d5830c7b229dbe61e16fe145f351 not found: ID does not exist"
Nov 03 02:01:01.832911 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.832834    5508 scope.go:122] "RemoveContainer" containerID="feb361101af9cce6df95c418560ab3b90fd9a87dd13925b97b3aff330d7c1b1a"
Nov 03 02:01:01.842312 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.842294    5508 scope.go:122] "RemoveContainer" containerID="feb361101af9cce6df95c418560ab3b90fd9a87dd13925b97b3aff330d7c1b1a"
Nov 03 02:01:01.842588 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:01:01.842568    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb361101af9cce6df95c418560ab3b90fd9a87dd13925b97b3aff330d7c1b1a\": container with ID starting with feb361101af9cce6df95c418560ab3b90fd9a87dd13925b97b3aff330d7c1b1a not found: ID does not exist" containerID="feb361101af9cce6df95c418560ab3b90fd9a87dd13925b97b3aff330d7c1b1a"
Nov 03 02:01:01.842670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.842597    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb361101af9cce6df95c418560ab3b90fd9a87dd13925b97b3aff330d7c1b1a"} err="failed to get container status \"feb361101af9cce6df95c418560ab3b90fd9a87dd13925b97b3aff330d7c1b1a\": rpc error: code = NotFound desc = could not find container \"feb361101af9cce6df95c418560ab3b90fd9a87dd13925b97b3aff330d7c1b1a\": container with ID starting with feb361101af9cce6df95c418560ab3b90fd9a87dd13925b97b3aff330d7c1b1a not found: ID does not exist"
Nov 03 02:01:01.985199 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.985164    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"]
Nov 03 02:01:01.989287 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:01.989256    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b177c9-664864d57f-mkgjc"]
Nov 03 02:01:02.003449 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.003429    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 02:01:02.086454 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.083706    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebea57f5-714b-454d-843c-5cc7d546abcd-kube-api-access-lmm2b\" (UniqueName: \"kubernetes.io/projected/ebea57f5-714b-454d-843c-5cc7d546abcd-kube-api-access-lmm2b\") pod \"ebea57f5-714b-454d-843c-5cc7d546abcd\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") "
Nov 03 02:01:02.086454 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.083791    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-maas-api-tls\") pod \"ebea57f5-714b-454d-843c-5cc7d546abcd\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") "
Nov 03 02:01:02.086454 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.083867    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-metrics-certs\") pod \"ebea57f5-714b-454d-843c-5cc7d546abcd\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") "
Nov 03 02:01:02.086454 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.084029    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebea57f5-714b-454d-843c-5cc7d546abcd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebea57f5-714b-454d-843c-5cc7d546abcd-openshift-service-ca\") pod \"ebea57f5-714b-454d-843c-5cc7d546abcd\" (UID: \"ebea57f5-714b-454d-843c-5cc7d546abcd\") "
Nov 03 02:01:02.086454 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.084783    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebea57f5-714b-454d-843c-5cc7d546abcd-openshift-service-ca" pod "ebea57f5-714b-454d-843c-5cc7d546abcd" (UID: "ebea57f5-714b-454d-843c-5cc7d546abcd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:02.098761 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.092841    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebea57f5-714b-454d-843c-5cc7d546abcd-kube-api-access-lmm2b" pod "ebea57f5-714b-454d-843c-5cc7d546abcd" (UID: "ebea57f5-714b-454d-843c-5cc7d546abcd"). InnerVolumeSpecName "kube-api-access-lmm2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:02.106856 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.102655    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"]
Nov 03 02:01:02.106856 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.102822    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-maas-api-tls" pod "ebea57f5-714b-454d-843c-5cc7d546abcd" (UID: "ebea57f5-714b-454d-843c-5cc7d546abcd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:01:02.107103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.107076    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b177c9-7bc9dbbb9b-gn8zv"]
Nov 03 02:01:02.110063 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.110011    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-metrics-certs" pod "ebea57f5-714b-454d-843c-5cc7d546abcd" (UID: "ebea57f5-714b-454d-843c-5cc7d546abcd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:01:02.186248 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.186197    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebea57f5-714b-454d-843c-5cc7d546abcd-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:02.186248 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.186238    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmm2b\" (UniqueName: \"kubernetes.io/projected/ebea57f5-714b-454d-843c-5cc7d546abcd-kube-api-access-lmm2b\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:02.186578 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.186268    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:02.186578 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.186282    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebea57f5-714b-454d-843c-5cc7d546abcd-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:02.451994 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.451962    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 02:01:02.531970 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.531935    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"]
Nov 03 02:01:02.554882 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.554852    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b177c9-rhl4r"]
Nov 03 02:01:02.604586 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.604382    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-data" pod "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" (UID: "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:02.608447 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.598138    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-data\") pod \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") "
Nov 03 02:01:02.608698 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.608681    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-podinfo\") pod \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") "
Nov 03 02:01:02.608983 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.608964    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-socket\") pod \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") "
Nov 03 02:01:02.609098 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.609085    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istiod-ca-cert\") pod \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") "
Nov 03 02:01:02.609218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.609204    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-credential-socket\") pod \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") "
Nov 03 02:01:02.609384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.609368    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-kube-api-access-m2jnt\" (UniqueName: \"kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-kube-api-access-m2jnt\") pod \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") "
Nov 03 02:01:02.609530 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.609513    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-certs\") pod \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") "
Nov 03 02:01:02.609674 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.609662    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-envoy\") pod \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") "
Nov 03 02:01:02.609789 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.609777    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-token\" (UniqueName: \"kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-token\") pod \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\" (UID: \"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c\") "
Nov 03 02:01:02.610908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.610311    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:02.619831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.610593    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-socket" pod "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" (UID: "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:02.619831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.611090    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-certs" pod "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" (UID: "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:02.619831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.611633    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istiod-ca-cert" pod "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" (UID: "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:02.619831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.611839    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-credential-socket" pod "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" (UID: "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:02.663368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.645654    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-envoy" pod "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" (UID: "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:02.663368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.646131    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-podinfo" pod "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" (UID: "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:01:02.663368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.646275    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-kube-api-access-m2jnt" pod "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" (UID: "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c"). InnerVolumeSpecName "kube-api-access-m2jnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:02.663368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.648336    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-token" pod "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" (UID: "9a0e9d58-d98f-4c20-bb5e-c094ff18a48c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:02.698866 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.698830    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5ad1411-1fa4-4f96-9133-aec8efe8b530" path="/var/lib/kubelet/pods/b5ad1411-1fa4-4f96-9133-aec8efe8b530/volumes"
Nov 03 02:01:02.699649 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.699626    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93be790-6233-4f77-b8c5-5f289079cd50" path="/var/lib/kubelet/pods/f93be790-6233-4f77-b8c5-5f289079cd50/volumes"
Nov 03 02:01:02.700237 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.700221    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9f60d09-e74d-432e-84ef-b74af7c18842" path="/var/lib/kubelet/pods/f9f60d09-e74d-432e-84ef-b74af7c18842/volumes"
Nov 03 02:01:02.720749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.720729    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2jnt\" (UniqueName: \"kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-kube-api-access-m2jnt\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:02.721005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.720750    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:02.721005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.720761    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:02.721005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.720770    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:02.721005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.720779    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:02.721005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.720787    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:02.721005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.720796    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:02.721005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.720805    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:02.831052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.831011    5508 generic.go:356] "Generic (PLEG): container finished" podID="ebea57f5-714b-454d-843c-5cc7d546abcd" containerID="bd2bd39e59f1b6ac95ac6bb54a4503de594067f2c76546288ba68deb7e827c66" exitCode=0
Nov 03 02:01:02.831243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.831089    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"
Nov 03 02:01:02.831243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.831104    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk" event={"ID":"ebea57f5-714b-454d-843c-5cc7d546abcd","Type":"ContainerDied","Data":"bd2bd39e59f1b6ac95ac6bb54a4503de594067f2c76546288ba68deb7e827c66"}
Nov 03 02:01:02.831243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.831165    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk" event={"ID":"ebea57f5-714b-454d-843c-5cc7d546abcd","Type":"ContainerDied","Data":"5781204e0411fe7540656e667cd0c6a65112b55b2affa896a7ebe410eeb24af3"}
Nov 03 02:01:02.831243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.831193    5508 scope.go:122] "RemoveContainer" containerID="bd2bd39e59f1b6ac95ac6bb54a4503de594067f2c76546288ba68deb7e827c66"
Nov 03 02:01:02.834924 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.834900    5508 generic.go:356] "Generic (PLEG): container finished" podID="9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" containerID="06212159354a4f6e7f334edb6d69fef2240eb27e429495059907fcc3aeba1ff6" exitCode=0
Nov 03 02:01:02.835052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.834960    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"
Nov 03 02:01:02.835052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.834991    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx" event={"ID":"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c","Type":"ContainerDied","Data":"06212159354a4f6e7f334edb6d69fef2240eb27e429495059907fcc3aeba1ff6"}
Nov 03 02:01:02.835052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.835027    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx" event={"ID":"9a0e9d58-d98f-4c20-bb5e-c094ff18a48c","Type":"ContainerDied","Data":"429cb121da4f6c69bff3e88e59b5f986dc797fb5130cea01ca89bd40584c201e"}
Nov 03 02:01:02.844505 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.844485    5508 scope.go:122] "RemoveContainer" containerID="bd2bd39e59f1b6ac95ac6bb54a4503de594067f2c76546288ba68deb7e827c66"
Nov 03 02:01:02.845132 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:01:02.845093    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2bd39e59f1b6ac95ac6bb54a4503de594067f2c76546288ba68deb7e827c66\": container with ID starting with bd2bd39e59f1b6ac95ac6bb54a4503de594067f2c76546288ba68deb7e827c66 not found: ID does not exist" containerID="bd2bd39e59f1b6ac95ac6bb54a4503de594067f2c76546288ba68deb7e827c66"
Nov 03 02:01:02.845214 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.845140    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2bd39e59f1b6ac95ac6bb54a4503de594067f2c76546288ba68deb7e827c66"} err="failed to get container status \"bd2bd39e59f1b6ac95ac6bb54a4503de594067f2c76546288ba68deb7e827c66\": rpc error: code = NotFound desc = could not find container \"bd2bd39e59f1b6ac95ac6bb54a4503de594067f2c76546288ba68deb7e827c66\": container with ID starting with bd2bd39e59f1b6ac95ac6bb54a4503de594067f2c76546288ba68deb7e827c66 not found: ID does not exist"
Nov 03 02:01:02.845214 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.845156    5508 scope.go:122] "RemoveContainer" containerID="06212159354a4f6e7f334edb6d69fef2240eb27e429495059907fcc3aeba1ff6"
Nov 03 02:01:02.853749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.853729    5508 scope.go:122] "RemoveContainer" containerID="06212159354a4f6e7f334edb6d69fef2240eb27e429495059907fcc3aeba1ff6"
Nov 03 02:01:02.854020 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:01:02.854002    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06212159354a4f6e7f334edb6d69fef2240eb27e429495059907fcc3aeba1ff6\": container with ID starting with 06212159354a4f6e7f334edb6d69fef2240eb27e429495059907fcc3aeba1ff6 not found: ID does not exist" containerID="06212159354a4f6e7f334edb6d69fef2240eb27e429495059907fcc3aeba1ff6"
Nov 03 02:01:02.854086 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.854026    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06212159354a4f6e7f334edb6d69fef2240eb27e429495059907fcc3aeba1ff6"} err="failed to get container status \"06212159354a4f6e7f334edb6d69fef2240eb27e429495059907fcc3aeba1ff6\": rpc error: code = NotFound desc = could not find container \"06212159354a4f6e7f334edb6d69fef2240eb27e429495059907fcc3aeba1ff6\": container with ID starting with 06212159354a4f6e7f334edb6d69fef2240eb27e429495059907fcc3aeba1ff6 not found: ID does not exist"
Nov 03 02:01:02.994051 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.994011    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"]
Nov 03 02:01:02.997084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:02.997045    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-jv2jx"]
Nov 03 02:01:03.097623 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:03.097592    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"]
Nov 03 02:01:03.102129 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:03.102106    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e2db44-5d9cd5d6cd-65ddk"]
Nov 03 02:01:03.917503 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:03.917465    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:01:04.692439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:04.692405    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" path="/var/lib/kubelet/pods/9a0e9d58-d98f-4c20-bb5e-c094ff18a48c/volumes"
Nov 03 02:01:04.692898 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:04.692884    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebea57f5-714b-454d-843c-5cc7d546abcd" path="/var/lib/kubelet/pods/ebea57f5-714b-454d-843c-5cc7d546abcd/volumes"
Nov 03 02:01:05.725473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:05.725435    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"]
Nov 03 02:01:05.725942 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:05.725696    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5" podUID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerName="istio-proxy" containerID="cri-o://a14d2d85f02c7238787043fc298f857f2bd7dd7054144b0c693befd599b08510" gracePeriod=30
Nov 03 02:01:08.444418 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:08.444386    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"]
Nov 03 02:01:08.444970 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:08.444595    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6" podUID="cc85e883-f97a-417c-b58a-8535213343c0" containerName="istio-proxy" containerID="cri-o://6a284344597eb5cba19515d035dc41167afeb16e432f1869344ca53d638431d3" gracePeriod=30
Nov 03 02:01:10.916049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:10.916020    5508 generic.go:356] "Generic (PLEG): container finished" podID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerID="a14d2d85f02c7238787043fc298f857f2bd7dd7054144b0c693befd599b08510" exitCode=0
Nov 03 02:01:10.916496 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:10.916096    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5" event={"ID":"9762f99e-d6cd-4369-bc2b-e11dbf56a32f","Type":"ContainerDied","Data":"a14d2d85f02c7238787043fc298f857f2bd7dd7054144b0c693befd599b08510"}
Nov 03 02:01:10.916496 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:10.916177    5508 scope.go:122] "RemoveContainer" containerID="c5d098335a95cbc91d7edfe1f95d405e5255285c0b215761616242e1d0ae5c39"
Nov 03 02:01:11.003614 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.003557    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 02:01:11.056672 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.056639    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-certs\") pod \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") "
Nov 03 02:01:11.056846 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.056687    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-envoy\") pod \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") "
Nov 03 02:01:11.056846 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.056746    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-token\" (UniqueName: \"kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-token\") pod \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") "
Nov 03 02:01:11.056846 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.056774    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istiod-ca-cert\") pod \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") "
Nov 03 02:01:11.056846 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.056818    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-socket\") pod \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") "
Nov 03 02:01:11.057076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.056849    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-podinfo\") pod \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") "
Nov 03 02:01:11.057076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.056886    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-data\") pod \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") "
Nov 03 02:01:11.057076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.056924    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-credential-socket\") pod \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") "
Nov 03 02:01:11.057076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.056924    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-certs" pod "9762f99e-d6cd-4369-bc2b-e11dbf56a32f" (UID: "9762f99e-d6cd-4369-bc2b-e11dbf56a32f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:11.057076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.056994    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-kube-api-access-jpshn\" (UniqueName: \"kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-kube-api-access-jpshn\") pod \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\" (UID: \"9762f99e-d6cd-4369-bc2b-e11dbf56a32f\") "
Nov 03 02:01:11.057362 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.057203    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-socket" pod "9762f99e-d6cd-4369-bc2b-e11dbf56a32f" (UID: "9762f99e-d6cd-4369-bc2b-e11dbf56a32f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:11.057362 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.057214    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istiod-ca-cert" pod "9762f99e-d6cd-4369-bc2b-e11dbf56a32f" (UID: "9762f99e-d6cd-4369-bc2b-e11dbf56a32f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:11.057362 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.057300    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-data" pod "9762f99e-d6cd-4369-bc2b-e11dbf56a32f" (UID: "9762f99e-d6cd-4369-bc2b-e11dbf56a32f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:11.057519 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.057497    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-credential-socket" pod "9762f99e-d6cd-4369-bc2b-e11dbf56a32f" (UID: "9762f99e-d6cd-4369-bc2b-e11dbf56a32f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:11.057577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.057516    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:11.057577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.057539    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:11.057577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.057554    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:11.057577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.057570    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:11.059242 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.059217    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-envoy" pod "9762f99e-d6cd-4369-bc2b-e11dbf56a32f" (UID: "9762f99e-d6cd-4369-bc2b-e11dbf56a32f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:11.059455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.059430    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-token" pod "9762f99e-d6cd-4369-bc2b-e11dbf56a32f" (UID: "9762f99e-d6cd-4369-bc2b-e11dbf56a32f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:11.059632 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.059609    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-podinfo" pod "9762f99e-d6cd-4369-bc2b-e11dbf56a32f" (UID: "9762f99e-d6cd-4369-bc2b-e11dbf56a32f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:01:11.059857 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.059834    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-kube-api-access-jpshn" pod "9762f99e-d6cd-4369-bc2b-e11dbf56a32f" (UID: "9762f99e-d6cd-4369-bc2b-e11dbf56a32f"). InnerVolumeSpecName "kube-api-access-jpshn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:11.162989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.159133    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:11.162989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.159180    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:11.162989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.159196    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:11.162989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.159210    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:11.162989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.159224    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpshn\" (UniqueName: \"kubernetes.io/projected/9762f99e-d6cd-4369-bc2b-e11dbf56a32f-kube-api-access-jpshn\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:11.925547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.925517    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"
Nov 03 02:01:11.926040 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.925543    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5" event={"ID":"9762f99e-d6cd-4369-bc2b-e11dbf56a32f","Type":"ContainerDied","Data":"22e56758e90ad8beb198051cd5619a83c574173fccc95d72cafe7dd67ab6dd64"}
Nov 03 02:01:11.926040 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:11.925611    5508 scope.go:122] "RemoveContainer" containerID="a14d2d85f02c7238787043fc298f857f2bd7dd7054144b0c693befd599b08510"
Nov 03 02:01:12.126417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:12.126379    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"]
Nov 03 02:01:12.172816 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:12.172779    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-706bdd-openshift-default-7bcd6575b9-kbbx5"]
Nov 03 02:01:12.694978 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:12.694942    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" path="/var/lib/kubelet/pods/9762f99e-d6cd-4369-bc2b-e11dbf56a32f/volumes"
Nov 03 02:01:13.717068 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.717013    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 02:01:13.790076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790051    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-credential-socket\") pod \"cc85e883-f97a-417c-b58a-8535213343c0\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") "
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790120    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc85e883-f97a-417c-b58a-8535213343c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc85e883-f97a-417c-b58a-8535213343c0-istio-podinfo\") pod \"cc85e883-f97a-417c-b58a-8535213343c0\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") "
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790200    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-socket\") pod \"cc85e883-f97a-417c-b58a-8535213343c0\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") "
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790229    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc85e883-f97a-417c-b58a-8535213343c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc85e883-f97a-417c-b58a-8535213343c0-istiod-ca-cert\") pod \"cc85e883-f97a-417c-b58a-8535213343c0\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") "
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790255    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-certs\") pod \"cc85e883-f97a-417c-b58a-8535213343c0\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") "
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790295    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-kube-api-access-m9ns7\" (UniqueName: \"kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-kube-api-access-m9ns7\") pod \"cc85e883-f97a-417c-b58a-8535213343c0\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") "
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790335    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-envoy\") pod \"cc85e883-f97a-417c-b58a-8535213343c0\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") "
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790383    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-credential-socket" pod "cc85e883-f97a-417c-b58a-8535213343c0" (UID: "cc85e883-f97a-417c-b58a-8535213343c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790403    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-istio-token\" (UniqueName: \"kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-istio-token\") pod \"cc85e883-f97a-417c-b58a-8535213343c0\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") "
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790478    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-data\") pod \"cc85e883-f97a-417c-b58a-8535213343c0\" (UID: \"cc85e883-f97a-417c-b58a-8535213343c0\") "
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790505    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-socket" pod "cc85e883-f97a-417c-b58a-8535213343c0" (UID: "cc85e883-f97a-417c-b58a-8535213343c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790655    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc85e883-f97a-417c-b58a-8535213343c0-istiod-ca-cert" pod "cc85e883-f97a-417c-b58a-8535213343c0" (UID: "cc85e883-f97a-417c-b58a-8535213343c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790871    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-data" pod "cc85e883-f97a-417c-b58a-8535213343c0" (UID: "cc85e883-f97a-417c-b58a-8535213343c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.790937    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-certs" pod "cc85e883-f97a-417c-b58a-8535213343c0" (UID: "cc85e883-f97a-417c-b58a-8535213343c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.791079    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.791101    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.791115    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.791128    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc85e883-f97a-417c-b58a-8535213343c0-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.791141    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.792540    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-istio-token" pod "cc85e883-f97a-417c-b58a-8535213343c0" (UID: "cc85e883-f97a-417c-b58a-8535213343c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.792784    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-envoy" pod "cc85e883-f97a-417c-b58a-8535213343c0" (UID: "cc85e883-f97a-417c-b58a-8535213343c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.792845    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc85e883-f97a-417c-b58a-8535213343c0-istio-podinfo" pod "cc85e883-f97a-417c-b58a-8535213343c0" (UID: "cc85e883-f97a-417c-b58a-8535213343c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:01:13.795606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.792928    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-kube-api-access-m9ns7" pod "cc85e883-f97a-417c-b58a-8535213343c0" (UID: "cc85e883-f97a-417c-b58a-8535213343c0"). InnerVolumeSpecName "kube-api-access-m9ns7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:13.813295 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:01:13.813252    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:01:13.892649 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.892613    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9ns7\" (UniqueName: \"kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-kube-api-access-m9ns7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:13.892799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.892652    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc85e883-f97a-417c-b58a-8535213343c0-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:13.892799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.892669    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc85e883-f97a-417c-b58a-8535213343c0-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:13.892799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.892683    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc85e883-f97a-417c-b58a-8535213343c0-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:13.942181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.942154    5508 generic.go:356] "Generic (PLEG): container finished" podID="cc85e883-f97a-417c-b58a-8535213343c0" containerID="6a284344597eb5cba19515d035dc41167afeb16e432f1869344ca53d638431d3" exitCode=0
Nov 03 02:01:13.942317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.942215    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"
Nov 03 02:01:13.942317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.942236    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6" event={"ID":"cc85e883-f97a-417c-b58a-8535213343c0","Type":"ContainerDied","Data":"6a284344597eb5cba19515d035dc41167afeb16e432f1869344ca53d638431d3"}
Nov 03 02:01:13.942317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.942289    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6" event={"ID":"cc85e883-f97a-417c-b58a-8535213343c0","Type":"ContainerDied","Data":"2c776356943d4f326c85d314c1e767480e8c4dc5a7bb3e6e313a737cf0f586bc"}
Nov 03 02:01:13.942317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.942314    5508 scope.go:122] "RemoveContainer" containerID="6a284344597eb5cba19515d035dc41167afeb16e432f1869344ca53d638431d3"
Nov 03 02:01:13.943984 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.943966    5508 generic.go:356] "Generic (PLEG): container finished" podID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerID="e63bb301ce12690968e0a956756241bfced2c4e100d55aa61091b1901b3aaeb0" exitCode=0
Nov 03 02:01:13.944089 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.944038    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5" event={"ID":"23107beb-9060-480d-9f1b-6ba175e4d4ef","Type":"ContainerDied","Data":"e63bb301ce12690968e0a956756241bfced2c4e100d55aa61091b1901b3aaeb0"}
Nov 03 02:01:13.944592 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.944574    5508 scope.go:122] "RemoveContainer" containerID="e63bb301ce12690968e0a956756241bfced2c4e100d55aa61091b1901b3aaeb0"
Nov 03 02:01:13.946150 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.946122    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:01:13.946253 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.946201    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:01:13.946320 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.946251    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:01:13.970108 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:13.970092    5508 scope.go:122] "RemoveContainer" containerID="e640d927b58acd93605e7b8f5b596db74f6d33f7847ab8864bc0a1dfd763b4eb"
Nov 03 02:01:14.002106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:14.002087    5508 scope.go:122] "RemoveContainer" containerID="6a284344597eb5cba19515d035dc41167afeb16e432f1869344ca53d638431d3"
Nov 03 02:01:14.002416 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:01:14.002392    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a284344597eb5cba19515d035dc41167afeb16e432f1869344ca53d638431d3\": container with ID starting with 6a284344597eb5cba19515d035dc41167afeb16e432f1869344ca53d638431d3 not found: ID does not exist" containerID="6a284344597eb5cba19515d035dc41167afeb16e432f1869344ca53d638431d3"
Nov 03 02:01:14.002495 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:14.002431    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a284344597eb5cba19515d035dc41167afeb16e432f1869344ca53d638431d3"} err="failed to get container status \"6a284344597eb5cba19515d035dc41167afeb16e432f1869344ca53d638431d3\": rpc error: code = NotFound desc = could not find container \"6a284344597eb5cba19515d035dc41167afeb16e432f1869344ca53d638431d3\": container with ID starting with 6a284344597eb5cba19515d035dc41167afeb16e432f1869344ca53d638431d3 not found: ID does not exist"
Nov 03 02:01:14.002495 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:14.002456    5508 scope.go:122] "RemoveContainer" containerID="e640d927b58acd93605e7b8f5b596db74f6d33f7847ab8864bc0a1dfd763b4eb"
Nov 03 02:01:14.002809 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:01:14.002776    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e640d927b58acd93605e7b8f5b596db74f6d33f7847ab8864bc0a1dfd763b4eb\": container with ID starting with e640d927b58acd93605e7b8f5b596db74f6d33f7847ab8864bc0a1dfd763b4eb not found: ID does not exist" containerID="e640d927b58acd93605e7b8f5b596db74f6d33f7847ab8864bc0a1dfd763b4eb"
Nov 03 02:01:14.002870 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:14.002817    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e640d927b58acd93605e7b8f5b596db74f6d33f7847ab8864bc0a1dfd763b4eb"} err="failed to get container status \"e640d927b58acd93605e7b8f5b596db74f6d33f7847ab8864bc0a1dfd763b4eb\": rpc error: code = NotFound desc = could not find container \"e640d927b58acd93605e7b8f5b596db74f6d33f7847ab8864bc0a1dfd763b4eb\": container with ID starting with e640d927b58acd93605e7b8f5b596db74f6d33f7847ab8864bc0a1dfd763b4eb not found: ID does not exist"
Nov 03 02:01:14.208895 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:14.208865    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"]
Nov 03 02:01:14.212445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:14.212417    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e2db44-openshift-default-84987f6989-bkpj6"]
Nov 03 02:01:14.694096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:14.694065    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc85e883-f97a-417c-b58a-8535213343c0" path="/var/lib/kubelet/pods/cc85e883-f97a-417c-b58a-8535213343c0/volumes"
Nov 03 02:01:14.961008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:14.960965    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5" event={"ID":"23107beb-9060-480d-9f1b-6ba175e4d4ef","Type":"ContainerStarted","Data":"515ec6e97f486e250bf0caf2386e52815ca11b99f50d3015795621b501e2d0a6"}
Nov 03 02:01:15.185535 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:15.185504    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 02:01:15.185740 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:15.185716    5508 patch_prober.go:29] interesting pod/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 02:01:15.185787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:15.185771    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5" podUID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 02:01:16.185524 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:16.185487    5508 patch_prober.go:29] interesting pod/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 02:01:16.185928 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:16.185548    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5" podUID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 02:01:17.186389 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:17.186333    5508 patch_prober.go:29] interesting pod/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 02:01:17.186846 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:17.186406    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5" podUID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 02:01:18.186805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:18.186767    5508 patch_prober.go:29] interesting pod/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 02:01:18.187301 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:18.186833    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5" podUID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 02:01:19.185677 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:19.185634    5508 patch_prober.go:29] interesting pod/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Nov 03 02:01:19.185900 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:19.185713    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5" podUID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Nov 03 02:01:20.190704 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:20.190674    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 02:01:20.191123 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:20.191063    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 02:01:20.191775 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:20.191756    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 02:01:24.026854 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.026823    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"]
Nov 03 02:01:24.027282 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027269    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f93be790-6233-4f77-b8c5-5f289079cd50" containerName="payload-pre-processing"
Nov 03 02:01:24.027323 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027282    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93be790-6233-4f77-b8c5-5f289079cd50" containerName="payload-pre-processing"
Nov 03 02:01:24.027323 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027291    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f9f60d09-e74d-432e-84ef-b74af7c18842" containerName="payload-processing"
Nov 03 02:01:24.027323 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027297    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f60d09-e74d-432e-84ef-b74af7c18842" containerName="payload-processing"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027328    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b5ad1411-1fa4-4f96-9133-aec8efe8b530" containerName="revoke-keys"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027333    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ad1411-1fa4-4f96-9133-aec8efe8b530" containerName="revoke-keys"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027369    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" containerName="istio-proxy"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027374    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" containerName="istio-proxy"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027381    5508 cpu_manager.go:414] "containerMap: removing container" podUID="019e9d63-5cc5-4a76-8079-b27929c4174d" containerName="maas-api"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027386    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019e9d63-5cc5-4a76-8079-b27929c4174d" containerName="maas-api"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027391    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ebea57f5-714b-454d-843c-5cc7d546abcd" containerName="maas-api"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027395    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebea57f5-714b-454d-843c-5cc7d546abcd" containerName="maas-api"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027402    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerName="istio-proxy"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027406    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerName="istio-proxy"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027414    5508 cpu_manager.go:414] "containerMap: removing container" podUID="cc85e883-f97a-417c-b58a-8535213343c0" containerName="istio-proxy"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027418    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc85e883-f97a-417c-b58a-8535213343c0" containerName="istio-proxy"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027425    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerName="istio-proxy"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027429    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerName="istio-proxy"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027436    5508 cpu_manager.go:414] "containerMap: removing container" podUID="cc85e883-f97a-417c-b58a-8535213343c0" containerName="istio-proxy"
Nov 03 02:01:24.027453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027440    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc85e883-f97a-417c-b58a-8535213343c0" containerName="istio-proxy"
Nov 03 02:01:24.028112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027510    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ad1411-1fa4-4f96-9133-aec8efe8b530" containerName="revoke-keys"
Nov 03 02:01:24.028112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027519    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0e9d58-d98f-4c20-bb5e-c094ff18a48c" containerName="istio-proxy"
Nov 03 02:01:24.028112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027525    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="019e9d63-5cc5-4a76-8079-b27929c4174d" containerName="maas-api"
Nov 03 02:01:24.028112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027531    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebea57f5-714b-454d-843c-5cc7d546abcd" containerName="maas-api"
Nov 03 02:01:24.028112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027536    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerName="istio-proxy"
Nov 03 02:01:24.028112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027541    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc85e883-f97a-417c-b58a-8535213343c0" containerName="istio-proxy"
Nov 03 02:01:24.028112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027545    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9762f99e-d6cd-4369-bc2b-e11dbf56a32f" containerName="istio-proxy"
Nov 03 02:01:24.028112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027551    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc85e883-f97a-417c-b58a-8535213343c0" containerName="istio-proxy"
Nov 03 02:01:24.028112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027559    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93be790-6233-4f77-b8c5-5f289079cd50" containerName="payload-pre-processing"
Nov 03 02:01:24.028112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.027564    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f60d09-e74d-432e-84ef-b74af7c18842" containerName="payload-processing"
Nov 03 02:01:24.031844 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.031819    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"
Nov 03 02:01:24.080578 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.080552    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:01:24.126683 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.126656    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:01:24.129339 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.129314    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"]
Nov 03 02:01:24.138036 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.138011    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f20122-1247-46aa-88ae-a499568d0969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c5959f-954z6\" (UID: \"75f20122-1247-46aa-88ae-a499568d0969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"
Nov 03 02:01:24.138136 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.138092    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b865q\" (UniqueName: \"kubernetes.io/projected/75f20122-1247-46aa-88ae-a499568d0969-kube-api-access-b865q\") pod \"maas-api-revoke-sub-30c5959f-954z6\" (UID: \"75f20122-1247-46aa-88ae-a499568d0969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"
Nov 03 02:01:24.138179 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.138143    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f20122-1247-46aa-88ae-a499568d0969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c5959f-954z6\" (UID: \"75f20122-1247-46aa-88ae-a499568d0969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"
Nov 03 02:01:24.239247 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.239217    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f20122-1247-46aa-88ae-a499568d0969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c5959f-954z6\" (UID: \"75f20122-1247-46aa-88ae-a499568d0969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"
Nov 03 02:01:24.239411 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.239266    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b865q\" (UniqueName: \"kubernetes.io/projected/75f20122-1247-46aa-88ae-a499568d0969-kube-api-access-b865q\") pod \"maas-api-revoke-sub-30c5959f-954z6\" (UID: \"75f20122-1247-46aa-88ae-a499568d0969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"
Nov 03 02:01:24.239411 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.239312    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f20122-1247-46aa-88ae-a499568d0969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c5959f-954z6\" (UID: \"75f20122-1247-46aa-88ae-a499568d0969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"
Nov 03 02:01:24.239858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.239836    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f20122-1247-46aa-88ae-a499568d0969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c5959f-954z6\" (UID: \"75f20122-1247-46aa-88ae-a499568d0969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"
Nov 03 02:01:24.241597 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.241577    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f20122-1247-46aa-88ae-a499568d0969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c5959f-954z6\" (UID: \"75f20122-1247-46aa-88ae-a499568d0969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"
Nov 03 02:01:24.296116 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.296061    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b865q\" (UniqueName: \"kubernetes.io/projected/75f20122-1247-46aa-88ae-a499568d0969-kube-api-access-b865q\") pod \"maas-api-revoke-sub-30c5959f-954z6\" (UID: \"75f20122-1247-46aa-88ae-a499568d0969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"
Nov 03 02:01:24.344377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.344317    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"
Nov 03 02:01:24.717380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:24.717330    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"]
Nov 03 02:01:24.724277 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:01:24.724249    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f20122_1247_46aa_88ae_a499568d0969.slice/crio-fcefd8f4fd08358b2a33aa95fc312bc3002ef0417757b828a2a4fd3646214413 WatchSource:0}: Error finding container fcefd8f4fd08358b2a33aa95fc312bc3002ef0417757b828a2a4fd3646214413: Status 404 returned error can't find the container with id fcefd8f4fd08358b2a33aa95fc312bc3002ef0417757b828a2a4fd3646214413
Nov 03 02:01:25.039062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:25.038983    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6" event={"ID":"75f20122-1247-46aa-88ae-a499568d0969","Type":"ContainerStarted","Data":"fcefd8f4fd08358b2a33aa95fc312bc3002ef0417757b828a2a4fd3646214413"}
Nov 03 02:01:26.063577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:26.063079    5508 generic.go:356] "Generic (PLEG): container finished" podID="75f20122-1247-46aa-88ae-a499568d0969" containerID="b1421ec386572f391a89498f0116020cf7dbc9eae8438dd5b1c124ac0fc2b4a1" exitCode=0
Nov 03 02:01:26.063577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:26.063178    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6" event={"ID":"75f20122-1247-46aa-88ae-a499568d0969","Type":"ContainerDied","Data":"b1421ec386572f391a89498f0116020cf7dbc9eae8438dd5b1c124ac0fc2b4a1"}
Nov 03 02:01:27.218433 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:27.218409    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"
Nov 03 02:01:27.279392 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:27.279362    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75f20122-1247-46aa-88ae-a499568d0969-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f20122-1247-46aa-88ae-a499568d0969-maas-api-xks-ca\") pod \"75f20122-1247-46aa-88ae-a499568d0969\" (UID: \"75f20122-1247-46aa-88ae-a499568d0969\") "
Nov 03 02:01:27.279657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:27.279445    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f20122-1247-46aa-88ae-a499568d0969-kube-api-access-b865q\" (UniqueName: \"kubernetes.io/projected/75f20122-1247-46aa-88ae-a499568d0969-kube-api-access-b865q\") pod \"75f20122-1247-46aa-88ae-a499568d0969\" (UID: \"75f20122-1247-46aa-88ae-a499568d0969\") "
Nov 03 02:01:27.279657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:27.279486    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f20122-1247-46aa-88ae-a499568d0969-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f20122-1247-46aa-88ae-a499568d0969-maas-api-openshift-ca\") pod \"75f20122-1247-46aa-88ae-a499568d0969\" (UID: \"75f20122-1247-46aa-88ae-a499568d0969\") "
Nov 03 02:01:27.279921 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:27.279887    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f20122-1247-46aa-88ae-a499568d0969-maas-api-openshift-ca" pod "75f20122-1247-46aa-88ae-a499568d0969" (UID: "75f20122-1247-46aa-88ae-a499568d0969"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:27.281541 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:27.281512    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f20122-1247-46aa-88ae-a499568d0969-maas-api-xks-ca" pod "75f20122-1247-46aa-88ae-a499568d0969" (UID: "75f20122-1247-46aa-88ae-a499568d0969"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:01:27.281692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:27.281676    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f20122-1247-46aa-88ae-a499568d0969-kube-api-access-b865q" pod "75f20122-1247-46aa-88ae-a499568d0969" (UID: "75f20122-1247-46aa-88ae-a499568d0969"). InnerVolumeSpecName "kube-api-access-b865q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:27.380959 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:27.380870    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75f20122-1247-46aa-88ae-a499568d0969-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:27.380959 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:27.380899    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75f20122-1247-46aa-88ae-a499568d0969-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:27.380959 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:27.380908    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b865q\" (UniqueName: \"kubernetes.io/projected/75f20122-1247-46aa-88ae-a499568d0969-kube-api-access-b865q\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:28.079105 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:28.079061    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6" event={"ID":"75f20122-1247-46aa-88ae-a499568d0969","Type":"ContainerDied","Data":"fcefd8f4fd08358b2a33aa95fc312bc3002ef0417757b828a2a4fd3646214413"}
Nov 03 02:01:28.079105 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:28.079085    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"
Nov 03 02:01:28.079394 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:28.079125    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcefd8f4fd08358b2a33aa95fc312bc3002ef0417757b828a2a4fd3646214413"
Nov 03 02:01:30.520230 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.520199    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"]
Nov 03 02:01:30.520681 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.520439    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv" podUID="56d79e96-1752-414a-acc6-5b80b18381ab" containerName="payload-processing" containerID="cri-o://de3d8a9a678057363009b9b9b2d0982e815430cafc06766ac04238588098ece0" gracePeriod=30
Nov 03 02:01:30.620720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.620688    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"]
Nov 03 02:01:30.620914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.620892    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl" podUID="66e2845e-0c51-428b-a854-a5a681d43341" containerName="payload-pre-processing" containerID="cri-o://9ef9133121869c6e069b4eb9666b94891f437515f6b16870279f3c87a670d7d7" gracePeriod=30
Nov 03 02:01:30.798928 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.798894    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"
Nov 03 02:01:30.823685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.823653    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d79e96-1752-414a-acc6-5b80b18381ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d79e96-1752-414a-acc6-5b80b18381ab-plugins-config-volume\") pod \"56d79e96-1752-414a-acc6-5b80b18381ab\" (UID: \"56d79e96-1752-414a-acc6-5b80b18381ab\") "
Nov 03 02:01:30.823916 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.823833    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d79e96-1752-414a-acc6-5b80b18381ab-kube-api-access-xpcv8\" (UniqueName: \"kubernetes.io/projected/56d79e96-1752-414a-acc6-5b80b18381ab-kube-api-access-xpcv8\") pod \"56d79e96-1752-414a-acc6-5b80b18381ab\" (UID: \"56d79e96-1752-414a-acc6-5b80b18381ab\") "
Nov 03 02:01:30.824156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.824134    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d79e96-1752-414a-acc6-5b80b18381ab-plugins-config-volume" pod "56d79e96-1752-414a-acc6-5b80b18381ab" (UID: "56d79e96-1752-414a-acc6-5b80b18381ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:30.824317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.824297    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d79e96-1752-414a-acc6-5b80b18381ab-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:30.826914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.826888    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d79e96-1752-414a-acc6-5b80b18381ab-kube-api-access-xpcv8" pod "56d79e96-1752-414a-acc6-5b80b18381ab" (UID: "56d79e96-1752-414a-acc6-5b80b18381ab"). InnerVolumeSpecName "kube-api-access-xpcv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:30.892182 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.892154    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"
Nov 03 02:01:30.925334 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.925301    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e2845e-0c51-428b-a854-a5a681d43341-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e2845e-0c51-428b-a854-a5a681d43341-plugins-config-volume\") pod \"66e2845e-0c51-428b-a854-a5a681d43341\" (UID: \"66e2845e-0c51-428b-a854-a5a681d43341\") "
Nov 03 02:01:30.925527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.925443    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e2845e-0c51-428b-a854-a5a681d43341-kube-api-access-bqtq8\" (UniqueName: \"kubernetes.io/projected/66e2845e-0c51-428b-a854-a5a681d43341-kube-api-access-bqtq8\") pod \"66e2845e-0c51-428b-a854-a5a681d43341\" (UID: \"66e2845e-0c51-428b-a854-a5a681d43341\") "
Nov 03 02:01:30.925739 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.925714    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e2845e-0c51-428b-a854-a5a681d43341-plugins-config-volume" pod "66e2845e-0c51-428b-a854-a5a681d43341" (UID: "66e2845e-0c51-428b-a854-a5a681d43341"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:30.925851 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.925835    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpcv8\" (UniqueName: \"kubernetes.io/projected/56d79e96-1752-414a-acc6-5b80b18381ab-kube-api-access-xpcv8\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:30.925914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.925853    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66e2845e-0c51-428b-a854-a5a681d43341-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:30.927470 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:30.927446    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e2845e-0c51-428b-a854-a5a681d43341-kube-api-access-bqtq8" pod "66e2845e-0c51-428b-a854-a5a681d43341" (UID: "66e2845e-0c51-428b-a854-a5a681d43341"). InnerVolumeSpecName "kube-api-access-bqtq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:31.027448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.027363    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqtq8\" (UniqueName: \"kubernetes.io/projected/66e2845e-0c51-428b-a854-a5a681d43341-kube-api-access-bqtq8\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:31.105752 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.105715    5508 generic.go:356] "Generic (PLEG): container finished" podID="66e2845e-0c51-428b-a854-a5a681d43341" containerID="9ef9133121869c6e069b4eb9666b94891f437515f6b16870279f3c87a670d7d7" exitCode=0
Nov 03 02:01:31.105752 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.105761    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"
Nov 03 02:01:31.106012 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.105796    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl" event={"ID":"66e2845e-0c51-428b-a854-a5a681d43341","Type":"ContainerDied","Data":"9ef9133121869c6e069b4eb9666b94891f437515f6b16870279f3c87a670d7d7"}
Nov 03 02:01:31.106012 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.105857    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl" event={"ID":"66e2845e-0c51-428b-a854-a5a681d43341","Type":"ContainerDied","Data":"7b4d0a4fd714c36b7b1b998683bb3ece3a71aa55df5ed80aa86b9313f758cdc6"}
Nov 03 02:01:31.106012 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.105877    5508 scope.go:122] "RemoveContainer" containerID="9ef9133121869c6e069b4eb9666b94891f437515f6b16870279f3c87a670d7d7"
Nov 03 02:01:31.108181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.107864    5508 generic.go:356] "Generic (PLEG): container finished" podID="56d79e96-1752-414a-acc6-5b80b18381ab" containerID="de3d8a9a678057363009b9b9b2d0982e815430cafc06766ac04238588098ece0" exitCode=0
Nov 03 02:01:31.108181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.107926    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"
Nov 03 02:01:31.108181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.108006    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv" event={"ID":"56d79e96-1752-414a-acc6-5b80b18381ab","Type":"ContainerDied","Data":"de3d8a9a678057363009b9b9b2d0982e815430cafc06766ac04238588098ece0"}
Nov 03 02:01:31.108181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.108048    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv" event={"ID":"56d79e96-1752-414a-acc6-5b80b18381ab","Type":"ContainerDied","Data":"addb02d1a086b0eacb6525d3ff80ab96a990383b0331779bae85372354f996a5"}
Nov 03 02:01:31.118272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.118226    5508 scope.go:122] "RemoveContainer" containerID="9ef9133121869c6e069b4eb9666b94891f437515f6b16870279f3c87a670d7d7"
Nov 03 02:01:31.118636 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:01:31.118614    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef9133121869c6e069b4eb9666b94891f437515f6b16870279f3c87a670d7d7\": container with ID starting with 9ef9133121869c6e069b4eb9666b94891f437515f6b16870279f3c87a670d7d7 not found: ID does not exist" containerID="9ef9133121869c6e069b4eb9666b94891f437515f6b16870279f3c87a670d7d7"
Nov 03 02:01:31.118735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.118647    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef9133121869c6e069b4eb9666b94891f437515f6b16870279f3c87a670d7d7"} err="failed to get container status \"9ef9133121869c6e069b4eb9666b94891f437515f6b16870279f3c87a670d7d7\": rpc error: code = NotFound desc = could not find container \"9ef9133121869c6e069b4eb9666b94891f437515f6b16870279f3c87a670d7d7\": container with ID starting with 9ef9133121869c6e069b4eb9666b94891f437515f6b16870279f3c87a670d7d7 not found: ID does not exist"
Nov 03 02:01:31.118735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.118671    5508 scope.go:122] "RemoveContainer" containerID="de3d8a9a678057363009b9b9b2d0982e815430cafc06766ac04238588098ece0"
Nov 03 02:01:31.127834 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.127818    5508 scope.go:122] "RemoveContainer" containerID="de3d8a9a678057363009b9b9b2d0982e815430cafc06766ac04238588098ece0"
Nov 03 02:01:31.128133 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:01:31.128117    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3d8a9a678057363009b9b9b2d0982e815430cafc06766ac04238588098ece0\": container with ID starting with de3d8a9a678057363009b9b9b2d0982e815430cafc06766ac04238588098ece0 not found: ID does not exist" containerID="de3d8a9a678057363009b9b9b2d0982e815430cafc06766ac04238588098ece0"
Nov 03 02:01:31.128187 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.128144    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3d8a9a678057363009b9b9b2d0982e815430cafc06766ac04238588098ece0"} err="failed to get container status \"de3d8a9a678057363009b9b9b2d0982e815430cafc06766ac04238588098ece0\": rpc error: code = NotFound desc = could not find container \"de3d8a9a678057363009b9b9b2d0982e815430cafc06766ac04238588098ece0\": container with ID starting with de3d8a9a678057363009b9b9b2d0982e815430cafc06766ac04238588098ece0 not found: ID does not exist"
Nov 03 02:01:31.267743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.267700    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"]
Nov 03 02:01:31.271891 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.271854    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738acb-57796f4b44-mqdkv"]
Nov 03 02:01:31.369259 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.369171    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"]
Nov 03 02:01:31.373863 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:31.373835    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738acb-857c5f554c-qcdbl"]
Nov 03 02:01:32.691783 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:32.691745    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d79e96-1752-414a-acc6-5b80b18381ab" path="/var/lib/kubelet/pods/56d79e96-1752-414a-acc6-5b80b18381ab/volumes"
Nov 03 02:01:32.692213 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:32.692196    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e2845e-0c51-428b-a854-a5a681d43341" path="/var/lib/kubelet/pods/66e2845e-0c51-428b-a854-a5a681d43341/volumes"
Nov 03 02:01:40.240645 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.240614    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"]
Nov 03 02:01:40.241086 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.241073    5508 cpu_manager.go:414] "containerMap: removing container" podUID="66e2845e-0c51-428b-a854-a5a681d43341" containerName="payload-pre-processing"
Nov 03 02:01:40.241137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.241088    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e2845e-0c51-428b-a854-a5a681d43341" containerName="payload-pre-processing"
Nov 03 02:01:40.241137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.241098    5508 cpu_manager.go:414] "containerMap: removing container" podUID="56d79e96-1752-414a-acc6-5b80b18381ab" containerName="payload-processing"
Nov 03 02:01:40.241137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.241103    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d79e96-1752-414a-acc6-5b80b18381ab" containerName="payload-processing"
Nov 03 02:01:40.241137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.241124    5508 cpu_manager.go:414] "containerMap: removing container" podUID="75f20122-1247-46aa-88ae-a499568d0969" containerName="revoke-keys"
Nov 03 02:01:40.241137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.241128    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f20122-1247-46aa-88ae-a499568d0969" containerName="revoke-keys"
Nov 03 02:01:40.241322 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.241186    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f20122-1247-46aa-88ae-a499568d0969" containerName="revoke-keys"
Nov 03 02:01:40.241322 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.241200    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e2845e-0c51-428b-a854-a5a681d43341" containerName="payload-pre-processing"
Nov 03 02:01:40.241322 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.241206    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d79e96-1752-414a-acc6-5b80b18381ab" containerName="payload-processing"
Nov 03 02:01:40.244580 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.244561    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"
Nov 03 02:01:40.293099 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.293068    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:01:40.328540 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.328506    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb1653a-cnhzk\" (UID: \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"
Nov 03 02:01:40.328739 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.328676    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb1653a-cnhzk\" (UID: \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"
Nov 03 02:01:40.328868 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.328841    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4f52\" (UniqueName: \"kubernetes.io/projected/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-kube-api-access-d4f52\") pod \"maas-api-revoke-sub-bdb1653a-cnhzk\" (UID: \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"
Nov 03 02:01:40.339290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.339261    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:01:40.341893 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.341873    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"]
Nov 03 02:01:40.429651 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.429623    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb1653a-cnhzk\" (UID: \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"
Nov 03 02:01:40.429770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.429706    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb1653a-cnhzk\" (UID: \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"
Nov 03 02:01:40.429770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.429764    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4f52\" (UniqueName: \"kubernetes.io/projected/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-kube-api-access-d4f52\") pod \"maas-api-revoke-sub-bdb1653a-cnhzk\" (UID: \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"
Nov 03 02:01:40.430383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.430337    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb1653a-cnhzk\" (UID: \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"
Nov 03 02:01:40.432056 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.432033    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb1653a-cnhzk\" (UID: \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"
Nov 03 02:01:40.487136 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.487107    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4f52\" (UniqueName: \"kubernetes.io/projected/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-kube-api-access-d4f52\") pod \"maas-api-revoke-sub-bdb1653a-cnhzk\" (UID: \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"
Nov 03 02:01:40.557738 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.557678    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"
Nov 03 02:01:40.933052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:40.933020    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"]
Nov 03 02:01:40.940254 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:01:40.940205    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced98a04_fd64_4ad5_b27e_4fd75fe326ee.slice/crio-e4f2ab7ecf5c346283addd1f6d6df06e2e62f2f317f8de86e74a5387c63ba32d WatchSource:0}: Error finding container e4f2ab7ecf5c346283addd1f6d6df06e2e62f2f317f8de86e74a5387c63ba32d: Status 404 returned error can't find the container with id e4f2ab7ecf5c346283addd1f6d6df06e2e62f2f317f8de86e74a5387c63ba32d
Nov 03 02:01:41.178518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:41.178486    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk" event={"ID":"ced98a04-fd64-4ad5-b27e-4fd75fe326ee","Type":"ContainerStarted","Data":"e4f2ab7ecf5c346283addd1f6d6df06e2e62f2f317f8de86e74a5387c63ba32d"}
Nov 03 02:01:42.185146 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:42.185115    5508 generic.go:356] "Generic (PLEG): container finished" podID="ced98a04-fd64-4ad5-b27e-4fd75fe326ee" containerID="9e9132afdc94da8616e37c1eb3d95859cf72dd9037549bddc4cf60f3f010c503" exitCode=0
Nov 03 02:01:42.185530 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:42.185206    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk" event={"ID":"ced98a04-fd64-4ad5-b27e-4fd75fe326ee","Type":"ContainerDied","Data":"9e9132afdc94da8616e37c1eb3d95859cf72dd9037549bddc4cf60f3f010c503"}
Nov 03 02:01:43.337731 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:43.337708    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"
Nov 03 02:01:43.469664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:43.469593    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-xks-ca\") pod \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\" (UID: \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\") "
Nov 03 02:01:43.469820 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:43.469740    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-openshift-ca\") pod \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\" (UID: \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\") "
Nov 03 02:01:43.469820 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:43.469779    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-kube-api-access-d4f52\" (UniqueName: \"kubernetes.io/projected/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-kube-api-access-d4f52\") pod \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\" (UID: \"ced98a04-fd64-4ad5-b27e-4fd75fe326ee\") "
Nov 03 02:01:43.470157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:43.470124    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-openshift-ca" pod "ced98a04-fd64-4ad5-b27e-4fd75fe326ee" (UID: "ced98a04-fd64-4ad5-b27e-4fd75fe326ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:43.471749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:43.471724    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-xks-ca" pod "ced98a04-fd64-4ad5-b27e-4fd75fe326ee" (UID: "ced98a04-fd64-4ad5-b27e-4fd75fe326ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:01:43.471860 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:43.471783    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-kube-api-access-d4f52" pod "ced98a04-fd64-4ad5-b27e-4fd75fe326ee" (UID: "ced98a04-fd64-4ad5-b27e-4fd75fe326ee"). InnerVolumeSpecName "kube-api-access-d4f52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:43.571285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:43.571261    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:43.571285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:43.571285    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:43.571455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:43.571296    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4f52\" (UniqueName: \"kubernetes.io/projected/ced98a04-fd64-4ad5-b27e-4fd75fe326ee-kube-api-access-d4f52\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:43.810406 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:01:43.810313    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:01:44.199096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:44.199061    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk" event={"ID":"ced98a04-fd64-4ad5-b27e-4fd75fe326ee","Type":"ContainerDied","Data":"e4f2ab7ecf5c346283addd1f6d6df06e2e62f2f317f8de86e74a5387c63ba32d"}
Nov 03 02:01:44.199290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:44.199120    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4f2ab7ecf5c346283addd1f6d6df06e2e62f2f317f8de86e74a5387c63ba32d"
Nov 03 02:01:44.199290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:44.199071    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"
Nov 03 02:01:44.995790 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:44.995747    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"]
Nov 03 02:01:49.997025 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:49.996982    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm" podUID="58766af5-ad89-4a97-aeef-5cb49c27ba2f" containerName="payload-pre-processing" containerID="cri-o://d70fe736a58b6425a00c75b0225e5541d4fb656edcccb0689a29838a6df20045" gracePeriod=25
Nov 03 02:01:50.199123 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:50.199088    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:01:50.236511 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:50.236482    5508 generic.go:356] "Generic (PLEG): container finished" podID="58766af5-ad89-4a97-aeef-5cb49c27ba2f" containerID="d70fe736a58b6425a00c75b0225e5541d4fb656edcccb0689a29838a6df20045" exitCode=0
Nov 03 02:01:50.236639 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:50.236558    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm" event={"ID":"58766af5-ad89-4a97-aeef-5cb49c27ba2f","Type":"ContainerDied","Data":"d70fe736a58b6425a00c75b0225e5541d4fb656edcccb0689a29838a6df20045"}
Nov 03 02:01:50.236639 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:50.236599    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm" event={"ID":"58766af5-ad89-4a97-aeef-5cb49c27ba2f","Type":"ContainerDied","Data":"83867521fa50ae8b19551186c8af77f740d1eaed347585fc5de1046562f8504c"}
Nov 03 02:01:50.236639 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:50.236614    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83867521fa50ae8b19551186c8af77f740d1eaed347585fc5de1046562f8504c"
Nov 03 02:01:50.249229 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:50.249161    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"
Nov 03 02:01:50.347969 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:50.347941    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58766af5-ad89-4a97-aeef-5cb49c27ba2f-kube-api-access-ndmws\" (UniqueName: \"kubernetes.io/projected/58766af5-ad89-4a97-aeef-5cb49c27ba2f-kube-api-access-ndmws\") pod \"58766af5-ad89-4a97-aeef-5cb49c27ba2f\" (UID: \"58766af5-ad89-4a97-aeef-5cb49c27ba2f\") "
Nov 03 02:01:50.348122 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:50.348114    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58766af5-ad89-4a97-aeef-5cb49c27ba2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58766af5-ad89-4a97-aeef-5cb49c27ba2f-plugins-config-volume\") pod \"58766af5-ad89-4a97-aeef-5cb49c27ba2f\" (UID: \"58766af5-ad89-4a97-aeef-5cb49c27ba2f\") "
Nov 03 02:01:50.348554 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:50.348523    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58766af5-ad89-4a97-aeef-5cb49c27ba2f-plugins-config-volume" pod "58766af5-ad89-4a97-aeef-5cb49c27ba2f" (UID: "58766af5-ad89-4a97-aeef-5cb49c27ba2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:50.349979 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:50.349958    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58766af5-ad89-4a97-aeef-5cb49c27ba2f-kube-api-access-ndmws" pod "58766af5-ad89-4a97-aeef-5cb49c27ba2f" (UID: "58766af5-ad89-4a97-aeef-5cb49c27ba2f"). InnerVolumeSpecName "kube-api-access-ndmws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:50.449907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:50.449874    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndmws\" (UniqueName: \"kubernetes.io/projected/58766af5-ad89-4a97-aeef-5cb49c27ba2f-kube-api-access-ndmws\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:50.449907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:50.449901    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58766af5-ad89-4a97-aeef-5cb49c27ba2f-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:51.243313 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:51.243275    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"
Nov 03 02:01:51.396928 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:51.396895    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"]
Nov 03 02:01:51.443094 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:51.443059    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-db583d-5f687c89f9-twsbm"]
Nov 03 02:01:52.690950 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:52.690917    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58766af5-ad89-4a97-aeef-5cb49c27ba2f" path="/var/lib/kubelet/pods/58766af5-ad89-4a97-aeef-5cb49c27ba2f/volumes"
Nov 03 02:01:54.971427 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:54.971395    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"]
Nov 03 02:01:54.971924 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:54.971910    5508 cpu_manager.go:414] "containerMap: removing container" podUID="58766af5-ad89-4a97-aeef-5cb49c27ba2f" containerName="payload-pre-processing"
Nov 03 02:01:54.971991 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:54.971926    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58766af5-ad89-4a97-aeef-5cb49c27ba2f" containerName="payload-pre-processing"
Nov 03 02:01:54.971991 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:54.971942    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ced98a04-fd64-4ad5-b27e-4fd75fe326ee" containerName="revoke-keys"
Nov 03 02:01:54.971991 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:54.971948    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced98a04-fd64-4ad5-b27e-4fd75fe326ee" containerName="revoke-keys"
Nov 03 02:01:54.972090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:54.972031    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced98a04-fd64-4ad5-b27e-4fd75fe326ee" containerName="revoke-keys"
Nov 03 02:01:54.972090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:54.972050    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="58766af5-ad89-4a97-aeef-5cb49c27ba2f" containerName="payload-pre-processing"
Nov 03 02:01:54.976927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:54.976901    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"
Nov 03 02:01:55.025799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.025771    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:01:55.071773 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.071737    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:01:55.075912 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.075880    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"]
Nov 03 02:01:55.102824 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.102796    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39160055-zgs7f\" (UID: \"83d97342-b645-4a63-8ca8-378c099e98e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"
Nov 03 02:01:55.102946 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.102866    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39160055-zgs7f\" (UID: \"83d97342-b645-4a63-8ca8-378c099e98e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"
Nov 03 02:01:55.103003 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.102963    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pttw7\" (UniqueName: \"kubernetes.io/projected/83d97342-b645-4a63-8ca8-378c099e98e6-kube-api-access-pttw7\") pod \"maas-api-revoke-sub-39160055-zgs7f\" (UID: \"83d97342-b645-4a63-8ca8-378c099e98e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"
Nov 03 02:01:55.203709 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.203674    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39160055-zgs7f\" (UID: \"83d97342-b645-4a63-8ca8-378c099e98e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"
Nov 03 02:01:55.203852 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.203745    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39160055-zgs7f\" (UID: \"83d97342-b645-4a63-8ca8-378c099e98e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"
Nov 03 02:01:55.203852 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.203782    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pttw7\" (UniqueName: \"kubernetes.io/projected/83d97342-b645-4a63-8ca8-378c099e98e6-kube-api-access-pttw7\") pod \"maas-api-revoke-sub-39160055-zgs7f\" (UID: \"83d97342-b645-4a63-8ca8-378c099e98e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"
Nov 03 02:01:55.204523 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.204502    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39160055-zgs7f\" (UID: \"83d97342-b645-4a63-8ca8-378c099e98e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"
Nov 03 02:01:55.206100 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.206083    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39160055-zgs7f\" (UID: \"83d97342-b645-4a63-8ca8-378c099e98e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"
Nov 03 02:01:55.260858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.260791    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pttw7\" (UniqueName: \"kubernetes.io/projected/83d97342-b645-4a63-8ca8-378c099e98e6-kube-api-access-pttw7\") pod \"maas-api-revoke-sub-39160055-zgs7f\" (UID: \"83d97342-b645-4a63-8ca8-378c099e98e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"
Nov 03 02:01:55.294730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.294699    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"
Nov 03 02:01:55.673499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:55.673243    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"]
Nov 03 02:01:56.277787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.277685    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f" event={"ID":"83d97342-b645-4a63-8ca8-378c099e98e6","Type":"ContainerStarted","Data":"54937a97b19e4eabe1a8b10af99177807ec3c64368319250151d1edca20c8f1b"}
Nov 03 02:01:56.277787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.277748    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f" event={"ID":"83d97342-b645-4a63-8ca8-378c099e98e6","Type":"ContainerStarted","Data":"10aab7d724cc81a0eb475243c01e0f01baf355ea4904ba261a954bc66d4cc7e8"}
Nov 03 02:01:56.278747 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.278727    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"]
Nov 03 02:01:56.282783 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.282763    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"
Nov 03 02:01:56.315030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.315005    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdcd9448-4lv4x\" (UID: \"c9fb8b31-e75e-4e28-affa-303892a342a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"
Nov 03 02:01:56.315168 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.315127    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdcd9448-4lv4x\" (UID: \"c9fb8b31-e75e-4e28-affa-303892a342a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"
Nov 03 02:01:56.315252 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.315234    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96stj\" (UniqueName: \"kubernetes.io/projected/c9fb8b31-e75e-4e28-affa-303892a342a7-kube-api-access-96stj\") pod \"maas-api-revoke-sub-fdcd9448-4lv4x\" (UID: \"c9fb8b31-e75e-4e28-affa-303892a342a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"
Nov 03 02:01:56.374823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.374787    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"]
Nov 03 02:01:56.386975 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.386917    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f" podStartSLOduration=2.386901726 podStartE2EDuration="2.386901726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:56.385009302 +0000 UTC m=+2595.810615766" watchObservedRunningTime="2026-11-03 02:01:56.386901726 +0000 UTC m=+2595.812508192"
Nov 03 02:01:56.416815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.416773    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdcd9448-4lv4x\" (UID: \"c9fb8b31-e75e-4e28-affa-303892a342a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"
Nov 03 02:01:56.416962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.416843    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96stj\" (UniqueName: \"kubernetes.io/projected/c9fb8b31-e75e-4e28-affa-303892a342a7-kube-api-access-96stj\") pod \"maas-api-revoke-sub-fdcd9448-4lv4x\" (UID: \"c9fb8b31-e75e-4e28-affa-303892a342a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"
Nov 03 02:01:56.416962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.416941    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdcd9448-4lv4x\" (UID: \"c9fb8b31-e75e-4e28-affa-303892a342a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"
Nov 03 02:01:56.417597 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.417572    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdcd9448-4lv4x\" (UID: \"c9fb8b31-e75e-4e28-affa-303892a342a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"
Nov 03 02:01:56.419101 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.419081    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdcd9448-4lv4x\" (UID: \"c9fb8b31-e75e-4e28-affa-303892a342a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"
Nov 03 02:01:56.473929 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.473901    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96stj\" (UniqueName: \"kubernetes.io/projected/c9fb8b31-e75e-4e28-affa-303892a342a7-kube-api-access-96stj\") pod \"maas-api-revoke-sub-fdcd9448-4lv4x\" (UID: \"c9fb8b31-e75e-4e28-affa-303892a342a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"
Nov 03 02:01:56.597982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.597890    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"
Nov 03 02:01:56.969397 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:56.969363    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"]
Nov 03 02:01:56.976755 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:01:56.976719    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9fb8b31_e75e_4e28_affa_303892a342a7.slice/crio-8c5f295e0d5fb8959e5b120e01ec91461b413ee84592858845e564f4baaf3d36 WatchSource:0}: Error finding container 8c5f295e0d5fb8959e5b120e01ec91461b413ee84592858845e564f4baaf3d36: Status 404 returned error can't find the container with id 8c5f295e0d5fb8959e5b120e01ec91461b413ee84592858845e564f4baaf3d36
Nov 03 02:01:57.285756 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:57.285720    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x" event={"ID":"c9fb8b31-e75e-4e28-affa-303892a342a7","Type":"ContainerStarted","Data":"c0256ff1851cc9c3b5d571ac6e3aea3782165d83b16d1c052f660df8b939bb5f"}
Nov 03 02:01:57.286192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:57.285780    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x" event={"ID":"c9fb8b31-e75e-4e28-affa-303892a342a7","Type":"ContainerStarted","Data":"8c5f295e0d5fb8959e5b120e01ec91461b413ee84592858845e564f4baaf3d36"}
Nov 03 02:01:57.287702 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:57.287674    5508 generic.go:356] "Generic (PLEG): container finished" podID="83d97342-b645-4a63-8ca8-378c099e98e6" containerID="54937a97b19e4eabe1a8b10af99177807ec3c64368319250151d1edca20c8f1b" exitCode=0
Nov 03 02:01:57.287836 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:57.287755    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f" event={"ID":"83d97342-b645-4a63-8ca8-378c099e98e6","Type":"ContainerDied","Data":"54937a97b19e4eabe1a8b10af99177807ec3c64368319250151d1edca20c8f1b"}
Nov 03 02:01:57.395679 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:57.395628    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x" podStartSLOduration=1.395613787 podStartE2EDuration="1.395613787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:01:57.394029285 +0000 UTC m=+2596.819635747" watchObservedRunningTime="2026-11-03 02:01:57.395613787 +0000 UTC m=+2596.821220249"
Nov 03 02:01:58.295939 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:58.295848    5508 generic.go:356] "Generic (PLEG): container finished" podID="c9fb8b31-e75e-4e28-affa-303892a342a7" containerID="c0256ff1851cc9c3b5d571ac6e3aea3782165d83b16d1c052f660df8b939bb5f" exitCode=0
Nov 03 02:01:58.296384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:58.295937    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x" event={"ID":"c9fb8b31-e75e-4e28-affa-303892a342a7","Type":"ContainerDied","Data":"c0256ff1851cc9c3b5d571ac6e3aea3782165d83b16d1c052f660df8b939bb5f"}
Nov 03 02:01:58.438848 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:58.438822    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"
Nov 03 02:01:58.541244 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:58.541203    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d97342-b645-4a63-8ca8-378c099e98e6-kube-api-access-pttw7\" (UniqueName: \"kubernetes.io/projected/83d97342-b645-4a63-8ca8-378c099e98e6-kube-api-access-pttw7\") pod \"83d97342-b645-4a63-8ca8-378c099e98e6\" (UID: \"83d97342-b645-4a63-8ca8-378c099e98e6\") "
Nov 03 02:01:58.541485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:58.541283    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-xks-ca\") pod \"83d97342-b645-4a63-8ca8-378c099e98e6\" (UID: \"83d97342-b645-4a63-8ca8-378c099e98e6\") "
Nov 03 02:01:58.541485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:58.541381    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-openshift-ca\") pod \"83d97342-b645-4a63-8ca8-378c099e98e6\" (UID: \"83d97342-b645-4a63-8ca8-378c099e98e6\") "
Nov 03 02:01:58.541812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:58.541778    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-openshift-ca" pod "83d97342-b645-4a63-8ca8-378c099e98e6" (UID: "83d97342-b645-4a63-8ca8-378c099e98e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:58.543489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:58.543468    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d97342-b645-4a63-8ca8-378c099e98e6-kube-api-access-pttw7" pod "83d97342-b645-4a63-8ca8-378c099e98e6" (UID: "83d97342-b645-4a63-8ca8-378c099e98e6"). InnerVolumeSpecName "kube-api-access-pttw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:58.543570 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:58.543472    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-xks-ca" pod "83d97342-b645-4a63-8ca8-378c099e98e6" (UID: "83d97342-b645-4a63-8ca8-378c099e98e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:01:58.643257 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:58.643225    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:58.643257 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:58.643250    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pttw7\" (UniqueName: \"kubernetes.io/projected/83d97342-b645-4a63-8ca8-378c099e98e6-kube-api-access-pttw7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:58.643257 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:58.643262    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83d97342-b645-4a63-8ca8-378c099e98e6-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:59.303556 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:59.303519    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"
Nov 03 02:01:59.303968 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:59.303547    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f" event={"ID":"83d97342-b645-4a63-8ca8-378c099e98e6","Type":"ContainerDied","Data":"10aab7d724cc81a0eb475243c01e0f01baf355ea4904ba261a954bc66d4cc7e8"}
Nov 03 02:01:59.303968 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:59.303600    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10aab7d724cc81a0eb475243c01e0f01baf355ea4904ba261a954bc66d4cc7e8"
Nov 03 02:01:59.444559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:59.444533    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"
Nov 03 02:01:59.552735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:59.552705    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-openshift-ca\") pod \"c9fb8b31-e75e-4e28-affa-303892a342a7\" (UID: \"c9fb8b31-e75e-4e28-affa-303892a342a7\") "
Nov 03 02:01:59.552930 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:59.552813    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9fb8b31-e75e-4e28-affa-303892a342a7-kube-api-access-96stj\" (UniqueName: \"kubernetes.io/projected/c9fb8b31-e75e-4e28-affa-303892a342a7-kube-api-access-96stj\") pod \"c9fb8b31-e75e-4e28-affa-303892a342a7\" (UID: \"c9fb8b31-e75e-4e28-affa-303892a342a7\") "
Nov 03 02:01:59.552930 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:59.552874    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-xks-ca\") pod \"c9fb8b31-e75e-4e28-affa-303892a342a7\" (UID: \"c9fb8b31-e75e-4e28-affa-303892a342a7\") "
Nov 03 02:01:59.553131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:59.553105    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-openshift-ca" pod "c9fb8b31-e75e-4e28-affa-303892a342a7" (UID: "c9fb8b31-e75e-4e28-affa-303892a342a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:01:59.555101 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:59.555045    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fb8b31-e75e-4e28-affa-303892a342a7-kube-api-access-96stj" pod "c9fb8b31-e75e-4e28-affa-303892a342a7" (UID: "c9fb8b31-e75e-4e28-affa-303892a342a7"). InnerVolumeSpecName "kube-api-access-96stj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:01:59.555411 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:59.555394    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-xks-ca" pod "c9fb8b31-e75e-4e28-affa-303892a342a7" (UID: "c9fb8b31-e75e-4e28-affa-303892a342a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:01:59.653897 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:59.653863    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96stj\" (UniqueName: \"kubernetes.io/projected/c9fb8b31-e75e-4e28-affa-303892a342a7-kube-api-access-96stj\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:59.653897 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:59.653890    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:01:59.653897 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:01:59.653900    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9fb8b31-e75e-4e28-affa-303892a342a7-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:00.310279 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:00.310249    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"
Nov 03 02:02:00.310717 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:00.310277    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x" event={"ID":"c9fb8b31-e75e-4e28-affa-303892a342a7","Type":"ContainerDied","Data":"8c5f295e0d5fb8959e5b120e01ec91461b413ee84592858845e564f4baaf3d36"}
Nov 03 02:02:00.310717 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:00.310322    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c5f295e0d5fb8959e5b120e01ec91461b413ee84592858845e564f4baaf3d36"
Nov 03 02:02:04.123828 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.123795    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"]
Nov 03 02:02:04.124328 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.123987    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5" podUID="b310f9ce-a97a-4ec8-89eb-e9d266b93253" containerName="payload-processing" containerID="cri-o://b55cd6d406664ccc05e9ca568c7e9e082de21a7a3523392210acb2992c038efc" gracePeriod=30
Nov 03 02:02:04.179427 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.177748    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"]
Nov 03 02:02:04.179427 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.178069    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw" podUID="9172158a-823e-413f-a32f-d0a4d39ef09b" containerName="payload-pre-processing" containerID="cri-o://0bec3b21a46b0b123822146c16f3f95134b49a58af0f55f955823744ffb633fe" gracePeriod=30
Nov 03 02:02:04.337820 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.337787    5508 generic.go:356] "Generic (PLEG): container finished" podID="b310f9ce-a97a-4ec8-89eb-e9d266b93253" containerID="b55cd6d406664ccc05e9ca568c7e9e082de21a7a3523392210acb2992c038efc" exitCode=0
Nov 03 02:02:04.338024 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.337873    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5" event={"ID":"b310f9ce-a97a-4ec8-89eb-e9d266b93253","Type":"ContainerDied","Data":"b55cd6d406664ccc05e9ca568c7e9e082de21a7a3523392210acb2992c038efc"}
Nov 03 02:02:04.339616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.339582    5508 generic.go:356] "Generic (PLEG): container finished" podID="9172158a-823e-413f-a32f-d0a4d39ef09b" containerID="0bec3b21a46b0b123822146c16f3f95134b49a58af0f55f955823744ffb633fe" exitCode=0
Nov 03 02:02:04.339731 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.339650    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw" event={"ID":"9172158a-823e-413f-a32f-d0a4d39ef09b","Type":"ContainerDied","Data":"0bec3b21a46b0b123822146c16f3f95134b49a58af0f55f955823744ffb633fe"}
Nov 03 02:02:04.436372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.436321    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"
Nov 03 02:02:04.455361 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.455331    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"
Nov 03 02:02:04.515596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.515561    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9172158a-823e-413f-a32f-d0a4d39ef09b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9172158a-823e-413f-a32f-d0a4d39ef09b-plugins-config-volume\") pod \"9172158a-823e-413f-a32f-d0a4d39ef09b\" (UID: \"9172158a-823e-413f-a32f-d0a4d39ef09b\") "
Nov 03 02:02:04.515596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.515597    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b310f9ce-a97a-4ec8-89eb-e9d266b93253-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b310f9ce-a97a-4ec8-89eb-e9d266b93253-plugins-config-volume\") pod \"b310f9ce-a97a-4ec8-89eb-e9d266b93253\" (UID: \"b310f9ce-a97a-4ec8-89eb-e9d266b93253\") "
Nov 03 02:02:04.515861 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.515721    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b310f9ce-a97a-4ec8-89eb-e9d266b93253-kube-api-access-z2w6b\" (UniqueName: \"kubernetes.io/projected/b310f9ce-a97a-4ec8-89eb-e9d266b93253-kube-api-access-z2w6b\") pod \"b310f9ce-a97a-4ec8-89eb-e9d266b93253\" (UID: \"b310f9ce-a97a-4ec8-89eb-e9d266b93253\") "
Nov 03 02:02:04.515861 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.515826    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9172158a-823e-413f-a32f-d0a4d39ef09b-kube-api-access-f45tn\" (UniqueName: \"kubernetes.io/projected/9172158a-823e-413f-a32f-d0a4d39ef09b-kube-api-access-f45tn\") pod \"9172158a-823e-413f-a32f-d0a4d39ef09b\" (UID: \"9172158a-823e-413f-a32f-d0a4d39ef09b\") "
Nov 03 02:02:04.516074 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.516045    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9172158a-823e-413f-a32f-d0a4d39ef09b-plugins-config-volume" pod "9172158a-823e-413f-a32f-d0a4d39ef09b" (UID: "9172158a-823e-413f-a32f-d0a4d39ef09b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:04.516148 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.516086    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b310f9ce-a97a-4ec8-89eb-e9d266b93253-plugins-config-volume" pod "b310f9ce-a97a-4ec8-89eb-e9d266b93253" (UID: "b310f9ce-a97a-4ec8-89eb-e9d266b93253"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:04.516231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.516214    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9172158a-823e-413f-a32f-d0a4d39ef09b-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:04.516279 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.516238    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b310f9ce-a97a-4ec8-89eb-e9d266b93253-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:04.517921 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.517902    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b310f9ce-a97a-4ec8-89eb-e9d266b93253-kube-api-access-z2w6b" pod "b310f9ce-a97a-4ec8-89eb-e9d266b93253" (UID: "b310f9ce-a97a-4ec8-89eb-e9d266b93253"). InnerVolumeSpecName "kube-api-access-z2w6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:04.517979 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.517921    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9172158a-823e-413f-a32f-d0a4d39ef09b-kube-api-access-f45tn" pod "9172158a-823e-413f-a32f-d0a4d39ef09b" (UID: "9172158a-823e-413f-a32f-d0a4d39ef09b"). InnerVolumeSpecName "kube-api-access-f45tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:04.617060 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.617017    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2w6b\" (UniqueName: \"kubernetes.io/projected/b310f9ce-a97a-4ec8-89eb-e9d266b93253-kube-api-access-z2w6b\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:04.617060 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:04.617061    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f45tn\" (UniqueName: \"kubernetes.io/projected/9172158a-823e-413f-a32f-d0a4d39ef09b-kube-api-access-f45tn\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:05.347304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.347275    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"
Nov 03 02:02:05.347814 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.347327    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw" event={"ID":"9172158a-823e-413f-a32f-d0a4d39ef09b","Type":"ContainerDied","Data":"0ab6ab058dafaa1886c7dab481792f840ef329f7142c09fbec45530cad9c336f"}
Nov 03 02:02:05.347814 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.347407    5508 scope.go:122] "RemoveContainer" containerID="0bec3b21a46b0b123822146c16f3f95134b49a58af0f55f955823744ffb633fe"
Nov 03 02:02:05.348917 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.348900    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"
Nov 03 02:02:05.349047 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.348926    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5" event={"ID":"b310f9ce-a97a-4ec8-89eb-e9d266b93253","Type":"ContainerDied","Data":"9fd1e7319bd027ded8e3f7a685a677221ee96c0575360cbaa5c38754dd91de98"}
Nov 03 02:02:05.357208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.357192    5508 scope.go:122] "RemoveContainer" containerID="b55cd6d406664ccc05e9ca568c7e9e082de21a7a3523392210acb2992c038efc"
Nov 03 02:02:05.500760 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.500723    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"]
Nov 03 02:02:05.506513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.506488    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f95773-7f459c9cb7-wn7tw"]
Nov 03 02:02:05.607843 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.607806    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"]
Nov 03 02:02:05.613272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.613220    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f95773-59f4965db4-sznn5"]
Nov 03 02:02:05.716000 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.715971    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"]
Nov 03 02:02:05.716407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.716392    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b310f9ce-a97a-4ec8-89eb-e9d266b93253" containerName="payload-processing"
Nov 03 02:02:05.716453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.716408    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b310f9ce-a97a-4ec8-89eb-e9d266b93253" containerName="payload-processing"
Nov 03 02:02:05.716453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.716421    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9172158a-823e-413f-a32f-d0a4d39ef09b" containerName="payload-pre-processing"
Nov 03 02:02:05.716453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.716425    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9172158a-823e-413f-a32f-d0a4d39ef09b" containerName="payload-pre-processing"
Nov 03 02:02:05.716453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.716451    5508 cpu_manager.go:414] "containerMap: removing container" podUID="83d97342-b645-4a63-8ca8-378c099e98e6" containerName="revoke-keys"
Nov 03 02:02:05.716575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.716456    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d97342-b645-4a63-8ca8-378c099e98e6" containerName="revoke-keys"
Nov 03 02:02:05.716575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.716464    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c9fb8b31-e75e-4e28-affa-303892a342a7" containerName="revoke-keys"
Nov 03 02:02:05.716575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.716469    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9fb8b31-e75e-4e28-affa-303892a342a7" containerName="revoke-keys"
Nov 03 02:02:05.716575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.716532    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d97342-b645-4a63-8ca8-378c099e98e6" containerName="revoke-keys"
Nov 03 02:02:05.716575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.716540    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9fb8b31-e75e-4e28-affa-303892a342a7" containerName="revoke-keys"
Nov 03 02:02:05.716575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.716551    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b310f9ce-a97a-4ec8-89eb-e9d266b93253" containerName="payload-processing"
Nov 03 02:02:05.716575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.716558    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9172158a-823e-413f-a32f-d0a4d39ef09b" containerName="payload-pre-processing"
Nov 03 02:02:05.721243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.721225    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"
Nov 03 02:02:05.770078 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.769992    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:02:05.770078 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.770010    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:02:05.777827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.777803    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"]
Nov 03 02:02:05.830707 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.830679    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbdb56cc-zhwqw\" (UID: \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"
Nov 03 02:02:05.830872 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.830714    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zwkt\" (UniqueName: \"kubernetes.io/projected/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-kube-api-access-2zwkt\") pod \"maas-api-revoke-sub-dbdb56cc-zhwqw\" (UID: \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"
Nov 03 02:02:05.830872 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.830850    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbdb56cc-zhwqw\" (UID: \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"
Nov 03 02:02:05.932053 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.932018    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbdb56cc-zhwqw\" (UID: \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"
Nov 03 02:02:05.932231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.932109    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbdb56cc-zhwqw\" (UID: \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"
Nov 03 02:02:05.932231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.932134    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwkt\" (UniqueName: \"kubernetes.io/projected/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-kube-api-access-2zwkt\") pod \"maas-api-revoke-sub-dbdb56cc-zhwqw\" (UID: \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"
Nov 03 02:02:05.932883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.932846    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbdb56cc-zhwqw\" (UID: \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"
Nov 03 02:02:05.934655 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.934630    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbdb56cc-zhwqw\" (UID: \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"
Nov 03 02:02:05.989746 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:05.989710    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zwkt\" (UniqueName: \"kubernetes.io/projected/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-kube-api-access-2zwkt\") pod \"maas-api-revoke-sub-dbdb56cc-zhwqw\" (UID: \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"
Nov 03 02:02:06.033615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:06.033527    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"
Nov 03 02:02:06.410438 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:06.410407    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"]
Nov 03 02:02:06.417980 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:06.417948    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb8b73c_2f1d_4c1a_8eea_e9f04491879b.slice/crio-90ad5e7f80f6308ad98826c04b9974c8d9a58eb2a702d080444868d3e434385a WatchSource:0}: Error finding container 90ad5e7f80f6308ad98826c04b9974c8d9a58eb2a702d080444868d3e434385a: Status 404 returned error can't find the container with id 90ad5e7f80f6308ad98826c04b9974c8d9a58eb2a702d080444868d3e434385a
Nov 03 02:02:06.691067 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:06.691039    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9172158a-823e-413f-a32f-d0a4d39ef09b" path="/var/lib/kubelet/pods/9172158a-823e-413f-a32f-d0a4d39ef09b/volumes"
Nov 03 02:02:06.691439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:06.691426    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b310f9ce-a97a-4ec8-89eb-e9d266b93253" path="/var/lib/kubelet/pods/b310f9ce-a97a-4ec8-89eb-e9d266b93253/volumes"
Nov 03 02:02:07.367332 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:07.367300    5508 generic.go:356] "Generic (PLEG): container finished" podID="dbb8b73c-2f1d-4c1a-8eea-e9f04491879b" containerID="c8fbf3f64496943bb83ba2358f9e9ac6e7ecc6ea42a1a5bf16627b7cb46da0ee" exitCode=0
Nov 03 02:02:07.367501 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:07.367366    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw" event={"ID":"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b","Type":"ContainerDied","Data":"c8fbf3f64496943bb83ba2358f9e9ac6e7ecc6ea42a1a5bf16627b7cb46da0ee"}
Nov 03 02:02:07.367501 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:07.367425    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw" event={"ID":"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b","Type":"ContainerStarted","Data":"90ad5e7f80f6308ad98826c04b9974c8d9a58eb2a702d080444868d3e434385a"}
Nov 03 02:02:07.921210 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:07.921172    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"]
Nov 03 02:02:07.921735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:07.921470    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs" podUID="16676061-80e9-435c-923b-e4e9b6d128d0" containerName="istio-proxy" containerID="cri-o://ec63eefc74f9569aca9843bf09cd4075f82d37d1b89d48b8324cbc760261d554" gracePeriod=30
Nov 03 02:02:08.515834 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:08.515803    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"
Nov 03 02:02:08.561038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:08.560996    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-openshift-ca\") pod \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\" (UID: \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\") "
Nov 03 02:02:08.561224 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:08.561062    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-xks-ca\") pod \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\" (UID: \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\") "
Nov 03 02:02:08.561224 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:08.561168    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-kube-api-access-2zwkt\" (UniqueName: \"kubernetes.io/projected/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-kube-api-access-2zwkt\") pod \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\" (UID: \"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b\") "
Nov 03 02:02:08.561511 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:08.561486    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-openshift-ca" pod "dbb8b73c-2f1d-4c1a-8eea-e9f04491879b" (UID: "dbb8b73c-2f1d-4c1a-8eea-e9f04491879b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:08.563305 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:08.563282    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-xks-ca" pod "dbb8b73c-2f1d-4c1a-8eea-e9f04491879b" (UID: "dbb8b73c-2f1d-4c1a-8eea-e9f04491879b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:08.563454 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:08.563360    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-kube-api-access-2zwkt" pod "dbb8b73c-2f1d-4c1a-8eea-e9f04491879b" (UID: "dbb8b73c-2f1d-4c1a-8eea-e9f04491879b"). InnerVolumeSpecName "kube-api-access-2zwkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:08.662256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:08.662221    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zwkt\" (UniqueName: \"kubernetes.io/projected/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-kube-api-access-2zwkt\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:08.662256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:08.662250    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:08.662256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:08.662260    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:09.383476 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:09.383445    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"
Nov 03 02:02:09.383923 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:09.383476    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw" event={"ID":"dbb8b73c-2f1d-4c1a-8eea-e9f04491879b","Type":"ContainerDied","Data":"90ad5e7f80f6308ad98826c04b9974c8d9a58eb2a702d080444868d3e434385a"}
Nov 03 02:02:09.383923 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:09.383524    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90ad5e7f80f6308ad98826c04b9974c8d9a58eb2a702d080444868d3e434385a"
Nov 03 02:02:12.572504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.572465    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"]
Nov 03 02:02:12.573134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.572898    5508 cpu_manager.go:414] "containerMap: removing container" podUID="dbb8b73c-2f1d-4c1a-8eea-e9f04491879b" containerName="revoke-keys"
Nov 03 02:02:12.573134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.572908    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb8b73c-2f1d-4c1a-8eea-e9f04491879b" containerName="revoke-keys"
Nov 03 02:02:12.573134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.572975    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb8b73c-2f1d-4c1a-8eea-e9f04491879b" containerName="revoke-keys"
Nov 03 02:02:12.576672 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.576649    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"
Nov 03 02:02:12.625005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.624971    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:02:12.625114 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.625079    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:02:12.675671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.675640    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"]
Nov 03 02:02:12.719134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.719107    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8pg4\" (UniqueName: \"kubernetes.io/projected/3f38340d-e9cd-4d78-8881-cf97e97fe559-kube-api-access-l8pg4\") pod \"maas-api-revoke-sub-9bdf4d8c-7jqx5\" (UID: \"3f38340d-e9cd-4d78-8881-cf97e97fe559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"
Nov 03 02:02:12.719279 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.719155    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bdf4d8c-7jqx5\" (UID: \"3f38340d-e9cd-4d78-8881-cf97e97fe559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"
Nov 03 02:02:12.719279 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.719230    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bdf4d8c-7jqx5\" (UID: \"3f38340d-e9cd-4d78-8881-cf97e97fe559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"
Nov 03 02:02:12.820473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.820436    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8pg4\" (UniqueName: \"kubernetes.io/projected/3f38340d-e9cd-4d78-8881-cf97e97fe559-kube-api-access-l8pg4\") pod \"maas-api-revoke-sub-9bdf4d8c-7jqx5\" (UID: \"3f38340d-e9cd-4d78-8881-cf97e97fe559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"
Nov 03 02:02:12.820671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.820495    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bdf4d8c-7jqx5\" (UID: \"3f38340d-e9cd-4d78-8881-cf97e97fe559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"
Nov 03 02:02:12.820671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.820525    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bdf4d8c-7jqx5\" (UID: \"3f38340d-e9cd-4d78-8881-cf97e97fe559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"
Nov 03 02:02:12.821193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.821163    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bdf4d8c-7jqx5\" (UID: \"3f38340d-e9cd-4d78-8881-cf97e97fe559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"
Nov 03 02:02:12.823032 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.822981    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bdf4d8c-7jqx5\" (UID: \"3f38340d-e9cd-4d78-8881-cf97e97fe559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"
Nov 03 02:02:12.877409 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.877376    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8pg4\" (UniqueName: \"kubernetes.io/projected/3f38340d-e9cd-4d78-8881-cf97e97fe559-kube-api-access-l8pg4\") pod \"maas-api-revoke-sub-9bdf4d8c-7jqx5\" (UID: \"3f38340d-e9cd-4d78-8881-cf97e97fe559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"
Nov 03 02:02:12.891001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:12.890979    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"
Nov 03 02:02:13.163812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.163782    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 02:02:13.225217 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.225181    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16676061-80e9-435c-923b-e4e9b6d128d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16676061-80e9-435c-923b-e4e9b6d128d0-istiod-ca-cert\") pod \"16676061-80e9-435c-923b-e4e9b6d128d0\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") "
Nov 03 02:02:13.225429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.225238    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-istio-token\" (UniqueName: \"kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-istio-token\") pod \"16676061-80e9-435c-923b-e4e9b6d128d0\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") "
Nov 03 02:02:13.225429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.225344    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-certs\") pod \"16676061-80e9-435c-923b-e4e9b6d128d0\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") "
Nov 03 02:02:13.225429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.225415    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-envoy\") pod \"16676061-80e9-435c-923b-e4e9b6d128d0\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") "
Nov 03 02:02:13.225631 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.225513    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-socket\") pod \"16676061-80e9-435c-923b-e4e9b6d128d0\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") "
Nov 03 02:02:13.225631 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.225545    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-data\") pod \"16676061-80e9-435c-923b-e4e9b6d128d0\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") "
Nov 03 02:02:13.225631 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.225595    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-kube-api-access-87fkn\" (UniqueName: \"kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-kube-api-access-87fkn\") pod \"16676061-80e9-435c-923b-e4e9b6d128d0\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") "
Nov 03 02:02:13.225809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.225638    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16676061-80e9-435c-923b-e4e9b6d128d0-istiod-ca-cert" pod "16676061-80e9-435c-923b-e4e9b6d128d0" (UID: "16676061-80e9-435c-923b-e4e9b6d128d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:13.225809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.225658    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-credential-socket\") pod \"16676061-80e9-435c-923b-e4e9b6d128d0\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") "
Nov 03 02:02:13.225809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.225684    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-certs" pod "16676061-80e9-435c-923b-e4e9b6d128d0" (UID: "16676061-80e9-435c-923b-e4e9b6d128d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:13.225809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.225704    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16676061-80e9-435c-923b-e4e9b6d128d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16676061-80e9-435c-923b-e4e9b6d128d0-istio-podinfo\") pod \"16676061-80e9-435c-923b-e4e9b6d128d0\" (UID: \"16676061-80e9-435c-923b-e4e9b6d128d0\") "
Nov 03 02:02:13.226035 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.225903    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-socket" pod "16676061-80e9-435c-923b-e4e9b6d128d0" (UID: "16676061-80e9-435c-923b-e4e9b6d128d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:13.226327 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.226105    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:13.226327 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.226135    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16676061-80e9-435c-923b-e4e9b6d128d0-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:13.226327 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.226148    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:13.226582 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.226400    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-credential-socket" pod "16676061-80e9-435c-923b-e4e9b6d128d0" (UID: "16676061-80e9-435c-923b-e4e9b6d128d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:13.226637 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.226611    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-data" pod "16676061-80e9-435c-923b-e4e9b6d128d0" (UID: "16676061-80e9-435c-923b-e4e9b6d128d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:13.228507 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.228482    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-kube-api-access-87fkn" pod "16676061-80e9-435c-923b-e4e9b6d128d0" (UID: "16676061-80e9-435c-923b-e4e9b6d128d0"). InnerVolumeSpecName "kube-api-access-87fkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:13.228612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.228506    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-istio-token" pod "16676061-80e9-435c-923b-e4e9b6d128d0" (UID: "16676061-80e9-435c-923b-e4e9b6d128d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:13.228668 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.228641    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-envoy" pod "16676061-80e9-435c-923b-e4e9b6d128d0" (UID: "16676061-80e9-435c-923b-e4e9b6d128d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:13.229770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.229745    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16676061-80e9-435c-923b-e4e9b6d128d0-istio-podinfo" pod "16676061-80e9-435c-923b-e4e9b6d128d0" (UID: "16676061-80e9-435c-923b-e4e9b6d128d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:02:13.261829 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.261786    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"]
Nov 03 02:02:13.262119 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.262093    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy" containerID="cri-o://d59d15404f51cd9ec64c3090fef2ce359a10cffcf47fa78e3845f6d700d9047a" gracePeriod=30
Nov 03 02:02:13.265272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.265228    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"]
Nov 03 02:02:13.277669 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:13.277636    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f38340d_e9cd_4d78_8881_cf97e97fe559.slice/crio-93ab3e7c1913e16327d1e7ac18c5d83188c11dd17fda11c0e138868325f71271 WatchSource:0}: Error finding container 93ab3e7c1913e16327d1e7ac18c5d83188c11dd17fda11c0e138868325f71271: Status 404 returned error can't find the container with id 93ab3e7c1913e16327d1e7ac18c5d83188c11dd17fda11c0e138868325f71271
Nov 03 02:02:13.326953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.326930    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:13.326953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.326953    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:13.327146 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.326963    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87fkn\" (UniqueName: \"kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-kube-api-access-87fkn\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:13.327146 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.326973    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16676061-80e9-435c-923b-e4e9b6d128d0-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:13.327146 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.326983    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16676061-80e9-435c-923b-e4e9b6d128d0-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:13.327146 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.326996    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16676061-80e9-435c-923b-e4e9b6d128d0-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:13.411009 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.410979    5508 generic.go:356] "Generic (PLEG): container finished" podID="16676061-80e9-435c-923b-e4e9b6d128d0" containerID="ec63eefc74f9569aca9843bf09cd4075f82d37d1b89d48b8324cbc760261d554" exitCode=0
Nov 03 02:02:13.411193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.411031    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"
Nov 03 02:02:13.411193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.411063    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs" event={"ID":"16676061-80e9-435c-923b-e4e9b6d128d0","Type":"ContainerDied","Data":"ec63eefc74f9569aca9843bf09cd4075f82d37d1b89d48b8324cbc760261d554"}
Nov 03 02:02:13.411193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.411116    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs" event={"ID":"16676061-80e9-435c-923b-e4e9b6d128d0","Type":"ContainerDied","Data":"a462cd7db3a62145729f6849194b90d65c3d097e1abe665147a9f34f4bb166a7"}
Nov 03 02:02:13.411193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.411144    5508 scope.go:122] "RemoveContainer" containerID="ec63eefc74f9569aca9843bf09cd4075f82d37d1b89d48b8324cbc760261d554"
Nov 03 02:02:13.412416 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.412394    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5" event={"ID":"3f38340d-e9cd-4d78-8881-cf97e97fe559","Type":"ContainerStarted","Data":"93ab3e7c1913e16327d1e7ac18c5d83188c11dd17fda11c0e138868325f71271"}
Nov 03 02:02:13.420787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.420771    5508 scope.go:122] "RemoveContainer" containerID="f4cb768843505ccc95c1f799d4d650c4dae09efa3e4a1fc5c9d6004938fc9550"
Nov 03 02:02:13.429303 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.429285    5508 scope.go:122] "RemoveContainer" containerID="ec63eefc74f9569aca9843bf09cd4075f82d37d1b89d48b8324cbc760261d554"
Nov 03 02:02:13.430006 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:02:13.429984    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec63eefc74f9569aca9843bf09cd4075f82d37d1b89d48b8324cbc760261d554\": container with ID starting with ec63eefc74f9569aca9843bf09cd4075f82d37d1b89d48b8324cbc760261d554 not found: ID does not exist" containerID="ec63eefc74f9569aca9843bf09cd4075f82d37d1b89d48b8324cbc760261d554"
Nov 03 02:02:13.430089 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.430012    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec63eefc74f9569aca9843bf09cd4075f82d37d1b89d48b8324cbc760261d554"} err="failed to get container status \"ec63eefc74f9569aca9843bf09cd4075f82d37d1b89d48b8324cbc760261d554\": rpc error: code = NotFound desc = could not find container \"ec63eefc74f9569aca9843bf09cd4075f82d37d1b89d48b8324cbc760261d554\": container with ID starting with ec63eefc74f9569aca9843bf09cd4075f82d37d1b89d48b8324cbc760261d554 not found: ID does not exist"
Nov 03 02:02:13.430089 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.430027    5508 scope.go:122] "RemoveContainer" containerID="f4cb768843505ccc95c1f799d4d650c4dae09efa3e4a1fc5c9d6004938fc9550"
Nov 03 02:02:13.430305 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:02:13.430285    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4cb768843505ccc95c1f799d4d650c4dae09efa3e4a1fc5c9d6004938fc9550\": container with ID starting with f4cb768843505ccc95c1f799d4d650c4dae09efa3e4a1fc5c9d6004938fc9550 not found: ID does not exist" containerID="f4cb768843505ccc95c1f799d4d650c4dae09efa3e4a1fc5c9d6004938fc9550"
Nov 03 02:02:13.430387 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.430316    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4cb768843505ccc95c1f799d4d650c4dae09efa3e4a1fc5c9d6004938fc9550"} err="failed to get container status \"f4cb768843505ccc95c1f799d4d650c4dae09efa3e4a1fc5c9d6004938fc9550\": rpc error: code = NotFound desc = could not find container \"f4cb768843505ccc95c1f799d4d650c4dae09efa3e4a1fc5c9d6004938fc9550\": container with ID starting with f4cb768843505ccc95c1f799d4d650c4dae09efa3e4a1fc5c9d6004938fc9550 not found: ID does not exist"
Nov 03 02:02:13.566094 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.566063    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"]
Nov 03 02:02:13.570296 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:13.570266    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f95773-openshift-default-6f658db89c-wqlgs"]
Nov 03 02:02:13.811570 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:02:13.811540    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:02:14.419671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:14.419637    5508 generic.go:356] "Generic (PLEG): container finished" podID="3f38340d-e9cd-4d78-8881-cf97e97fe559" containerID="3a71fce94a3ef29f34c520ed01c4d80ee89019f3123729575d0ca191a596ec4e" exitCode=0
Nov 03 02:02:14.419846 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:14.419727    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5" event={"ID":"3f38340d-e9cd-4d78-8881-cf97e97fe559","Type":"ContainerDied","Data":"3a71fce94a3ef29f34c520ed01c4d80ee89019f3123729575d0ca191a596ec4e"}
Nov 03 02:02:14.694237 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:14.694205    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16676061-80e9-435c-923b-e4e9b6d128d0" path="/var/lib/kubelet/pods/16676061-80e9-435c-923b-e4e9b6d128d0/volumes"
Nov 03 02:02:14.841154 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:14.841121    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:02:14.841626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:14.841173    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:02:15.085845 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.085751    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"]
Nov 03 02:02:15.086037 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.085948    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b" podUID="c61eafee-cfd7-41d6-825e-1e358e76ddc2" containerName="maas-api" containerID="cri-o://6d0bfad6aecac1c32cc12adf11fd0a50418b01e605a78834b5688a0353bdffa7" gracePeriod=30
Nov 03 02:02:15.366675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.366646    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:02:15.428071 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.428041    5508 generic.go:356] "Generic (PLEG): container finished" podID="c61eafee-cfd7-41d6-825e-1e358e76ddc2" containerID="6d0bfad6aecac1c32cc12adf11fd0a50418b01e605a78834b5688a0353bdffa7" exitCode=0
Nov 03 02:02:15.428243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.428085    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"
Nov 03 02:02:15.428243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.428129    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b" event={"ID":"c61eafee-cfd7-41d6-825e-1e358e76ddc2","Type":"ContainerDied","Data":"6d0bfad6aecac1c32cc12adf11fd0a50418b01e605a78834b5688a0353bdffa7"}
Nov 03 02:02:15.428243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.428177    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b" event={"ID":"c61eafee-cfd7-41d6-825e-1e358e76ddc2","Type":"ContainerDied","Data":"97280fce7de49bb60f6dd60c260ceb19baa64eb833986523ae6daeb372652912"}
Nov 03 02:02:15.428243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.428200    5508 scope.go:122] "RemoveContainer" containerID="6d0bfad6aecac1c32cc12adf11fd0a50418b01e605a78834b5688a0353bdffa7"
Nov 03 02:02:15.439340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.439314    5508 scope.go:122] "RemoveContainer" containerID="6d0bfad6aecac1c32cc12adf11fd0a50418b01e605a78834b5688a0353bdffa7"
Nov 03 02:02:15.439631 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:02:15.439613    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0bfad6aecac1c32cc12adf11fd0a50418b01e605a78834b5688a0353bdffa7\": container with ID starting with 6d0bfad6aecac1c32cc12adf11fd0a50418b01e605a78834b5688a0353bdffa7 not found: ID does not exist" containerID="6d0bfad6aecac1c32cc12adf11fd0a50418b01e605a78834b5688a0353bdffa7"
Nov 03 02:02:15.439678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.439640    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0bfad6aecac1c32cc12adf11fd0a50418b01e605a78834b5688a0353bdffa7"} err="failed to get container status \"6d0bfad6aecac1c32cc12adf11fd0a50418b01e605a78834b5688a0353bdffa7\": rpc error: code = NotFound desc = could not find container \"6d0bfad6aecac1c32cc12adf11fd0a50418b01e605a78834b5688a0353bdffa7\": container with ID starting with 6d0bfad6aecac1c32cc12adf11fd0a50418b01e605a78834b5688a0353bdffa7 not found: ID does not exist"
Nov 03 02:02:15.447384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.447358    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61eafee-cfd7-41d6-825e-1e358e76ddc2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61eafee-cfd7-41d6-825e-1e358e76ddc2-openshift-service-ca\") pod \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") "
Nov 03 02:02:15.447524 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.447420    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61eafee-cfd7-41d6-825e-1e358e76ddc2-kube-api-access-9gbh4\" (UniqueName: \"kubernetes.io/projected/c61eafee-cfd7-41d6-825e-1e358e76ddc2-kube-api-access-9gbh4\") pod \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") "
Nov 03 02:02:15.447524 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.447464    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-metrics-certs\") pod \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") "
Nov 03 02:02:15.447636 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.447538    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-maas-api-tls\") pod \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\" (UID: \"c61eafee-cfd7-41d6-825e-1e358e76ddc2\") "
Nov 03 02:02:15.447773 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.447754    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61eafee-cfd7-41d6-825e-1e358e76ddc2-openshift-service-ca" pod "c61eafee-cfd7-41d6-825e-1e358e76ddc2" (UID: "c61eafee-cfd7-41d6-825e-1e358e76ddc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:15.447992 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.447977    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61eafee-cfd7-41d6-825e-1e358e76ddc2-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:15.455232 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.449756    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-metrics-certs" pod "c61eafee-cfd7-41d6-825e-1e358e76ddc2" (UID: "c61eafee-cfd7-41d6-825e-1e358e76ddc2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:15.455232 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.450361    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61eafee-cfd7-41d6-825e-1e358e76ddc2-kube-api-access-9gbh4" pod "c61eafee-cfd7-41d6-825e-1e358e76ddc2" (UID: "c61eafee-cfd7-41d6-825e-1e358e76ddc2"). InnerVolumeSpecName "kube-api-access-9gbh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:15.455599 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.455579    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-maas-api-tls" pod "c61eafee-cfd7-41d6-825e-1e358e76ddc2" (UID: "c61eafee-cfd7-41d6-825e-1e358e76ddc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:15.549061 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.549018    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gbh4\" (UniqueName: \"kubernetes.io/projected/c61eafee-cfd7-41d6-825e-1e358e76ddc2-kube-api-access-9gbh4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:15.549061 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.549060    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:15.549373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.549075    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c61eafee-cfd7-41d6-825e-1e358e76ddc2-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:15.589138 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.589111    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"
Nov 03 02:02:15.750670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.750630    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f38340d-e9cd-4d78-8881-cf97e97fe559-kube-api-access-l8pg4\" (UniqueName: \"kubernetes.io/projected/3f38340d-e9cd-4d78-8881-cf97e97fe559-kube-api-access-l8pg4\") pod \"3f38340d-e9cd-4d78-8881-cf97e97fe559\" (UID: \"3f38340d-e9cd-4d78-8881-cf97e97fe559\") "
Nov 03 02:02:15.750872 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.750692    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-xks-ca\") pod \"3f38340d-e9cd-4d78-8881-cf97e97fe559\" (UID: \"3f38340d-e9cd-4d78-8881-cf97e97fe559\") "
Nov 03 02:02:15.750872 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.750760    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-openshift-ca\") pod \"3f38340d-e9cd-4d78-8881-cf97e97fe559\" (UID: \"3f38340d-e9cd-4d78-8881-cf97e97fe559\") "
Nov 03 02:02:15.751423 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.751146    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-openshift-ca" pod "3f38340d-e9cd-4d78-8881-cf97e97fe559" (UID: "3f38340d-e9cd-4d78-8881-cf97e97fe559"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:15.753098 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.753073    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f38340d-e9cd-4d78-8881-cf97e97fe559-kube-api-access-l8pg4" pod "3f38340d-e9cd-4d78-8881-cf97e97fe559" (UID: "3f38340d-e9cd-4d78-8881-cf97e97fe559"). InnerVolumeSpecName "kube-api-access-l8pg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:15.753739 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.753713    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-xks-ca" pod "3f38340d-e9cd-4d78-8881-cf97e97fe559" (UID: "3f38340d-e9cd-4d78-8881-cf97e97fe559"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:15.844815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.844745    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"]
Nov 03 02:02:15.848941 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.848909    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-db583d-75d7fbcb89-rsc4b"]
Nov 03 02:02:15.852159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.852128    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:15.852159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.852155    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8pg4\" (UniqueName: \"kubernetes.io/projected/3f38340d-e9cd-4d78-8881-cf97e97fe559-kube-api-access-l8pg4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:15.852393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:15.852169    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f38340d-e9cd-4d78-8881-cf97e97fe559-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:16.436630 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:16.436602    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"
Nov 03 02:02:16.436630 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:16.436620    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5" event={"ID":"3f38340d-e9cd-4d78-8881-cf97e97fe559","Type":"ContainerDied","Data":"93ab3e7c1913e16327d1e7ac18c5d83188c11dd17fda11c0e138868325f71271"}
Nov 03 02:02:16.436887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:16.436684    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93ab3e7c1913e16327d1e7ac18c5d83188c11dd17fda11c0e138868325f71271"
Nov 03 02:02:16.693643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:16.693562    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61eafee-cfd7-41d6-825e-1e358e76ddc2" path="/var/lib/kubelet/pods/c61eafee-cfd7-41d6-825e-1e358e76ddc2/volumes"
Nov 03 02:02:18.451552 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.451517    5508 generic.go:356] "Generic (PLEG): container finished" podID="21141eda-543d-4fa6-9404-8289591a86d6" containerID="d59d15404f51cd9ec64c3090fef2ce359a10cffcf47fa78e3845f6d700d9047a" exitCode=0
Nov 03 02:02:18.451999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.451602    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" event={"ID":"21141eda-543d-4fa6-9404-8289591a86d6","Type":"ContainerDied","Data":"d59d15404f51cd9ec64c3090fef2ce359a10cffcf47fa78e3845f6d700d9047a"}
Nov 03 02:02:18.451999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.451678    5508 scope.go:122] "RemoveContainer" containerID="20d24d8809523e1585b5c44693111bf0f78d4d726805c142d4cfc3b23b0d17db"
Nov 03 02:02:18.529488 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.529424    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 02:02:18.680978 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.680941    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21141eda-543d-4fa6-9404-8289591a86d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21141eda-543d-4fa6-9404-8289591a86d6-istiod-ca-cert\") pod \"21141eda-543d-4fa6-9404-8289591a86d6\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") "
Nov 03 02:02:18.681190 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.680996    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-socket\") pod \"21141eda-543d-4fa6-9404-8289591a86d6\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") "
Nov 03 02:02:18.681190 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.681024    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-istio-token\" (UniqueName: \"kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-istio-token\") pod \"21141eda-543d-4fa6-9404-8289591a86d6\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") "
Nov 03 02:02:18.681190 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.681163    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-data\") pod \"21141eda-543d-4fa6-9404-8289591a86d6\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") "
Nov 03 02:02:18.681430 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.681233    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-credential-socket\") pod \"21141eda-543d-4fa6-9404-8289591a86d6\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") "
Nov 03 02:02:18.681430 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.681317    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-socket" pod "21141eda-543d-4fa6-9404-8289591a86d6" (UID: "21141eda-543d-4fa6-9404-8289591a86d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:18.681430 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.681369    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-kube-api-access-8rdbg\" (UniqueName: \"kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-kube-api-access-8rdbg\") pod \"21141eda-543d-4fa6-9404-8289591a86d6\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") "
Nov 03 02:02:18.681430 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.681402    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-certs\") pod \"21141eda-543d-4fa6-9404-8289591a86d6\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") "
Nov 03 02:02:18.681642 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.681448    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21141eda-543d-4fa6-9404-8289591a86d6-istiod-ca-cert" pod "21141eda-543d-4fa6-9404-8289591a86d6" (UID: "21141eda-543d-4fa6-9404-8289591a86d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:18.681642 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.681454    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-envoy\") pod \"21141eda-543d-4fa6-9404-8289591a86d6\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") "
Nov 03 02:02:18.681642 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.681543    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-credential-socket" pod "21141eda-543d-4fa6-9404-8289591a86d6" (UID: "21141eda-543d-4fa6-9404-8289591a86d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:18.681642 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.681550    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21141eda-543d-4fa6-9404-8289591a86d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21141eda-543d-4fa6-9404-8289591a86d6-istio-podinfo\") pod \"21141eda-543d-4fa6-9404-8289591a86d6\" (UID: \"21141eda-543d-4fa6-9404-8289591a86d6\") "
Nov 03 02:02:18.681642 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.681589    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-data" pod "21141eda-543d-4fa6-9404-8289591a86d6" (UID: "21141eda-543d-4fa6-9404-8289591a86d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:18.681903 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.681770    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-certs" pod "21141eda-543d-4fa6-9404-8289591a86d6" (UID: "21141eda-543d-4fa6-9404-8289591a86d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:18.682029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.682006    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:18.682092 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.682034    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21141eda-543d-4fa6-9404-8289591a86d6-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:18.682092 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.682049    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:18.682092 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.682064    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:18.682092 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.682080    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:18.683738 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.683700    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-kube-api-access-8rdbg" pod "21141eda-543d-4fa6-9404-8289591a86d6" (UID: "21141eda-543d-4fa6-9404-8289591a86d6"). InnerVolumeSpecName "kube-api-access-8rdbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:18.683850 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.683782    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-envoy" pod "21141eda-543d-4fa6-9404-8289591a86d6" (UID: "21141eda-543d-4fa6-9404-8289591a86d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:18.683850 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.683814    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-istio-token" pod "21141eda-543d-4fa6-9404-8289591a86d6" (UID: "21141eda-543d-4fa6-9404-8289591a86d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:18.683850 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.683845    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21141eda-543d-4fa6-9404-8289591a86d6-istio-podinfo" pod "21141eda-543d-4fa6-9404-8289591a86d6" (UID: "21141eda-543d-4fa6-9404-8289591a86d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:02:18.783220 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.783138    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rdbg\" (UniqueName: \"kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-kube-api-access-8rdbg\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:18.783220 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.783168    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21141eda-543d-4fa6-9404-8289591a86d6-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:18.783220 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.783177    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21141eda-543d-4fa6-9404-8289591a86d6-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:18.783220 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:18.783188    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21141eda-543d-4fa6-9404-8289591a86d6-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:19.460520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:19.460485    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"
Nov 03 02:02:19.460939 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:19.460517    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt" event={"ID":"21141eda-543d-4fa6-9404-8289591a86d6","Type":"ContainerDied","Data":"d3683a0b3bcfc1a1ae77fcbc82ff1e61e359e4159a475f51621a2606e901033b"}
Nov 03 02:02:19.460939 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:19.460576    5508 scope.go:122] "RemoveContainer" containerID="d59d15404f51cd9ec64c3090fef2ce359a10cffcf47fa78e3845f6d700d9047a"
Nov 03 02:02:19.659694 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:19.659663    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"]
Nov 03 02:02:19.705777 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:19.705735    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-70235a-openshift-default-74d89989d5-hfcrt"]
Nov 03 02:02:20.320840 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:20.320809    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"]
Nov 03 02:02:20.692293 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:20.692264    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21141eda-543d-4fa6-9404-8289591a86d6" path="/var/lib/kubelet/pods/21141eda-543d-4fa6-9404-8289591a86d6/volumes"
Nov 03 02:02:22.445865 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.445831    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"]
Nov 03 02:02:22.446277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446263    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3f38340d-e9cd-4d78-8881-cf97e97fe559" containerName="revoke-keys"
Nov 03 02:02:22.446325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446279    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f38340d-e9cd-4d78-8881-cf97e97fe559" containerName="revoke-keys"
Nov 03 02:02:22.446325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446293    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c61eafee-cfd7-41d6-825e-1e358e76ddc2" containerName="maas-api"
Nov 03 02:02:22.446325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446299    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61eafee-cfd7-41d6-825e-1e358e76ddc2" containerName="maas-api"
Nov 03 02:02:22.446325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446308    5508 cpu_manager.go:414] "containerMap: removing container" podUID="16676061-80e9-435c-923b-e4e9b6d128d0" containerName="istio-proxy"
Nov 03 02:02:22.446325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446313    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16676061-80e9-435c-923b-e4e9b6d128d0" containerName="istio-proxy"
Nov 03 02:02:22.446925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446338    5508 cpu_manager.go:414] "containerMap: removing container" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy"
Nov 03 02:02:22.446925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446343    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy"
Nov 03 02:02:22.446925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446365    5508 cpu_manager.go:414] "containerMap: removing container" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy"
Nov 03 02:02:22.446925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446370    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy"
Nov 03 02:02:22.446925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446435    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="16676061-80e9-435c-923b-e4e9b6d128d0" containerName="istio-proxy"
Nov 03 02:02:22.446925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446447    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy"
Nov 03 02:02:22.446925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446455    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="21141eda-543d-4fa6-9404-8289591a86d6" containerName="istio-proxy"
Nov 03 02:02:22.446925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446464    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f38340d-e9cd-4d78-8881-cf97e97fe559" containerName="revoke-keys"
Nov 03 02:02:22.446925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.446471    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61eafee-cfd7-41d6-825e-1e358e76ddc2" containerName="maas-api"
Nov 03 02:02:22.451379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.451330    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.499942 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.499905    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9da431-openshift-default-dockercfg-bq94q\""
Nov 03 02:02:22.557082 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.557049    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"]
Nov 03 02:02:22.625660 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.625621    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.625837 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.625687    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.625837 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.625766    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.625837 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.625799    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.625837 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.625827    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvspd\" (UniqueName: \"kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-kube-api-access-mvspd\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.625999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.625913    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.625999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.625968    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ea6070-8986-41ae-98be-6e20baf08648-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.626072 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.626006    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ea6070-8986-41ae-98be-6e20baf08648-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.626072 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.626042    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.727223 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.727125    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.728029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.727752    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.728302 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.727283    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.728434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.728409    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.728504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.728481    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.728562 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.728521    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.728622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.728558    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvspd\" (UniqueName: \"kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-kube-api-access-mvspd\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.728622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.728612    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.729304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.728756    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ea6070-8986-41ae-98be-6e20baf08648-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.729304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.728811    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ea6070-8986-41ae-98be-6e20baf08648-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.729304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.728826    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.729304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.728922    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.729304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.729020    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.730144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.730123    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ea6070-8986-41ae-98be-6e20baf08648-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.731617 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.731595    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.732193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.732172    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ea6070-8986-41ae-98be-6e20baf08648-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.785930 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.785897    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.786091 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.786004    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvspd\" (UniqueName: \"kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-kube-api-access-mvspd\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:22.846198 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.846170    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"]
Nov 03 02:02:22.846639 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.846625    5508 cpu_manager.go:414] "containerMap: removing container" podUID="16676061-80e9-435c-923b-e4e9b6d128d0" containerName="istio-proxy"
Nov 03 02:02:22.846639 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.846640    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16676061-80e9-435c-923b-e4e9b6d128d0" containerName="istio-proxy"
Nov 03 02:02:22.846752 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.846710    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="16676061-80e9-435c-923b-e4e9b6d128d0" containerName="istio-proxy"
Nov 03 02:02:22.851062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.851042    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"
Nov 03 02:02:22.899782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.899751    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:02:22.899782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.899763    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:02:22.907211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:22.907188    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"]
Nov 03 02:02:23.031686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.031594    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b281df-9blgh\" (UID: \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"
Nov 03 02:02:23.031686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.031638    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b281df-9blgh\" (UID: \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"
Nov 03 02:02:23.031932 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.031843    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlnng\" (UniqueName: \"kubernetes.io/projected/97fb68aa-4dba-43d3-9eda-d0262926d4e9-kube-api-access-hlnng\") pod \"maas-api-revoke-sub-02b281df-9blgh\" (UID: \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"
Nov 03 02:02:23.065603 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.065575    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:23.136290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.133802    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlnng\" (UniqueName: \"kubernetes.io/projected/97fb68aa-4dba-43d3-9eda-d0262926d4e9-kube-api-access-hlnng\") pod \"maas-api-revoke-sub-02b281df-9blgh\" (UID: \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"
Nov 03 02:02:23.136290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.133955    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b281df-9blgh\" (UID: \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"
Nov 03 02:02:23.136290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.134018    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b281df-9blgh\" (UID: \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"
Nov 03 02:02:23.136290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.135207    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b281df-9blgh\" (UID: \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"
Nov 03 02:02:23.138328 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.138001    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b281df-9blgh\" (UID: \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"
Nov 03 02:02:23.190821 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.190784    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlnng\" (UniqueName: \"kubernetes.io/projected/97fb68aa-4dba-43d3-9eda-d0262926d4e9-kube-api-access-hlnng\") pod \"maas-api-revoke-sub-02b281df-9blgh\" (UID: \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"
Nov 03 02:02:23.239193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.239144    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"]
Nov 03 02:02:23.239458 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.239433    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5" podUID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerName="istio-proxy" containerID="cri-o://515ec6e97f486e250bf0caf2386e52815ca11b99f50d3015795621b501e2d0a6" gracePeriod=30
Nov 03 02:02:23.464601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.464567    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"
Nov 03 02:02:23.495080 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.495046    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"]
Nov 03 02:02:23.501195 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:23.501169    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ea6070_8986_41ae_98be_6e20baf08648.slice/crio-0424f6951fb804043623d84edddf926f18e7fd5609dfd224e63f9f0dd242b114 WatchSource:0}: Error finding container 0424f6951fb804043623d84edddf926f18e7fd5609dfd224e63f9f0dd242b114: Status 404 returned error can't find the container with id 0424f6951fb804043623d84edddf926f18e7fd5609dfd224e63f9f0dd242b114
Nov 03 02:02:23.503626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.503596    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:02:23.503726 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.503655    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:02:23.503726 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.503687    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:02:23.842892 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:23.842807    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"]
Nov 03 02:02:23.850377 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:23.848411    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fb68aa_4dba_43d3_9eda_d0262926d4e9.slice/crio-216829b04cd325345eb22dd44812e668d49c00849e5c4c4377dce0bb74ee3ca3 WatchSource:0}: Error finding container 216829b04cd325345eb22dd44812e668d49c00849e5c4c4377dce0bb74ee3ca3: Status 404 returned error can't find the container with id 216829b04cd325345eb22dd44812e668d49c00849e5c4c4377dce0bb74ee3ca3
Nov 03 02:02:24.505561 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:24.505444    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp" event={"ID":"41ea6070-8986-41ae-98be-6e20baf08648","Type":"ContainerStarted","Data":"98b363f396d884557f47e5d66b21e1fad20ac32bfcf5aa45a1d411ade58ebca1"}
Nov 03 02:02:24.506150 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:24.505562    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp" event={"ID":"41ea6070-8986-41ae-98be-6e20baf08648","Type":"ContainerStarted","Data":"0424f6951fb804043623d84edddf926f18e7fd5609dfd224e63f9f0dd242b114"}
Nov 03 02:02:24.507668 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:24.507642    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh" event={"ID":"97fb68aa-4dba-43d3-9eda-d0262926d4e9","Type":"ContainerStarted","Data":"0132d5a8263ede036f486ea2cc58ab64ae8ee19bc44fb5d762164c690d4fb0c9"}
Nov 03 02:02:24.507807 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:24.507683    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh" event={"ID":"97fb68aa-4dba-43d3-9eda-d0262926d4e9","Type":"ContainerStarted","Data":"216829b04cd325345eb22dd44812e668d49c00849e5c4c4377dce0bb74ee3ca3"}
Nov 03 02:02:24.619916 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:24.619862    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp" podStartSLOduration=2.619850146 podStartE2EDuration="2.619850146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:24.618612071 +0000 UTC m=+2624.044218551" watchObservedRunningTime="2026-11-03 02:02:24.619850146 +0000 UTC m=+2624.045456611"
Nov 03 02:02:24.728808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:24.728757    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh" podStartSLOduration=2.728744383 podStartE2EDuration="2.728744383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:24.726618803 +0000 UTC m=+2624.152225286" watchObservedRunningTime="2026-11-03 02:02:24.728744383 +0000 UTC m=+2624.154350868"
Nov 03 02:02:25.070431 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.069732    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:25.073823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.073456    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 03 02:02:25.073823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.073518    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp" podUID="41ea6070-8986-41ae-98be-6e20baf08648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 03 02:02:25.321986 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.321895    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld" podUID="867f1e43-883e-44ba-83ed-a3e19a552af0" containerName="payload-pre-processing" containerID="cri-o://b9ada2034edfe7deb5f4793fc9e6780fec55e35d8349c0bd0bc2bcfc3c68b83b" gracePeriod=25
Nov 03 02:02:25.515953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.515918    5508 generic.go:356] "Generic (PLEG): container finished" podID="867f1e43-883e-44ba-83ed-a3e19a552af0" containerID="b9ada2034edfe7deb5f4793fc9e6780fec55e35d8349c0bd0bc2bcfc3c68b83b" exitCode=0
Nov 03 02:02:25.516305 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.516024    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld" event={"ID":"867f1e43-883e-44ba-83ed-a3e19a552af0","Type":"ContainerDied","Data":"b9ada2034edfe7deb5f4793fc9e6780fec55e35d8349c0bd0bc2bcfc3c68b83b"}
Nov 03 02:02:25.523097 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.523065    5508 generic.go:356] "Generic (PLEG): container finished" podID="97fb68aa-4dba-43d3-9eda-d0262926d4e9" containerID="0132d5a8263ede036f486ea2cc58ab64ae8ee19bc44fb5d762164c690d4fb0c9" exitCode=0
Nov 03 02:02:25.524054 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.524033    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh" event={"ID":"97fb68aa-4dba-43d3-9eda-d0262926d4e9","Type":"ContainerDied","Data":"0132d5a8263ede036f486ea2cc58ab64ae8ee19bc44fb5d762164c690d4fb0c9"}
Nov 03 02:02:25.646473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.646430    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"
Nov 03 02:02:25.713343 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.713310    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"]
Nov 03 02:02:25.713927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.713901    5508 cpu_manager.go:414] "containerMap: removing container" podUID="867f1e43-883e-44ba-83ed-a3e19a552af0" containerName="payload-pre-processing"
Nov 03 02:02:25.714035 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.713928    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867f1e43-883e-44ba-83ed-a3e19a552af0" containerName="payload-pre-processing"
Nov 03 02:02:25.714097 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.714065    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="867f1e43-883e-44ba-83ed-a3e19a552af0" containerName="payload-pre-processing"
Nov 03 02:02:25.717479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.717454    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.777406 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.774134    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b22fcc-openshift-default-dockercfg-hdl2h\""
Nov 03 02:02:25.811388 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.807689    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867f1e43-883e-44ba-83ed-a3e19a552af0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867f1e43-883e-44ba-83ed-a3e19a552af0-plugins-config-volume\") pod \"867f1e43-883e-44ba-83ed-a3e19a552af0\" (UID: \"867f1e43-883e-44ba-83ed-a3e19a552af0\") "
Nov 03 02:02:25.814445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.814261    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867f1e43-883e-44ba-83ed-a3e19a552af0-kube-api-access-r2zp7\" (UniqueName: \"kubernetes.io/projected/867f1e43-883e-44ba-83ed-a3e19a552af0-kube-api-access-r2zp7\") pod \"867f1e43-883e-44ba-83ed-a3e19a552af0\" (UID: \"867f1e43-883e-44ba-83ed-a3e19a552af0\") "
Nov 03 02:02:25.815408 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.815380    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-data\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.815689 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.815594    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-credential-socket\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.815841 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.815825    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-istio-token\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.815985 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.815971    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmtvb\" (UniqueName: \"kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-kube-api-access-bmtvb\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.816160 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.816131    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-certs\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.816427 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.816396    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a28645-ef66-46b3-8a68-a999694dadc3-istio-podinfo\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.816756 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.816566    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-socket\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.817372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.817334    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"]
Nov 03 02:02:25.817873 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.817841    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867f1e43-883e-44ba-83ed-a3e19a552af0-plugins-config-volume" pod "867f1e43-883e-44ba-83ed-a3e19a552af0" (UID: "867f1e43-883e-44ba-83ed-a3e19a552af0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:25.820496 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.820472    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-envoy\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.820682 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.820653    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a28645-ef66-46b3-8a68-a999694dadc3-istiod-ca-cert\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.820779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.820474    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867f1e43-883e-44ba-83ed-a3e19a552af0-kube-api-access-r2zp7" pod "867f1e43-883e-44ba-83ed-a3e19a552af0" (UID: "867f1e43-883e-44ba-83ed-a3e19a552af0"). InnerVolumeSpecName "kube-api-access-r2zp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:25.820779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.820765    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867f1e43-883e-44ba-83ed-a3e19a552af0-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:25.921285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.921230    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a28645-ef66-46b3-8a68-a999694dadc3-istio-podinfo\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.921518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.921307    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-socket\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.921518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.921341    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-envoy\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.921518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.921376    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a28645-ef66-46b3-8a68-a999694dadc3-istiod-ca-cert\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.921518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.921399    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-data\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.921518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.921426    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-credential-socket\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.921518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.921449    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-istio-token\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.921518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.921478    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmtvb\" (UniqueName: \"kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-kube-api-access-bmtvb\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.921518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.921512    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-certs\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.921982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.921565    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2zp7\" (UniqueName: \"kubernetes.io/projected/867f1e43-883e-44ba-83ed-a3e19a552af0-kube-api-access-r2zp7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:25.921982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.921908    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-certs\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.921982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.921937    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-data\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.922215 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.922189    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-credential-socket\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.922404 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.922343    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-socket\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.922587 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.922562    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a28645-ef66-46b3-8a68-a999694dadc3-istiod-ca-cert\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.924107 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.924090    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-envoy\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.924490 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.924468    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a28645-ef66-46b3-8a68-a999694dadc3-istio-podinfo\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.983827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.983790    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-istio-token\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:25.986048 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:25.986025    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmtvb\" (UniqueName: \"kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-kube-api-access-bmtvb\") pod \"e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") " pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:26.030882 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.030855    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:26.072372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.071765    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 03 02:02:26.072372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.071833    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp" podUID="41ea6070-8986-41ae-98be-6e20baf08648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 03 02:02:26.494568 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.494529    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"]
Nov 03 02:02:26.500956 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:26.500923    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a28645_ef66_46b3_8a68_a999694dadc3.slice/crio-2620b136af3580ac99014422caefccab057baf392ee022d23076f21ccf8208b2 WatchSource:0}: Error finding container 2620b136af3580ac99014422caefccab057baf392ee022d23076f21ccf8208b2: Status 404 returned error can't find the container with id 2620b136af3580ac99014422caefccab057baf392ee022d23076f21ccf8208b2
Nov 03 02:02:26.503529 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.503497    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:02:26.503617 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.503563    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:02:26.503617 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.503590    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:02:26.529947 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.529919    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"
Nov 03 02:02:26.529947 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.529925    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld" event={"ID":"867f1e43-883e-44ba-83ed-a3e19a552af0","Type":"ContainerDied","Data":"0d5ba3c3a54b4c9235b7b19ceac33ed0d35cd3a37536081f3416f7a29103471f"}
Nov 03 02:02:26.530447 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.529992    5508 scope.go:122] "RemoveContainer" containerID="b9ada2034edfe7deb5f4793fc9e6780fec55e35d8349c0bd0bc2bcfc3c68b83b"
Nov 03 02:02:26.531858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.531595    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5" event={"ID":"65a28645-ef66-46b3-8a68-a999694dadc3","Type":"ContainerStarted","Data":"2620b136af3580ac99014422caefccab057baf392ee022d23076f21ccf8208b2"}
Nov 03 02:02:26.709490 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.708729    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"]
Nov 03 02:02:26.713703 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.713671    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e0907d-57b44df859-4kmld"]
Nov 03 02:02:26.735153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.735128    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"
Nov 03 02:02:26.831332 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.831237    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fb68aa-4dba-43d3-9eda-d0262926d4e9-kube-api-access-hlnng\" (UniqueName: \"kubernetes.io/projected/97fb68aa-4dba-43d3-9eda-d0262926d4e9-kube-api-access-hlnng\") pod \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\" (UID: \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\") "
Nov 03 02:02:26.831533 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.831390    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-openshift-ca\") pod \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\" (UID: \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\") "
Nov 03 02:02:26.831533 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.831462    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-xks-ca\") pod \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\" (UID: \"97fb68aa-4dba-43d3-9eda-d0262926d4e9\") "
Nov 03 02:02:26.831844 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.831795    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-openshift-ca" pod "97fb68aa-4dba-43d3-9eda-d0262926d4e9" (UID: "97fb68aa-4dba-43d3-9eda-d0262926d4e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:26.833607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.833581    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-xks-ca" pod "97fb68aa-4dba-43d3-9eda-d0262926d4e9" (UID: "97fb68aa-4dba-43d3-9eda-d0262926d4e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:26.833705 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.833686    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fb68aa-4dba-43d3-9eda-d0262926d4e9-kube-api-access-hlnng" pod "97fb68aa-4dba-43d3-9eda-d0262926d4e9" (UID: "97fb68aa-4dba-43d3-9eda-d0262926d4e9"). InnerVolumeSpecName "kube-api-access-hlnng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:26.932899 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.932865    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:26.932899 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.932894    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97fb68aa-4dba-43d3-9eda-d0262926d4e9-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:26.932899 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:26.932903    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlnng\" (UniqueName: \"kubernetes.io/projected/97fb68aa-4dba-43d3-9eda-d0262926d4e9-kube-api-access-hlnng\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:27.071374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:27.068809    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 03 02:02:27.071374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:27.068869    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp" podUID="41ea6070-8986-41ae-98be-6e20baf08648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 03 02:02:27.539842 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:27.539788    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5" event={"ID":"65a28645-ef66-46b3-8a68-a999694dadc3","Type":"ContainerStarted","Data":"917f79b61998840d91a024d2078e9bd3bdcf7022fb67ef672712812e61a81939"}
Nov 03 02:02:27.541153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:27.541129    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh" event={"ID":"97fb68aa-4dba-43d3-9eda-d0262926d4e9","Type":"ContainerDied","Data":"216829b04cd325345eb22dd44812e668d49c00849e5c4c4377dce0bb74ee3ca3"}
Nov 03 02:02:27.541334 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:27.541161    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"
Nov 03 02:02:27.541334 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:27.541166    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="216829b04cd325345eb22dd44812e668d49c00849e5c4c4377dce0bb74ee3ca3"
Nov 03 02:02:28.031098 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.031061    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:28.036608 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.036581    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:28.071372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.067688    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 03 02:02:28.071372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.067747    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp" podUID="41ea6070-8986-41ae-98be-6e20baf08648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 03 02:02:28.170374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.163452    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5" podStartSLOduration=3.163432775 podStartE2EDuration="3.163432775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:27.682001734 +0000 UTC m=+2627.107608199" watchObservedRunningTime="2026-11-03 02:02:28.163432775 +0000 UTC m=+2627.589039241"
Nov 03 02:02:28.539844 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.539573    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 02:02:28.572375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.571793    5508 generic.go:356] "Generic (PLEG): container finished" podID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerID="515ec6e97f486e250bf0caf2386e52815ca11b99f50d3015795621b501e2d0a6" exitCode=0
Nov 03 02:02:28.573473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.572907    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"
Nov 03 02:02:28.573473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.573054    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5" event={"ID":"23107beb-9060-480d-9f1b-6ba175e4d4ef","Type":"ContainerDied","Data":"515ec6e97f486e250bf0caf2386e52815ca11b99f50d3015795621b501e2d0a6"}
Nov 03 02:02:28.573473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.573102    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5" event={"ID":"23107beb-9060-480d-9f1b-6ba175e4d4ef","Type":"ContainerDied","Data":"4cc9eafe3a23dc60be407d7b3859e34a6b3ab04bd3510e3a9f08bb4812ed1936"}
Nov 03 02:02:28.573473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.573132    5508 scope.go:122] "RemoveContainer" containerID="515ec6e97f486e250bf0caf2386e52815ca11b99f50d3015795621b501e2d0a6"
Nov 03 02:02:28.577664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.577069    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:28.578845 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.578789    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:02:28.586963 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.586944    5508 scope.go:122] "RemoveContainer" containerID="e63bb301ce12690968e0a956756241bfced2c4e100d55aa61091b1901b3aaeb0"
Nov 03 02:02:28.595649 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.595620    5508 scope.go:122] "RemoveContainer" containerID="515ec6e97f486e250bf0caf2386e52815ca11b99f50d3015795621b501e2d0a6"
Nov 03 02:02:28.595932 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:02:28.595912    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515ec6e97f486e250bf0caf2386e52815ca11b99f50d3015795621b501e2d0a6\": container with ID starting with 515ec6e97f486e250bf0caf2386e52815ca11b99f50d3015795621b501e2d0a6 not found: ID does not exist" containerID="515ec6e97f486e250bf0caf2386e52815ca11b99f50d3015795621b501e2d0a6"
Nov 03 02:02:28.595992 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.595945    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515ec6e97f486e250bf0caf2386e52815ca11b99f50d3015795621b501e2d0a6"} err="failed to get container status \"515ec6e97f486e250bf0caf2386e52815ca11b99f50d3015795621b501e2d0a6\": rpc error: code = NotFound desc = could not find container \"515ec6e97f486e250bf0caf2386e52815ca11b99f50d3015795621b501e2d0a6\": container with ID starting with 515ec6e97f486e250bf0caf2386e52815ca11b99f50d3015795621b501e2d0a6 not found: ID does not exist"
Nov 03 02:02:28.595992 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.595962    5508 scope.go:122] "RemoveContainer" containerID="e63bb301ce12690968e0a956756241bfced2c4e100d55aa61091b1901b3aaeb0"
Nov 03 02:02:28.596229 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:02:28.596209    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e63bb301ce12690968e0a956756241bfced2c4e100d55aa61091b1901b3aaeb0\": container with ID starting with e63bb301ce12690968e0a956756241bfced2c4e100d55aa61091b1901b3aaeb0 not found: ID does not exist" containerID="e63bb301ce12690968e0a956756241bfced2c4e100d55aa61091b1901b3aaeb0"
Nov 03 02:02:28.596290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.596239    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e63bb301ce12690968e0a956756241bfced2c4e100d55aa61091b1901b3aaeb0"} err="failed to get container status \"e63bb301ce12690968e0a956756241bfced2c4e100d55aa61091b1901b3aaeb0\": rpc error: code = NotFound desc = could not find container \"e63bb301ce12690968e0a956756241bfced2c4e100d55aa61091b1901b3aaeb0\": container with ID starting with e63bb301ce12690968e0a956756241bfced2c4e100d55aa61091b1901b3aaeb0 not found: ID does not exist"
Nov 03 02:02:28.671864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.670670    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-credential-socket\") pod \"23107beb-9060-480d-9f1b-6ba175e4d4ef\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") "
Nov 03 02:02:28.671864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.670725    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-kube-api-access-w8psh\" (UniqueName: \"kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-kube-api-access-w8psh\") pod \"23107beb-9060-480d-9f1b-6ba175e4d4ef\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") "
Nov 03 02:02:28.671864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.670798    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-certs\") pod \"23107beb-9060-480d-9f1b-6ba175e4d4ef\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") "
Nov 03 02:02:28.671864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.670830    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23107beb-9060-480d-9f1b-6ba175e4d4ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23107beb-9060-480d-9f1b-6ba175e4d4ef-istiod-ca-cert\") pod \"23107beb-9060-480d-9f1b-6ba175e4d4ef\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") "
Nov 03 02:02:28.671864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.670869    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-socket\") pod \"23107beb-9060-480d-9f1b-6ba175e4d4ef\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") "
Nov 03 02:02:28.671864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.670914    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-token\" (UniqueName: \"kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-token\") pod \"23107beb-9060-480d-9f1b-6ba175e4d4ef\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") "
Nov 03 02:02:28.671864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.670954    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-data\") pod \"23107beb-9060-480d-9f1b-6ba175e4d4ef\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") "
Nov 03 02:02:28.671864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.671055    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-podinfo\") pod \"23107beb-9060-480d-9f1b-6ba175e4d4ef\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") "
Nov 03 02:02:28.671864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.671112    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-envoy\") pod \"23107beb-9060-480d-9f1b-6ba175e4d4ef\" (UID: \"23107beb-9060-480d-9f1b-6ba175e4d4ef\") "
Nov 03 02:02:28.678622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.677296    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23107beb-9060-480d-9f1b-6ba175e4d4ef-istiod-ca-cert" pod "23107beb-9060-480d-9f1b-6ba175e4d4ef" (UID: "23107beb-9060-480d-9f1b-6ba175e4d4ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:28.678622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.677597    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-certs" pod "23107beb-9060-480d-9f1b-6ba175e4d4ef" (UID: "23107beb-9060-480d-9f1b-6ba175e4d4ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:28.678622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.677795    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-socket" pod "23107beb-9060-480d-9f1b-6ba175e4d4ef" (UID: "23107beb-9060-480d-9f1b-6ba175e4d4ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:28.678622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.678029    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-data" pod "23107beb-9060-480d-9f1b-6ba175e4d4ef" (UID: "23107beb-9060-480d-9f1b-6ba175e4d4ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:28.678622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.678053    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-credential-socket" pod "23107beb-9060-480d-9f1b-6ba175e4d4ef" (UID: "23107beb-9060-480d-9f1b-6ba175e4d4ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:28.679603 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.679118    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-kube-api-access-w8psh" pod "23107beb-9060-480d-9f1b-6ba175e4d4ef" (UID: "23107beb-9060-480d-9f1b-6ba175e4d4ef"). InnerVolumeSpecName "kube-api-access-w8psh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:28.680640 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.680613    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-podinfo" pod "23107beb-9060-480d-9f1b-6ba175e4d4ef" (UID: "23107beb-9060-480d-9f1b-6ba175e4d4ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:02:28.681693 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.681659    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-token" pod "23107beb-9060-480d-9f1b-6ba175e4d4ef" (UID: "23107beb-9060-480d-9f1b-6ba175e4d4ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:28.681781 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.681702    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-envoy" pod "23107beb-9060-480d-9f1b-6ba175e4d4ef" (UID: "23107beb-9060-480d-9f1b-6ba175e4d4ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:02:28.694848 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.694826    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867f1e43-883e-44ba-83ed-a3e19a552af0" path="/var/lib/kubelet/pods/867f1e43-883e-44ba-83ed-a3e19a552af0/volumes"
Nov 03 02:02:28.774604 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.774573    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:28.774604 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.774599    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:28.774812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.774613    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:28.774812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.774627    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:28.774812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.774640    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:28.774812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.774652    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8psh\" (UniqueName: \"kubernetes.io/projected/23107beb-9060-480d-9f1b-6ba175e4d4ef-kube-api-access-w8psh\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:28.774812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.774664    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:28.774812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.774675    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23107beb-9060-480d-9f1b-6ba175e4d4ef-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:28.774812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.774689    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23107beb-9060-480d-9f1b-6ba175e4d4ef-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:28.894029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.893951    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"]
Nov 03 02:02:28.894504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.894485    5508 cpu_manager.go:414] "containerMap: removing container" podUID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerName="istio-proxy"
Nov 03 02:02:28.894607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.894505    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerName="istio-proxy"
Nov 03 02:02:28.894607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.894540    5508 cpu_manager.go:414] "containerMap: removing container" podUID="97fb68aa-4dba-43d3-9eda-d0262926d4e9" containerName="revoke-keys"
Nov 03 02:02:28.894607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.894548    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fb68aa-4dba-43d3-9eda-d0262926d4e9" containerName="revoke-keys"
Nov 03 02:02:28.894607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.894559    5508 cpu_manager.go:414] "containerMap: removing container" podUID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerName="istio-proxy"
Nov 03 02:02:28.894607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.894566    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerName="istio-proxy"
Nov 03 02:02:28.894885 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.894702    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerName="istio-proxy"
Nov 03 02:02:28.894885 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.894719    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fb68aa-4dba-43d3-9eda-d0262926d4e9" containerName="revoke-keys"
Nov 03 02:02:28.899333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.899312    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"
Nov 03 02:02:28.948131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.948103    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:02:28.948299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.948138    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:02:28.977369 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.969080    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"]
Nov 03 02:02:28.977369 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.977041    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a57d3de2-lvzzb\" (UID: \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"
Nov 03 02:02:28.977369 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.977275    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4m4g\" (UniqueName: \"kubernetes.io/projected/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-kube-api-access-b4m4g\") pod \"maas-api-revoke-sub-a57d3de2-lvzzb\" (UID: \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"
Nov 03 02:02:28.977369 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.977328    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a57d3de2-lvzzb\" (UID: \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"
Nov 03 02:02:28.983918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.983893    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"]
Nov 03 02:02:28.989627 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:28.989597    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-db583d-openshift-default-6d65dc8cbf-6zkl5"]
Nov 03 02:02:29.068778 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:29.068735    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 03 02:02:29.069107 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:29.069076    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp" podUID="41ea6070-8986-41ae-98be-6e20baf08648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 03 02:02:29.080372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:29.080068    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a57d3de2-lvzzb\" (UID: \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"
Nov 03 02:02:29.080372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:29.080110    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4m4g\" (UniqueName: \"kubernetes.io/projected/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-kube-api-access-b4m4g\") pod \"maas-api-revoke-sub-a57d3de2-lvzzb\" (UID: \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"
Nov 03 02:02:29.080372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:29.080158    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a57d3de2-lvzzb\" (UID: \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"
Nov 03 02:02:29.081527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:29.081501    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a57d3de2-lvzzb\" (UID: \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"
Nov 03 02:02:29.083329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:29.083303    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a57d3de2-lvzzb\" (UID: \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"
Nov 03 02:02:29.137874 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:29.137843    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4m4g\" (UniqueName: \"kubernetes.io/projected/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-kube-api-access-b4m4g\") pod \"maas-api-revoke-sub-a57d3de2-lvzzb\" (UID: \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"
Nov 03 02:02:29.212199 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:29.212163    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"
Nov 03 02:02:29.606931 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:29.606851    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"]
Nov 03 02:02:29.614487 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:29.614444    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cff4f0a_4a78_420a_91e2_a9cb9dd7fb13.slice/crio-c7cb0bc0f3da5497a6204f768c25a444f071720a579ca8e536153a92ed1ad7c8 WatchSource:0}: Error finding container c7cb0bc0f3da5497a6204f768c25a444f071720a579ca8e536153a92ed1ad7c8: Status 404 returned error can't find the container with id c7cb0bc0f3da5497a6204f768c25a444f071720a579ca8e536153a92ed1ad7c8
Nov 03 02:02:30.066164 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:30.066123    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 03 02:02:30.066337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:30.066193    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp" podUID="41ea6070-8986-41ae-98be-6e20baf08648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 03 02:02:30.589276 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:30.589225    5508 generic.go:356] "Generic (PLEG): container finished" podID="5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13" containerID="a2c6540eb9bea5c2c2c0b2ec5d43fcc23d3fa6b3a16e3bed27ee7cbe81deb815" exitCode=0
Nov 03 02:02:30.589429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:30.589281    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb" event={"ID":"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13","Type":"ContainerDied","Data":"a2c6540eb9bea5c2c2c0b2ec5d43fcc23d3fa6b3a16e3bed27ee7cbe81deb815"}
Nov 03 02:02:30.589429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:30.589338    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb" event={"ID":"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13","Type":"ContainerStarted","Data":"c7cb0bc0f3da5497a6204f768c25a444f071720a579ca8e536153a92ed1ad7c8"}
Nov 03 02:02:30.690776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:30.690747    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23107beb-9060-480d-9f1b-6ba175e4d4ef" path="/var/lib/kubelet/pods/23107beb-9060-480d-9f1b-6ba175e4d4ef/volumes"
Nov 03 02:02:31.217623 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.215187    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:31.217623 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.215638    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:31.217623 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.217387    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:02:31.739468 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.739446    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"
Nov 03 02:02:31.808204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.808179    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-kube-api-access-b4m4g\" (UniqueName: \"kubernetes.io/projected/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-kube-api-access-b4m4g\") pod \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\" (UID: \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\") "
Nov 03 02:02:31.808370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.808266    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-openshift-ca\") pod \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\" (UID: \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\") "
Nov 03 02:02:31.808370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.808341    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-xks-ca\") pod \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\" (UID: \"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13\") "
Nov 03 02:02:31.808717 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.808687    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-openshift-ca" pod "5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13" (UID: "5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:31.810385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.810364    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-xks-ca" pod "5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13" (UID: "5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:31.810484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.810390    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-kube-api-access-b4m4g" pod "5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13" (UID: "5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13"). InnerVolumeSpecName "kube-api-access-b4m4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:31.909419 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.909394    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:31.909419 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.909418    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4m4g\" (UniqueName: \"kubernetes.io/projected/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-kube-api-access-b4m4g\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:31.909589 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.909432    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:31.922870 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.922841    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"]
Nov 03 02:02:31.923285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.923272    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13" containerName="revoke-keys"
Nov 03 02:02:31.923331 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.923287    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13" containerName="revoke-keys"
Nov 03 02:02:31.923386 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.923376    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="23107beb-9060-480d-9f1b-6ba175e4d4ef" containerName="istio-proxy"
Nov 03 02:02:31.923430 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.923386    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13" containerName="revoke-keys"
Nov 03 02:02:31.926898 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.926875    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"
Nov 03 02:02:31.985324 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:31.985289    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"]
Nov 03 02:02:32.010001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.009968    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1af96121-wcvcb\" (UID: \"90e57e5e-1860-4edd-8e96-77f2e4140313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"
Nov 03 02:02:32.010176 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.010106    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1af96121-wcvcb\" (UID: \"90e57e5e-1860-4edd-8e96-77f2e4140313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"
Nov 03 02:02:32.010258 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.010238    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58j2g\" (UniqueName: \"kubernetes.io/projected/90e57e5e-1860-4edd-8e96-77f2e4140313-kube-api-access-58j2g\") pod \"maas-api-revoke-sub-1af96121-wcvcb\" (UID: \"90e57e5e-1860-4edd-8e96-77f2e4140313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"
Nov 03 02:02:32.111548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.111516    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58j2g\" (UniqueName: \"kubernetes.io/projected/90e57e5e-1860-4edd-8e96-77f2e4140313-kube-api-access-58j2g\") pod \"maas-api-revoke-sub-1af96121-wcvcb\" (UID: \"90e57e5e-1860-4edd-8e96-77f2e4140313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"
Nov 03 02:02:32.111722 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.111601    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1af96121-wcvcb\" (UID: \"90e57e5e-1860-4edd-8e96-77f2e4140313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"
Nov 03 02:02:32.111722 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.111647    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1af96121-wcvcb\" (UID: \"90e57e5e-1860-4edd-8e96-77f2e4140313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"
Nov 03 02:02:32.112283 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.112250    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1af96121-wcvcb\" (UID: \"90e57e5e-1860-4edd-8e96-77f2e4140313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"
Nov 03 02:02:32.113859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.113842    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1af96121-wcvcb\" (UID: \"90e57e5e-1860-4edd-8e96-77f2e4140313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"
Nov 03 02:02:32.168506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.168438    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58j2g\" (UniqueName: \"kubernetes.io/projected/90e57e5e-1860-4edd-8e96-77f2e4140313-kube-api-access-58j2g\") pod \"maas-api-revoke-sub-1af96121-wcvcb\" (UID: \"90e57e5e-1860-4edd-8e96-77f2e4140313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"
Nov 03 02:02:32.240078 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.240046    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"
Nov 03 02:02:32.603683 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.603604    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"
Nov 03 02:02:32.603683 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.603636    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb" event={"ID":"5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13","Type":"ContainerDied","Data":"c7cb0bc0f3da5497a6204f768c25a444f071720a579ca8e536153a92ed1ad7c8"}
Nov 03 02:02:32.603871 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.603708    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7cb0bc0f3da5497a6204f768c25a444f071720a579ca8e536153a92ed1ad7c8"
Nov 03 02:02:32.614954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:32.614928    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"]
Nov 03 02:02:32.622502 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:32.622477    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e57e5e_1860_4edd_8e96_77f2e4140313.slice/crio-e1225dcc3a8e48f0146a5ce597c059740b931248033165f4ea15dadcf09d37e4 WatchSource:0}: Error finding container e1225dcc3a8e48f0146a5ce597c059740b931248033165f4ea15dadcf09d37e4: Status 404 returned error can't find the container with id e1225dcc3a8e48f0146a5ce597c059740b931248033165f4ea15dadcf09d37e4
Nov 03 02:02:33.611420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.611382    5508 generic.go:356] "Generic (PLEG): container finished" podID="90e57e5e-1860-4edd-8e96-77f2e4140313" containerID="e6e8c3e2c4aa66e4c76419c6c36da947acede60eb8243f93e59bb0a51e5959f3" exitCode=0
Nov 03 02:02:33.611848 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.611543    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb" event={"ID":"90e57e5e-1860-4edd-8e96-77f2e4140313","Type":"ContainerDied","Data":"e6e8c3e2c4aa66e4c76419c6c36da947acede60eb8243f93e59bb0a51e5959f3"}
Nov 03 02:02:33.611848 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.611611    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb" event={"ID":"90e57e5e-1860-4edd-8e96-77f2e4140313","Type":"ContainerStarted","Data":"e1225dcc3a8e48f0146a5ce597c059740b931248033165f4ea15dadcf09d37e4"}
Nov 03 02:02:33.702371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.702310    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"]
Nov 03 02:02:33.706148 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.706125    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"
Nov 03 02:02:33.763857 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.763830    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"]
Nov 03 02:02:33.830635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.830556    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26672c2-1d84-4680-92c2-342140350153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79b49194-nxs82\" (UID: \"e26672c2-1d84-4680-92c2-342140350153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"
Nov 03 02:02:33.830635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.830616    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9drd6\" (UniqueName: \"kubernetes.io/projected/e26672c2-1d84-4680-92c2-342140350153-kube-api-access-9drd6\") pod \"maas-api-revoke-sub-79b49194-nxs82\" (UID: \"e26672c2-1d84-4680-92c2-342140350153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"
Nov 03 02:02:33.830827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.830742    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26672c2-1d84-4680-92c2-342140350153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79b49194-nxs82\" (UID: \"e26672c2-1d84-4680-92c2-342140350153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"
Nov 03 02:02:33.932344 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.932308    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9drd6\" (UniqueName: \"kubernetes.io/projected/e26672c2-1d84-4680-92c2-342140350153-kube-api-access-9drd6\") pod \"maas-api-revoke-sub-79b49194-nxs82\" (UID: \"e26672c2-1d84-4680-92c2-342140350153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"
Nov 03 02:02:33.932518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.932373    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26672c2-1d84-4680-92c2-342140350153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79b49194-nxs82\" (UID: \"e26672c2-1d84-4680-92c2-342140350153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"
Nov 03 02:02:33.932518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.932451    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26672c2-1d84-4680-92c2-342140350153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79b49194-nxs82\" (UID: \"e26672c2-1d84-4680-92c2-342140350153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"
Nov 03 02:02:33.933056 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.933036    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26672c2-1d84-4680-92c2-342140350153-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79b49194-nxs82\" (UID: \"e26672c2-1d84-4680-92c2-342140350153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"
Nov 03 02:02:33.934875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.934851    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26672c2-1d84-4680-92c2-342140350153-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79b49194-nxs82\" (UID: \"e26672c2-1d84-4680-92c2-342140350153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"
Nov 03 02:02:33.989084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:33.989053    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9drd6\" (UniqueName: \"kubernetes.io/projected/e26672c2-1d84-4680-92c2-342140350153-kube-api-access-9drd6\") pod \"maas-api-revoke-sub-79b49194-nxs82\" (UID: \"e26672c2-1d84-4680-92c2-342140350153\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"
Nov 03 02:02:34.027398 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.026559    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"
Nov 03 02:02:34.165989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.165956    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"]
Nov 03 02:02:34.171145 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.171120    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"
Nov 03 02:02:34.206440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.206419    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"]
Nov 03 02:02:34.209919 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.209902    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.223321 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.223302    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b22fcc\""
Nov 03 02:02:34.223705 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.223690    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b22fcc-dockercfg-jkk9r\""
Nov 03 02:02:34.229197 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.229177    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"]
Nov 03 02:02:34.236285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.236265    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df245af-91c3-42d2-9a61-bff747cd1701-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58\" (UID: \"7df245af-91c3-42d2-9a61-bff747cd1701\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"
Nov 03 02:02:34.236417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.236334    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8bs8\" (UniqueName: \"kubernetes.io/projected/7df245af-91c3-42d2-9a61-bff747cd1701-kube-api-access-b8bs8\") pod \"payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58\" (UID: \"7df245af-91c3-42d2-9a61-bff747cd1701\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"
Nov 03 02:02:34.301393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.296832    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"]
Nov 03 02:02:34.323379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.320732    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"]
Nov 03 02:02:34.329504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.328488    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0bf34a-openshift-default-dockercfg-s9n2w\""
Nov 03 02:02:34.330699 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.330673    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"
Nov 03 02:02:34.337739 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.337712    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.337877 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.337757    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.337877 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.337845    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df245af-91c3-42d2-9a61-bff747cd1701-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58\" (UID: \"7df245af-91c3-42d2-9a61-bff747cd1701\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"
Nov 03 02:02:34.337982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.337888    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.337982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.337942    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.337982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.337970    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q64gn\" (UniqueName: \"kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-kube-api-access-q64gn\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.338096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.337998    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.338096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.338033    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.338096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.338058    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.338096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.338084    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.338241 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.338122    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8bs8\" (UniqueName: \"kubernetes.io/projected/7df245af-91c3-42d2-9a61-bff747cd1701-kube-api-access-b8bs8\") pod \"payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58\" (UID: \"7df245af-91c3-42d2-9a61-bff747cd1701\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"
Nov 03 02:02:34.339601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.339578    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df245af-91c3-42d2-9a61-bff747cd1701-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58\" (UID: \"7df245af-91c3-42d2-9a61-bff747cd1701\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"
Nov 03 02:02:34.381144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.381069    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"]
Nov 03 02:02:34.395410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.395338    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8bs8\" (UniqueName: \"kubernetes.io/projected/7df245af-91c3-42d2-9a61-bff747cd1701-kube-api-access-b8bs8\") pod \"payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58\" (UID: \"7df245af-91c3-42d2-9a61-bff747cd1701\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"
Nov 03 02:02:34.439145 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439110    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.439310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439154    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.439310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439175    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pphww\" (UniqueName: \"kubernetes.io/projected/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-kube-api-access-pphww\") pod \"payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng\" (UID: \"a5e3b594-c7a5-4f96-9c13-d9789331f2ef\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"
Nov 03 02:02:34.439310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439213    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.439310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439231    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q64gn\" (UniqueName: \"kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-kube-api-access-q64gn\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.439310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439249    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng\" (UID: \"a5e3b594-c7a5-4f96-9c13-d9789331f2ef\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"
Nov 03 02:02:34.439310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439271    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.439310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439305    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.439690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439331    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.439690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439397    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.439690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439625    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.439849 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439742    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.439903 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439884    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.439938 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439913    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.439973 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439943    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.440010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.439975    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.441791 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.441764    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.441888 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.441826    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.446591 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.446565    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"]
Nov 03 02:02:34.452506 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:34.452484    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26672c2_1d84_4680_92c2_342140350153.slice/crio-efee56f76d51e7408f2b8f025f0285efcbbeb5a932a76bfaababa53dd5ce2f10 WatchSource:0}: Error finding container efee56f76d51e7408f2b8f025f0285efcbbeb5a932a76bfaababa53dd5ce2f10: Status 404 returned error can't find the container with id efee56f76d51e7408f2b8f025f0285efcbbeb5a932a76bfaababa53dd5ce2f10
Nov 03 02:02:34.484277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.484254    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"
Nov 03 02:02:34.496377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.496333    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.496462 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.496420    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q64gn\" (UniqueName: \"kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-kube-api-access-q64gn\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.522418 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.522384    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:34.542533 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.540793    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng\" (UID: \"a5e3b594-c7a5-4f96-9c13-d9789331f2ef\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"
Nov 03 02:02:34.542533 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.540926    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphww\" (UniqueName: \"kubernetes.io/projected/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-kube-api-access-pphww\") pod \"payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng\" (UID: \"a5e3b594-c7a5-4f96-9c13-d9789331f2ef\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"
Nov 03 02:02:34.542533 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.541899    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng\" (UID: \"a5e3b594-c7a5-4f96-9c13-d9789331f2ef\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"
Nov 03 02:02:34.619210 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.619168    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82" event={"ID":"e26672c2-1d84-4680-92c2-342140350153","Type":"ContainerStarted","Data":"efee56f76d51e7408f2b8f025f0285efcbbeb5a932a76bfaababa53dd5ce2f10"}
Nov 03 02:02:34.650088 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.650055    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphww\" (UniqueName: \"kubernetes.io/projected/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-kube-api-access-pphww\") pod \"payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng\" (UID: \"a5e3b594-c7a5-4f96-9c13-d9789331f2ef\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"
Nov 03 02:02:34.799094 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.799069    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"
Nov 03 02:02:34.861963 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.861938    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"]
Nov 03 02:02:34.869306 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:34.869283    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df245af_91c3_42d2_9a61_bff747cd1701.slice/crio-ca4946d202a251bdab8854f804ffd5d25e2a2a87225666415cddb82800764ac6 WatchSource:0}: Error finding container ca4946d202a251bdab8854f804ffd5d25e2a2a87225666415cddb82800764ac6: Status 404 returned error can't find the container with id ca4946d202a251bdab8854f804ffd5d25e2a2a87225666415cddb82800764ac6
Nov 03 02:02:34.945829 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.945748    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-xks-ca\") pod \"90e57e5e-1860-4edd-8e96-77f2e4140313\" (UID: \"90e57e5e-1860-4edd-8e96-77f2e4140313\") "
Nov 03 02:02:34.945829 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.945790    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90e57e5e-1860-4edd-8e96-77f2e4140313-kube-api-access-58j2g\" (UniqueName: \"kubernetes.io/projected/90e57e5e-1860-4edd-8e96-77f2e4140313-kube-api-access-58j2g\") pod \"90e57e5e-1860-4edd-8e96-77f2e4140313\" (UID: \"90e57e5e-1860-4edd-8e96-77f2e4140313\") "
Nov 03 02:02:34.946047 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.945832    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-openshift-ca\") pod \"90e57e5e-1860-4edd-8e96-77f2e4140313\" (UID: \"90e57e5e-1860-4edd-8e96-77f2e4140313\") "
Nov 03 02:02:34.946281 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.946249    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-openshift-ca" pod "90e57e5e-1860-4edd-8e96-77f2e4140313" (UID: "90e57e5e-1860-4edd-8e96-77f2e4140313"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:34.947415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.947372    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"
Nov 03 02:02:34.949239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.949205    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e57e5e-1860-4edd-8e96-77f2e4140313-kube-api-access-58j2g" pod "90e57e5e-1860-4edd-8e96-77f2e4140313" (UID: "90e57e5e-1860-4edd-8e96-77f2e4140313"). InnerVolumeSpecName "kube-api-access-58j2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:34.949416 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:34.949240    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-xks-ca" pod "90e57e5e-1860-4edd-8e96-77f2e4140313" (UID: "90e57e5e-1860-4edd-8e96-77f2e4140313"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:35.005480 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.005447    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"]
Nov 03 02:02:35.014872 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:35.014838    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad7557e_8b17_47c6_8ae5_86aee55e2c79.slice/crio-bdcecc3c9cac5197f4283bcc740efa8c385e1ea16102e6e49b76dde143b25e50 WatchSource:0}: Error finding container bdcecc3c9cac5197f4283bcc740efa8c385e1ea16102e6e49b76dde143b25e50: Status 404 returned error can't find the container with id bdcecc3c9cac5197f4283bcc740efa8c385e1ea16102e6e49b76dde143b25e50
Nov 03 02:02:35.018723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.018497    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:02:35.018723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.018605    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:02:35.018723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.018671    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:02:35.047390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.046986    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:35.047390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.047024    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58j2g\" (UniqueName: \"kubernetes.io/projected/90e57e5e-1860-4edd-8e96-77f2e4140313-kube-api-access-58j2g\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:35.047390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.047041    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90e57e5e-1860-4edd-8e96-77f2e4140313-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:35.344314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.344237    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"]
Nov 03 02:02:35.350061 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:35.350029    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e3b594_c7a5_4f96_9c13_d9789331f2ef.slice/crio-e364710c40a33ee4ed31db81e682c85a3e36b6011bc868ad5c60ea6960d7860f WatchSource:0}: Error finding container e364710c40a33ee4ed31db81e682c85a3e36b6011bc868ad5c60ea6960d7860f: Status 404 returned error can't find the container with id e364710c40a33ee4ed31db81e682c85a3e36b6011bc868ad5c60ea6960d7860f
Nov 03 02:02:35.626891 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.626861    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b" event={"ID":"fad7557e-8b17-47c6-8ae5-86aee55e2c79","Type":"ContainerStarted","Data":"c066a37ec8945da17654abcc599bf8792d1813113df4627e4fa970aa798f3dba"}
Nov 03 02:02:35.627235 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.626923    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b" event={"ID":"fad7557e-8b17-47c6-8ae5-86aee55e2c79","Type":"ContainerStarted","Data":"bdcecc3c9cac5197f4283bcc740efa8c385e1ea16102e6e49b76dde143b25e50"}
Nov 03 02:02:35.628718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.628693    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58" event={"ID":"7df245af-91c3-42d2-9a61-bff747cd1701","Type":"ContainerStarted","Data":"ab2d4f4a41029b6a7c006b3ba920858be7759d47d6a9d97b1839dc75f2dbebee"}
Nov 03 02:02:35.628826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.628747    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"
Nov 03 02:02:35.628826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.628772    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58" event={"ID":"7df245af-91c3-42d2-9a61-bff747cd1701","Type":"ContainerStarted","Data":"ca4946d202a251bdab8854f804ffd5d25e2a2a87225666415cddb82800764ac6"}
Nov 03 02:02:35.630232 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.630208    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng" event={"ID":"a5e3b594-c7a5-4f96-9c13-d9789331f2ef","Type":"ContainerStarted","Data":"a940ccac965c0f4778041b81558bbdd1fbe3eb552e30e5c58a31d0c8437e1930"}
Nov 03 02:02:35.630329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.630259    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng" event={"ID":"a5e3b594-c7a5-4f96-9c13-d9789331f2ef","Type":"ContainerStarted","Data":"e364710c40a33ee4ed31db81e682c85a3e36b6011bc868ad5c60ea6960d7860f"}
Nov 03 02:02:35.630329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.630291    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"
Nov 03 02:02:35.631516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.631496    5508 generic.go:356] "Generic (PLEG): container finished" podID="e26672c2-1d84-4680-92c2-342140350153" containerID="bc57ece718d325b6fdbf4c01f310fc05b33610c57216ca8588f57fc970df5e88" exitCode=0
Nov 03 02:02:35.631597 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.631576    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82" event={"ID":"e26672c2-1d84-4680-92c2-342140350153","Type":"ContainerDied","Data":"bc57ece718d325b6fdbf4c01f310fc05b33610c57216ca8588f57fc970df5e88"}
Nov 03 02:02:35.632714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.632697    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb" event={"ID":"90e57e5e-1860-4edd-8e96-77f2e4140313","Type":"ContainerDied","Data":"e1225dcc3a8e48f0146a5ce597c059740b931248033165f4ea15dadcf09d37e4"}
Nov 03 02:02:35.632714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.632713    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"
Nov 03 02:02:35.632842 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.632733    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1225dcc3a8e48f0146a5ce597c059740b931248033165f4ea15dadcf09d37e4"
Nov 03 02:02:35.742997 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:35.742949    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b" podStartSLOduration=1.7429348679999999 podStartE2EDuration="1.742934868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:35.739601475 +0000 UTC m=+2635.165207954" watchObservedRunningTime="2026-11-03 02:02:35.742934868 +0000 UTC m=+2635.168541331"
Nov 03 02:02:36.013966 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:36.013892    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58" podStartSLOduration=2.013873772 podStartE2EDuration="2.013873772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:36.013390889 +0000 UTC m=+2635.438997354" watchObservedRunningTime="2026-11-03 02:02:36.013873772 +0000 UTC m=+2635.439480239"
Nov 03 02:02:36.136820 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:36.133790    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng" podStartSLOduration=2.133768365 podStartE2EDuration="2.133768365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:36.13197915 +0000 UTC m=+2635.557585663" watchObservedRunningTime="2026-11-03 02:02:36.133768365 +0000 UTC m=+2635.559374831"
Nov 03 02:02:36.522614 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:36.522573    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:36.524171 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:36.524147    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 02:02:36.524305 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:36.524191    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b" podUID="fad7557e-8b17-47c6-8ae5-86aee55e2c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 02:02:36.843491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:36.843464    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"
Nov 03 02:02:36.989561 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:36.989524    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e26672c2-1d84-4680-92c2-342140350153-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26672c2-1d84-4680-92c2-342140350153-maas-api-openshift-ca\") pod \"e26672c2-1d84-4680-92c2-342140350153\" (UID: \"e26672c2-1d84-4680-92c2-342140350153\") "
Nov 03 02:02:36.989738 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:36.989656    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e26672c2-1d84-4680-92c2-342140350153-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26672c2-1d84-4680-92c2-342140350153-maas-api-xks-ca\") pod \"e26672c2-1d84-4680-92c2-342140350153\" (UID: \"e26672c2-1d84-4680-92c2-342140350153\") "
Nov 03 02:02:36.989738 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:36.989721    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e26672c2-1d84-4680-92c2-342140350153-kube-api-access-9drd6\" (UniqueName: \"kubernetes.io/projected/e26672c2-1d84-4680-92c2-342140350153-kube-api-access-9drd6\") pod \"e26672c2-1d84-4680-92c2-342140350153\" (UID: \"e26672c2-1d84-4680-92c2-342140350153\") "
Nov 03 02:02:36.989957 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:36.989935    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26672c2-1d84-4680-92c2-342140350153-maas-api-openshift-ca" pod "e26672c2-1d84-4680-92c2-342140350153" (UID: "e26672c2-1d84-4680-92c2-342140350153"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:36.990110 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:36.990089    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e26672c2-1d84-4680-92c2-342140350153-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:36.991720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:36.991689    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e26672c2-1d84-4680-92c2-342140350153-maas-api-xks-ca" pod "e26672c2-1d84-4680-92c2-342140350153" (UID: "e26672c2-1d84-4680-92c2-342140350153"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:36.991808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:36.991742    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26672c2-1d84-4680-92c2-342140350153-kube-api-access-9drd6" pod "e26672c2-1d84-4680-92c2-342140350153" (UID: "e26672c2-1d84-4680-92c2-342140350153"). InnerVolumeSpecName "kube-api-access-9drd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:37.091762 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:37.091689    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9drd6\" (UniqueName: \"kubernetes.io/projected/e26672c2-1d84-4680-92c2-342140350153-kube-api-access-9drd6\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:37.091762 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:37.091717    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e26672c2-1d84-4680-92c2-342140350153-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:37.523191 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:37.523143    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 02:02:37.523371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:37.523208    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b" podUID="fad7557e-8b17-47c6-8ae5-86aee55e2c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 02:02:37.645725 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:37.645694    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"
Nov 03 02:02:37.645879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:37.645719    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82" event={"ID":"e26672c2-1d84-4680-92c2-342140350153","Type":"ContainerDied","Data":"efee56f76d51e7408f2b8f025f0285efcbbeb5a932a76bfaababa53dd5ce2f10"}
Nov 03 02:02:37.645879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:37.645770    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efee56f76d51e7408f2b8f025f0285efcbbeb5a932a76bfaababa53dd5ce2f10"
Nov 03 02:02:38.523105 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:38.523070    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 02:02:38.523510 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:38.523121    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b" podUID="fad7557e-8b17-47c6-8ae5-86aee55e2c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 02:02:39.523457 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:39.523424    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 02:02:39.523866 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:39.523474    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b" podUID="fad7557e-8b17-47c6-8ae5-86aee55e2c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 02:02:40.523329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:40.523289    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 02:02:40.523518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:40.523377    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b" podUID="fad7557e-8b17-47c6-8ae5-86aee55e2c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 02:02:41.523414 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:41.523378    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 02:02:41.523613 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:41.523444    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b" podUID="fad7557e-8b17-47c6-8ae5-86aee55e2c79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 02:02:42.527208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:42.527177    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:42.527690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:42.527606    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:42.528371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:42.528333    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:02:43.492428 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:43.492394    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:02:43.811835 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:02:43.811756    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:02:46.642275 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.642243    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"
Nov 03 02:02:46.642744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.642323    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"
Nov 03 02:02:46.691198 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.691161    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"]
Nov 03 02:02:46.691602 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.691587    5508 cpu_manager.go:414] "containerMap: removing container" podUID="90e57e5e-1860-4edd-8e96-77f2e4140313" containerName="revoke-keys"
Nov 03 02:02:46.691660 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.691602    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90e57e5e-1860-4edd-8e96-77f2e4140313" containerName="revoke-keys"
Nov 03 02:02:46.691660 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.691629    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e26672c2-1d84-4680-92c2-342140350153" containerName="revoke-keys"
Nov 03 02:02:46.691660 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.691634    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e26672c2-1d84-4680-92c2-342140350153" containerName="revoke-keys"
Nov 03 02:02:46.691763 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.691701    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="90e57e5e-1860-4edd-8e96-77f2e4140313" containerName="revoke-keys"
Nov 03 02:02:46.691763 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.691714    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e26672c2-1d84-4680-92c2-342140350153" containerName="revoke-keys"
Nov 03 02:02:46.696958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.696939    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"
Nov 03 02:02:46.745937 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.745916    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:02:46.746096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.746078    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:02:46.748976 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.748955    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"]
Nov 03 02:02:46.794751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.794722    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ad60eed-jthg8\" (UID: \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"
Nov 03 02:02:46.794908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.794836    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztx5s\" (UniqueName: \"kubernetes.io/projected/da1775f2-b42d-4791-80ab-0ed77ac8acf3-kube-api-access-ztx5s\") pod \"maas-api-revoke-sub-9ad60eed-jthg8\" (UID: \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"
Nov 03 02:02:46.794908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.794889    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ad60eed-jthg8\" (UID: \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"
Nov 03 02:02:46.895647 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.895574    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ad60eed-jthg8\" (UID: \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"
Nov 03 02:02:46.895805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.895653    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ad60eed-jthg8\" (UID: \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"
Nov 03 02:02:46.895805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.895713    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztx5s\" (UniqueName: \"kubernetes.io/projected/da1775f2-b42d-4791-80ab-0ed77ac8acf3-kube-api-access-ztx5s\") pod \"maas-api-revoke-sub-9ad60eed-jthg8\" (UID: \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"
Nov 03 02:02:46.896289 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.896269    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ad60eed-jthg8\" (UID: \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"
Nov 03 02:02:46.897772 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.897745    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ad60eed-jthg8\" (UID: \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"
Nov 03 02:02:46.952504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:46.952464    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztx5s\" (UniqueName: \"kubernetes.io/projected/da1775f2-b42d-4791-80ab-0ed77ac8acf3-kube-api-access-ztx5s\") pod \"maas-api-revoke-sub-9ad60eed-jthg8\" (UID: \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"
Nov 03 02:02:47.009511 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.009479    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"
Nov 03 02:02:47.099493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.099465    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"]
Nov 03 02:02:47.100261 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.099690    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd" podUID="9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4" containerName="maas-api" containerID="cri-o://1e61dfc904622cc0cf35d3b0fca7f82cb5aeaab35fa4b483d5555ebfbd26c7f9" gracePeriod=30
Nov 03 02:02:47.371527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.371499    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:02:47.388475 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.388442    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"]
Nov 03 02:02:47.395422 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:47.395388    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1775f2_b42d_4791_80ab_0ed77ac8acf3.slice/crio-81597ef53bf27e325d7373124b727557e5884b4a4d14f2adf4cfa7151e13a5bc WatchSource:0}: Error finding container 81597ef53bf27e325d7373124b727557e5884b4a4d14f2adf4cfa7151e13a5bc: Status 404 returned error can't find the container with id 81597ef53bf27e325d7373124b727557e5884b4a4d14f2adf4cfa7151e13a5bc
Nov 03 02:02:47.399927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.399902    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-kube-api-access-wpwt9\" (UniqueName: \"kubernetes.io/projected/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-kube-api-access-wpwt9\") pod \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") "
Nov 03 02:02:47.400089 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.399954    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-maas-api-tls\") pod \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") "
Nov 03 02:02:47.400089 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.399986    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-openshift-service-ca\") pod \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") "
Nov 03 02:02:47.400089 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.400060    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-metrics-certs\") pod \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\" (UID: \"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4\") "
Nov 03 02:02:47.400635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.400611    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-openshift-service-ca" pod "9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4" (UID: "9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:47.402594 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.402570    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-maas-api-tls" pod "9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4" (UID: "9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:47.402703 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.402672    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-kube-api-access-wpwt9" pod "9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4" (UID: "9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4"). InnerVolumeSpecName "kube-api-access-wpwt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:47.405933 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.405904    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-metrics-certs" pod "9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4" (UID: "9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:47.502079 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.501997    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpwt9\" (UniqueName: \"kubernetes.io/projected/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-kube-api-access-wpwt9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:47.502640 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.502616    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:47.502826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.502810    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:47.502940 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.502926    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:47.744681 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.744599    5508 generic.go:356] "Generic (PLEG): container finished" podID="9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4" containerID="1e61dfc904622cc0cf35d3b0fca7f82cb5aeaab35fa4b483d5555ebfbd26c7f9" exitCode=0
Nov 03 02:02:47.745211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.744746    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd" event={"ID":"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4","Type":"ContainerDied","Data":"1e61dfc904622cc0cf35d3b0fca7f82cb5aeaab35fa4b483d5555ebfbd26c7f9"}
Nov 03 02:02:47.745211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.744801    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd" event={"ID":"9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4","Type":"ContainerDied","Data":"365db1dd29b870d39f9a4a47274ce715bf9d8703fcfdbd1ffad49dad4fde88d0"}
Nov 03 02:02:47.745211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.744835    5508 scope.go:122] "RemoveContainer" containerID="1e61dfc904622cc0cf35d3b0fca7f82cb5aeaab35fa4b483d5555ebfbd26c7f9"
Nov 03 02:02:47.745211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.744868    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"
Nov 03 02:02:47.747281 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.747258    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8" event={"ID":"da1775f2-b42d-4791-80ab-0ed77ac8acf3","Type":"ContainerStarted","Data":"3226670a10a4a5aad4ff1b0441f4d39f538c509c34580e30810431e0e987c0aa"}
Nov 03 02:02:47.747433 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.747305    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8" event={"ID":"da1775f2-b42d-4791-80ab-0ed77ac8acf3","Type":"ContainerStarted","Data":"81597ef53bf27e325d7373124b727557e5884b4a4d14f2adf4cfa7151e13a5bc"}
Nov 03 02:02:47.758030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.758010    5508 scope.go:122] "RemoveContainer" containerID="1e61dfc904622cc0cf35d3b0fca7f82cb5aeaab35fa4b483d5555ebfbd26c7f9"
Nov 03 02:02:47.758330 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:02:47.758305    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e61dfc904622cc0cf35d3b0fca7f82cb5aeaab35fa4b483d5555ebfbd26c7f9\": container with ID starting with 1e61dfc904622cc0cf35d3b0fca7f82cb5aeaab35fa4b483d5555ebfbd26c7f9 not found: ID does not exist" containerID="1e61dfc904622cc0cf35d3b0fca7f82cb5aeaab35fa4b483d5555ebfbd26c7f9"
Nov 03 02:02:47.758434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.758341    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e61dfc904622cc0cf35d3b0fca7f82cb5aeaab35fa4b483d5555ebfbd26c7f9"} err="failed to get container status \"1e61dfc904622cc0cf35d3b0fca7f82cb5aeaab35fa4b483d5555ebfbd26c7f9\": rpc error: code = NotFound desc = could not find container \"1e61dfc904622cc0cf35d3b0fca7f82cb5aeaab35fa4b483d5555ebfbd26c7f9\": container with ID starting with 1e61dfc904622cc0cf35d3b0fca7f82cb5aeaab35fa4b483d5555ebfbd26c7f9 not found: ID does not exist"
Nov 03 02:02:47.904302 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:47.904245    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8" podStartSLOduration=1.9042320560000001 podStartE2EDuration="1.904232056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:47.901786462 +0000 UTC m=+2647.327392944" watchObservedRunningTime="2026-11-03 02:02:47.904232056 +0000 UTC m=+2647.329838538"
Nov 03 02:02:48.007335 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:48.007250    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"]
Nov 03 02:02:48.011422 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:48.011390    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-e0907d-6fdc94698d-4lfkd"]
Nov 03 02:02:48.691157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:48.691121    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4" path="/var/lib/kubelet/pods/9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4/volumes"
Nov 03 02:02:48.755661 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:48.755628    5508 generic.go:356] "Generic (PLEG): container finished" podID="da1775f2-b42d-4791-80ab-0ed77ac8acf3" containerID="3226670a10a4a5aad4ff1b0441f4d39f538c509c34580e30810431e0e987c0aa" exitCode=0
Nov 03 02:02:48.756093 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:48.755683    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8" event={"ID":"da1775f2-b42d-4791-80ab-0ed77ac8acf3","Type":"ContainerDied","Data":"3226670a10a4a5aad4ff1b0441f4d39f538c509c34580e30810431e0e987c0aa"}
Nov 03 02:02:49.896202 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:49.896178    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"
Nov 03 02:02:49.938156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:49.938128    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-xks-ca\") pod \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\" (UID: \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\") "
Nov 03 02:02:49.938280 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:49.938179    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1775f2-b42d-4791-80ab-0ed77ac8acf3-kube-api-access-ztx5s\" (UniqueName: \"kubernetes.io/projected/da1775f2-b42d-4791-80ab-0ed77ac8acf3-kube-api-access-ztx5s\") pod \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\" (UID: \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\") "
Nov 03 02:02:49.938280 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:49.938249    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-openshift-ca\") pod \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\" (UID: \"da1775f2-b42d-4791-80ab-0ed77ac8acf3\") "
Nov 03 02:02:49.938665 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:49.938644    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-openshift-ca" pod "da1775f2-b42d-4791-80ab-0ed77ac8acf3" (UID: "da1775f2-b42d-4791-80ab-0ed77ac8acf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:49.938737 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:49.938724    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:49.940197 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:49.940168    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-xks-ca" pod "da1775f2-b42d-4791-80ab-0ed77ac8acf3" (UID: "da1775f2-b42d-4791-80ab-0ed77ac8acf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:49.940293 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:49.940250    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1775f2-b42d-4791-80ab-0ed77ac8acf3-kube-api-access-ztx5s" pod "da1775f2-b42d-4791-80ab-0ed77ac8acf3" (UID: "da1775f2-b42d-4791-80ab-0ed77ac8acf3"). InnerVolumeSpecName "kube-api-access-ztx5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:50.040196 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:50.040111    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da1775f2-b42d-4791-80ab-0ed77ac8acf3-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:50.040196 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:50.040150    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztx5s\" (UniqueName: \"kubernetes.io/projected/da1775f2-b42d-4791-80ab-0ed77ac8acf3-kube-api-access-ztx5s\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:50.769467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:50.769442    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"
Nov 03 02:02:50.769647 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:50.769505    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8" event={"ID":"da1775f2-b42d-4791-80ab-0ed77ac8acf3","Type":"ContainerDied","Data":"81597ef53bf27e325d7373124b727557e5884b4a4d14f2adf4cfa7151e13a5bc"}
Nov 03 02:02:50.769647 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:50.769557    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81597ef53bf27e325d7373124b727557e5884b4a4d14f2adf4cfa7151e13a5bc"
Nov 03 02:02:51.413002 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.412966    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"]
Nov 03 02:02:51.413442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.413414    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4" containerName="maas-api"
Nov 03 02:02:51.413442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.413426    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4" containerName="maas-api"
Nov 03 02:02:51.413527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.413464    5508 cpu_manager.go:414] "containerMap: removing container" podUID="da1775f2-b42d-4791-80ab-0ed77ac8acf3" containerName="revoke-keys"
Nov 03 02:02:51.413527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.413469    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1775f2-b42d-4791-80ab-0ed77ac8acf3" containerName="revoke-keys"
Nov 03 02:02:51.413610 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.413598    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1c45d1-f0d4-44a3-b99e-bcb07d09d2b4" containerName="maas-api"
Nov 03 02:02:51.413663 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.413629    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1775f2-b42d-4791-80ab-0ed77ac8acf3" containerName="revoke-keys"
Nov 03 02:02:51.417139 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.417118    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"
Nov 03 02:02:51.455814 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.455776    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59s7\" (UniqueName: \"kubernetes.io/projected/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-kube-api-access-j59s7\") pod \"maas-api-revoke-sub-931404d5-4v24x\" (UID: \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"
Nov 03 02:02:51.455987 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.455837    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-931404d5-4v24x\" (UID: \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"
Nov 03 02:02:51.455987 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.455946    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-931404d5-4v24x\" (UID: \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"
Nov 03 02:02:51.512122 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.512084    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:02:51.512122 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.512107    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:02:51.512411 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.512382    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:02:51.512475 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.512443    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:02:51.515242 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.515219    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"]
Nov 03 02:02:51.557197 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.557159    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59s7\" (UniqueName: \"kubernetes.io/projected/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-kube-api-access-j59s7\") pod \"maas-api-revoke-sub-931404d5-4v24x\" (UID: \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"
Nov 03 02:02:51.557422 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.557217    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-931404d5-4v24x\" (UID: \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"
Nov 03 02:02:51.557422 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.557266    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-931404d5-4v24x\" (UID: \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"
Nov 03 02:02:51.557929 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.557909    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-931404d5-4v24x\" (UID: \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"
Nov 03 02:02:51.559653 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.559633    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-931404d5-4v24x\" (UID: \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"
Nov 03 02:02:51.614451 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.614425    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59s7\" (UniqueName: \"kubernetes.io/projected/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-kube-api-access-j59s7\") pod \"maas-api-revoke-sub-931404d5-4v24x\" (UID: \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"
Nov 03 02:02:51.730729 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:51.730638    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"
Nov 03 02:02:52.101384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:52.101297    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"]
Nov 03 02:02:52.108200 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:52.108171    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd52e2c_f248_4dc9_b311_6948ffbb71f4.slice/crio-555f7d43ffd1b5bd74d332da8f8ee2a413ad8f29cd75d0ee4d32da15a955fae0 WatchSource:0}: Error finding container 555f7d43ffd1b5bd74d332da8f8ee2a413ad8f29cd75d0ee4d32da15a955fae0: Status 404 returned error can't find the container with id 555f7d43ffd1b5bd74d332da8f8ee2a413ad8f29cd75d0ee4d32da15a955fae0
Nov 03 02:02:52.783967 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:52.783929    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x" event={"ID":"2cd52e2c-f248-4dc9-b311-6948ffbb71f4","Type":"ContainerStarted","Data":"dfbce68f9500614f74e440057cd8dd6026e556432ed38135e4f5bf850ddc79c3"}
Nov 03 02:02:52.784420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:52.783988    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x" event={"ID":"2cd52e2c-f248-4dc9-b311-6948ffbb71f4","Type":"ContainerStarted","Data":"555f7d43ffd1b5bd74d332da8f8ee2a413ad8f29cd75d0ee4d32da15a955fae0"}
Nov 03 02:02:52.925681 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:52.925629    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x" podStartSLOduration=1.9256151959999999 podStartE2EDuration="1.925615196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:52.923941448 +0000 UTC m=+2652.349547911" watchObservedRunningTime="2026-11-03 02:02:52.925615196 +0000 UTC m=+2652.351221655"
Nov 03 02:02:53.791695 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:53.791654    5508 generic.go:356] "Generic (PLEG): container finished" podID="2cd52e2c-f248-4dc9-b311-6948ffbb71f4" containerID="dfbce68f9500614f74e440057cd8dd6026e556432ed38135e4f5bf850ddc79c3" exitCode=0
Nov 03 02:02:53.792175 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:53.791730    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x" event={"ID":"2cd52e2c-f248-4dc9-b311-6948ffbb71f4","Type":"ContainerDied","Data":"dfbce68f9500614f74e440057cd8dd6026e556432ed38135e4f5bf850ddc79c3"}
Nov 03 02:02:54.945518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:54.945490    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"
Nov 03 02:02:54.991127 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:54.991097    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-xks-ca\") pod \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\" (UID: \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\") "
Nov 03 02:02:54.991316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:54.991141    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-kube-api-access-j59s7\" (UniqueName: \"kubernetes.io/projected/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-kube-api-access-j59s7\") pod \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\" (UID: \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\") "
Nov 03 02:02:54.991316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:54.991169    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-openshift-ca\") pod \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\" (UID: \"2cd52e2c-f248-4dc9-b311-6948ffbb71f4\") "
Nov 03 02:02:54.991652 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:54.991632    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-openshift-ca" pod "2cd52e2c-f248-4dc9-b311-6948ffbb71f4" (UID: "2cd52e2c-f248-4dc9-b311-6948ffbb71f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:02:54.993052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:54.993022    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-xks-ca" pod "2cd52e2c-f248-4dc9-b311-6948ffbb71f4" (UID: "2cd52e2c-f248-4dc9-b311-6948ffbb71f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:02:54.993169 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:54.993156    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-kube-api-access-j59s7" pod "2cd52e2c-f248-4dc9-b311-6948ffbb71f4" (UID: "2cd52e2c-f248-4dc9-b311-6948ffbb71f4"). InnerVolumeSpecName "kube-api-access-j59s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:02:55.092380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:55.092270    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:55.092380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:55.092302    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j59s7\" (UniqueName: \"kubernetes.io/projected/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-kube-api-access-j59s7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:55.092380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:55.092313    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd52e2c-f248-4dc9-b311-6948ffbb71f4-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:02:55.805431 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:55.805405    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"
Nov 03 02:02:55.805624 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:55.805406    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x" event={"ID":"2cd52e2c-f248-4dc9-b311-6948ffbb71f4","Type":"ContainerDied","Data":"555f7d43ffd1b5bd74d332da8f8ee2a413ad8f29cd75d0ee4d32da15a955fae0"}
Nov 03 02:02:55.805624 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:55.805527    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="555f7d43ffd1b5bd74d332da8f8ee2a413ad8f29cd75d0ee4d32da15a955fae0"
Nov 03 02:02:58.132262 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.132221    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"]
Nov 03 02:02:58.133296 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.133267    5508 cpu_manager.go:414] "containerMap: removing container" podUID="2cd52e2c-f248-4dc9-b311-6948ffbb71f4" containerName="revoke-keys"
Nov 03 02:02:58.133296 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.133294    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd52e2c-f248-4dc9-b311-6948ffbb71f4" containerName="revoke-keys"
Nov 03 02:02:58.133570 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.133554    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd52e2c-f248-4dc9-b311-6948ffbb71f4" containerName="revoke-keys"
Nov 03 02:02:58.139462 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.139436    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.196894 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.196866    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e1793d-openshift-default-dockercfg-4jbqs\""
Nov 03 02:02:58.223246 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.223217    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.223390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.223259    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgp4f\" (UniqueName: \"kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-kube-api-access-wgp4f\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.223438 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.223394    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.223438 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.223416    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.223505 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.223488    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.223548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.223536    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.223591 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.223557    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.223591 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.223579    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.223673 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.223654    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.246617 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.246589    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"]
Nov 03 02:02:58.324408 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.324339    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.324607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.324419    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgp4f\" (UniqueName: \"kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-kube-api-access-wgp4f\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.324607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.324477    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.324607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.324506    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.324607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.324549    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.324838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.324613    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.324838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.324639    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.324838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.324669    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.324838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.324697    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.325057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.324829    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.325057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.324938    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-data\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.325165 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.325085    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-credential-socket\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.325231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.325195    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istiod-ca-cert\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.325288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.325228    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-certs\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.326685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.326662    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-envoy\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.327050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.327022    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-podinfo\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.390422 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.385129    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-token\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.390422 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.385185    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgp4f\" (UniqueName: \"kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-kube-api-access-wgp4f\") pod \"e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") " pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.452556 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.452528    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:02:58.884682 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.884646    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"]
Nov 03 02:02:58.891682 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:02:58.891651    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bbc58e2_b00c_4bc7_ba27_cabe0d546740.slice/crio-b247b27313d9bc39ad377c915029cab81c243b80e5cfcf0ee90b4168b8c0fd1e WatchSource:0}: Error finding container b247b27313d9bc39ad377c915029cab81c243b80e5cfcf0ee90b4168b8c0fd1e: Status 404 returned error can't find the container with id b247b27313d9bc39ad377c915029cab81c243b80e5cfcf0ee90b4168b8c0fd1e
Nov 03 02:02:58.894582 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.894534    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:02:58.894695 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.894666    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:02:58.894737 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:58.894714    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:02:59.832529 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:59.832490    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9" event={"ID":"8bbc58e2-b00c-4bc7-ba27-cabe0d546740","Type":"ContainerStarted","Data":"0329c02ba8a4fddc392fd98bb2c37033e2de7ad8403dd27b07687706870377d2"}
Nov 03 02:02:59.832959 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:59.832548    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9" event={"ID":"8bbc58e2-b00c-4bc7-ba27-cabe0d546740","Type":"ContainerStarted","Data":"b247b27313d9bc39ad377c915029cab81c243b80e5cfcf0ee90b4168b8c0fd1e"}
Nov 03 02:02:59.994607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:02:59.994535    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9" podStartSLOduration=1.994515053 podStartE2EDuration="1.994515053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:02:59.991643353 +0000 UTC m=+2659.417249830" watchObservedRunningTime="2026-11-03 02:02:59.994515053 +0000 UTC m=+2659.420121518"
Nov 03 02:03:00.460372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:00.458664    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:03:00.465391 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:00.463452    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 02:03:00.465391 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:00.463508    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9" podUID="8bbc58e2-b00c-4bc7-ba27-cabe0d546740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 02:03:01.469373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.465444    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 02:03:01.469373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.465505    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9" podUID="8bbc58e2-b00c-4bc7-ba27-cabe0d546740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 02:03:01.605372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.588804    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"]
Nov 03 02:03:01.605372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.597561    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"
Nov 03 02:03:01.633830 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.633797    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"]
Nov 03 02:03:01.670827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.670775    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:03:01.672073 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.672038    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:03:01.672538 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.672519    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:03:01.672781 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.672766    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:03:01.683371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.681925    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-438d237c-2cv8r\" (UID: \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"
Nov 03 02:03:01.683371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.682060    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-438d237c-2cv8r\" (UID: \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"
Nov 03 02:03:01.683371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.682186    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qv6s\" (UniqueName: \"kubernetes.io/projected/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-kube-api-access-7qv6s\") pod \"maas-api-revoke-sub-438d237c-2cv8r\" (UID: \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"
Nov 03 02:03:01.790440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.786959    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-438d237c-2cv8r\" (UID: \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"
Nov 03 02:03:01.790440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.787107    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qv6s\" (UniqueName: \"kubernetes.io/projected/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-kube-api-access-7qv6s\") pod \"maas-api-revoke-sub-438d237c-2cv8r\" (UID: \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"
Nov 03 02:03:01.790440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.787148    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-438d237c-2cv8r\" (UID: \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"
Nov 03 02:03:01.790440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.788419    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-438d237c-2cv8r\" (UID: \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"
Nov 03 02:03:01.805895 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.805861    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-438d237c-2cv8r\" (UID: \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"
Nov 03 02:03:01.865378 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.864548    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qv6s\" (UniqueName: \"kubernetes.io/projected/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-kube-api-access-7qv6s\") pod \"maas-api-revoke-sub-438d237c-2cv8r\" (UID: \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"
Nov 03 02:03:01.940969 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:01.940928    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"
Nov 03 02:03:02.405538 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:02.404989    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"]
Nov 03 02:03:02.463368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:02.459720    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 02:03:02.463368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:02.459780    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9" podUID="8bbc58e2-b00c-4bc7-ba27-cabe0d546740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 02:03:02.937677 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:02.937071    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r" event={"ID":"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c","Type":"ContainerStarted","Data":"94329a1ff7fcf48c33c54ee7303ffd9ad95d9fc04a11e4d4e2836be692a8356d"}
Nov 03 02:03:03.375813 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.375720    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"]
Nov 03 02:03:03.387181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.387155    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:03.429459 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.429429    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"]
Nov 03 02:03:03.440582 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.440556    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-metrics-certs\") pod \"maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:03.440729 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.440672    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:03.440729 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.440712    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:03.440842 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.440729    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89nfs\" (UniqueName: \"kubernetes.io/projected/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-kube-api-access-89nfs\") pod \"maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:03.457371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.456654    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 02:03:03.457371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.456719    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9" podUID="8bbc58e2-b00c-4bc7-ba27-cabe0d546740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 02:03:03.481692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.481658    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b22fcc-serving-cert\""
Nov 03 02:03:03.485401 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.482396    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 02:03:03.485401 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.482503    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sxqdt\""
Nov 03 02:03:03.541479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.541445    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:03.541692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.541516    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:03.541692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.541545    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nfs\" (UniqueName: \"kubernetes.io/projected/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-kube-api-access-89nfs\") pod \"maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:03.541804 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:03:03.541703    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-serving-cert: secret "maas-api-e2e-praxis-a-b22fcc-serving-cert" not found
Nov 03 02:03:03.541804 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:03:03.541760    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-maas-api-tls podName:b0bf6da9-6b35-4113-a0b8-bf0dd61f498e nodeName:}" failed. No retries permitted until 2026-11-03 02:03:04.041744555 +0000 UTC m=+2663.467351016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-maas-api-tls") pod "maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm" (UID: "b0bf6da9-6b35-4113-a0b8-bf0dd61f498e") : secret "maas-api-e2e-praxis-a-b22fcc-serving-cert" not found
Nov 03 02:03:03.541804 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.541707    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-metrics-certs\") pod \"maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:03.542101 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.542078    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:03.544126 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.544103    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-metrics-certs\") pod \"maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:03.599088 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.599055    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89nfs\" (UniqueName: \"kubernetes.io/projected/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-kube-api-access-89nfs\") pod \"maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:03.946646 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.946607    5508 generic.go:356] "Generic (PLEG): container finished" podID="7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c" containerID="202071504b638f4596fb7fc0456c6d295881c9f8524731a70521ddae584abe0e" exitCode=0
Nov 03 02:03:03.947125 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:03.946694    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r" event={"ID":"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c","Type":"ContainerDied","Data":"202071504b638f4596fb7fc0456c6d295881c9f8524731a70521ddae584abe0e"}
Nov 03 02:03:04.045796 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:04.045761    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:04.048143 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:04.048116    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:04.301664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:04.301578    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:04.459372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:04.455799    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 02:03:04.459372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:04.455867    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9" podUID="8bbc58e2-b00c-4bc7-ba27-cabe0d546740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 02:03:04.762862 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:04.762833    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"]
Nov 03 02:03:04.772398 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:04.770459    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bf6da9_6b35_4113_a0b8_bf0dd61f498e.slice/crio-5c3264b7cc7b79580e09aacae35f8b4ddfbe2c88b687153fab71fc229372cf28 WatchSource:0}: Error finding container 5c3264b7cc7b79580e09aacae35f8b4ddfbe2c88b687153fab71fc229372cf28: Status 404 returned error can't find the container with id 5c3264b7cc7b79580e09aacae35f8b4ddfbe2c88b687153fab71fc229372cf28
Nov 03 02:03:04.970153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:04.966727    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm" event={"ID":"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e","Type":"ContainerStarted","Data":"5c3264b7cc7b79580e09aacae35f8b4ddfbe2c88b687153fab71fc229372cf28"}
Nov 03 02:03:05.113357 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.113327    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"
Nov 03 02:03:05.171494 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.171468    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-xks-ca\") pod \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\" (UID: \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\") "
Nov 03 02:03:05.171656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.171567    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-kube-api-access-7qv6s\" (UniqueName: \"kubernetes.io/projected/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-kube-api-access-7qv6s\") pod \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\" (UID: \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\") "
Nov 03 02:03:05.171656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.171633    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-openshift-ca\") pod \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\" (UID: \"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c\") "
Nov 03 02:03:05.172053 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.172026    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-openshift-ca" pod "7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c" (UID: "7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:03:05.173800 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.173772    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-kube-api-access-7qv6s" pod "7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c" (UID: "7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c"). InnerVolumeSpecName "kube-api-access-7qv6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:03:05.174094 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.174074    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-xks-ca" pod "7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c" (UID: "7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:03:05.273310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.273214    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:05.273310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.273271    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qv6s\" (UniqueName: \"kubernetes.io/projected/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-kube-api-access-7qv6s\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:05.273310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.273287    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:05.453216 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.453173    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Nov 03 02:03:05.453405 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.453262    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9" podUID="8bbc58e2-b00c-4bc7-ba27-cabe0d546740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Nov 03 02:03:05.979280 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.979257    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"
Nov 03 02:03:05.980333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.979286    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r" event={"ID":"7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c","Type":"ContainerDied","Data":"94329a1ff7fcf48c33c54ee7303ffd9ad95d9fc04a11e4d4e2836be692a8356d"}
Nov 03 02:03:05.980442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:05.980386    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94329a1ff7fcf48c33c54ee7303ffd9ad95d9fc04a11e4d4e2836be692a8356d"
Nov 03 02:03:06.993183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:06.993108    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm" event={"ID":"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e","Type":"ContainerStarted","Data":"5ed18d8727b9ecf02a3bdc6a2326c7e1c648543e697717d4c7cdc0739178db86"}
Nov 03 02:03:06.994183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:06.994158    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:07.116246 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:07.116088    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm" podStartSLOduration=2.967967261 podStartE2EDuration="4.116067712s" totalImagesPullingTime="1.148100451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:07.108774365 +0000 UTC m=+2666.534380829" watchObservedRunningTime="2026-11-03 02:03:07.116067712 +0000 UTC m=+2666.541674177"
Nov 03 02:03:07.494553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:07.494505    5508 patch_prober.go:29] interesting pod/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:03:07.494770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:07.494589    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9" podUID="8bbc58e2-b00c-4bc7-ba27-cabe0d546740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:03:07.666571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:07.666527    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:03:07.667537 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:07.667512    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:03:07.670208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:07.668502    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:03:08.230498 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.230468    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"]
Nov 03 02:03:08.230970 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.230954    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c" containerName="revoke-keys"
Nov 03 02:03:08.231026 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.230971    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c" containerName="revoke-keys"
Nov 03 02:03:08.231108 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.231095    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c" containerName="revoke-keys"
Nov 03 02:03:08.235787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.235768    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"
Nov 03 02:03:08.284792 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.284770    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:03:08.285049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.285029    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:03:08.294427 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.294383    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"]
Nov 03 02:03:08.312423 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.312397    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scl2z\" (UniqueName: \"kubernetes.io/projected/3d016891-f4ed-4759-9c09-e0e139b60aaa-kube-api-access-scl2z\") pod \"maas-api-revoke-sub-84dc44ff-fgw7l\" (UID: \"3d016891-f4ed-4759-9c09-e0e139b60aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"
Nov 03 02:03:08.312547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.312439    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84dc44ff-fgw7l\" (UID: \"3d016891-f4ed-4759-9c09-e0e139b60aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"
Nov 03 02:03:08.312547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.312459    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84dc44ff-fgw7l\" (UID: \"3d016891-f4ed-4759-9c09-e0e139b60aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"
Nov 03 02:03:08.413654 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.413622    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scl2z\" (UniqueName: \"kubernetes.io/projected/3d016891-f4ed-4759-9c09-e0e139b60aaa-kube-api-access-scl2z\") pod \"maas-api-revoke-sub-84dc44ff-fgw7l\" (UID: \"3d016891-f4ed-4759-9c09-e0e139b60aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"
Nov 03 02:03:08.413852 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.413667    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84dc44ff-fgw7l\" (UID: \"3d016891-f4ed-4759-9c09-e0e139b60aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"
Nov 03 02:03:08.413852 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.413687    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84dc44ff-fgw7l\" (UID: \"3d016891-f4ed-4759-9c09-e0e139b60aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"
Nov 03 02:03:08.414262 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.414233    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84dc44ff-fgw7l\" (UID: \"3d016891-f4ed-4759-9c09-e0e139b60aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"
Nov 03 02:03:08.415985 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.415967    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84dc44ff-fgw7l\" (UID: \"3d016891-f4ed-4759-9c09-e0e139b60aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"
Nov 03 02:03:08.473054 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.473018    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scl2z\" (UniqueName: \"kubernetes.io/projected/3d016891-f4ed-4759-9c09-e0e139b60aaa-kube-api-access-scl2z\") pod \"maas-api-revoke-sub-84dc44ff-fgw7l\" (UID: \"3d016891-f4ed-4759-9c09-e0e139b60aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"
Nov 03 02:03:08.548670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.548607    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"
Nov 03 02:03:08.964039 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:08.964002    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"]
Nov 03 02:03:08.970808 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:08.970767    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d016891_f4ed_4759_9c09_e0e139b60aaa.slice/crio-ff3f56f4cbff9951f2ca221652f772c169c70d476a7c0ff2e6059ee526f11669 WatchSource:0}: Error finding container ff3f56f4cbff9951f2ca221652f772c169c70d476a7c0ff2e6059ee526f11669: Status 404 returned error can't find the container with id ff3f56f4cbff9951f2ca221652f772c169c70d476a7c0ff2e6059ee526f11669
Nov 03 02:03:09.007542 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:09.007516    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l" event={"ID":"3d016891-f4ed-4759-9c09-e0e139b60aaa","Type":"ContainerStarted","Data":"ff3f56f4cbff9951f2ca221652f772c169c70d476a7c0ff2e6059ee526f11669"}
Nov 03 02:03:10.015654 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:10.015621    5508 generic.go:356] "Generic (PLEG): container finished" podID="3d016891-f4ed-4759-9c09-e0e139b60aaa" containerID="97f2529a691201fe7fee8ab5dbf7b2598691c3fae3d0e853ecde16e122aa9d49" exitCode=0
Nov 03 02:03:10.016037 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:10.015675    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l" event={"ID":"3d016891-f4ed-4759-9c09-e0e139b60aaa","Type":"ContainerDied","Data":"97f2529a691201fe7fee8ab5dbf7b2598691c3fae3d0e853ecde16e122aa9d49"}
Nov 03 02:03:11.175105 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:11.175072    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"
Nov 03 02:03:11.244469 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:11.244317    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-xks-ca\") pod \"3d016891-f4ed-4759-9c09-e0e139b60aaa\" (UID: \"3d016891-f4ed-4759-9c09-e0e139b60aaa\") "
Nov 03 02:03:11.244469 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:11.244454    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-openshift-ca\") pod \"3d016891-f4ed-4759-9c09-e0e139b60aaa\" (UID: \"3d016891-f4ed-4759-9c09-e0e139b60aaa\") "
Nov 03 02:03:11.244922 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:11.244521    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d016891-f4ed-4759-9c09-e0e139b60aaa-kube-api-access-scl2z\" (UniqueName: \"kubernetes.io/projected/3d016891-f4ed-4759-9c09-e0e139b60aaa-kube-api-access-scl2z\") pod \"3d016891-f4ed-4759-9c09-e0e139b60aaa\" (UID: \"3d016891-f4ed-4759-9c09-e0e139b60aaa\") "
Nov 03 02:03:11.244922 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:11.244835    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-openshift-ca" pod "3d016891-f4ed-4759-9c09-e0e139b60aaa" (UID: "3d016891-f4ed-4759-9c09-e0e139b60aaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:03:11.245046 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:11.244983    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:11.246770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:11.246747    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d016891-f4ed-4759-9c09-e0e139b60aaa-kube-api-access-scl2z" pod "3d016891-f4ed-4759-9c09-e0e139b60aaa" (UID: "3d016891-f4ed-4759-9c09-e0e139b60aaa"). InnerVolumeSpecName "kube-api-access-scl2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:03:11.246770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:11.246754    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-xks-ca" pod "3d016891-f4ed-4759-9c09-e0e139b60aaa" (UID: "3d016891-f4ed-4759-9c09-e0e139b60aaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:03:11.346086 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:11.346007    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scl2z\" (UniqueName: \"kubernetes.io/projected/3d016891-f4ed-4759-9c09-e0e139b60aaa-kube-api-access-scl2z\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:11.346086 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:11.346037    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d016891-f4ed-4759-9c09-e0e139b60aaa-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:12.028513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:12.028483    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"
Nov 03 02:03:12.028672 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:12.028515    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l" event={"ID":"3d016891-f4ed-4759-9c09-e0e139b60aaa","Type":"ContainerDied","Data":"ff3f56f4cbff9951f2ca221652f772c169c70d476a7c0ff2e6059ee526f11669"}
Nov 03 02:03:12.028672 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:12.028561    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff3f56f4cbff9951f2ca221652f772c169c70d476a7c0ff2e6059ee526f11669"
Nov 03 02:03:13.006725 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:13.006694    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:03:13.811019 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:03:13.810984    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:03:16.859962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:16.859930    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"]
Nov 03 02:03:16.860444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:16.860395    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3d016891-f4ed-4759-9c09-e0e139b60aaa" containerName="revoke-keys"
Nov 03 02:03:16.860444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:16.860406    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d016891-f4ed-4759-9c09-e0e139b60aaa" containerName="revoke-keys"
Nov 03 02:03:16.860525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:16.860473    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d016891-f4ed-4759-9c09-e0e139b60aaa" containerName="revoke-keys"
Nov 03 02:03:16.863817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:16.863799    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"
Nov 03 02:03:16.912250 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:16.912221    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ce07762-4nnp6\" (UID: \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"
Nov 03 02:03:16.912416 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:16.912291    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf5w6\" (UniqueName: \"kubernetes.io/projected/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-kube-api-access-tf5w6\") pod \"maas-api-revoke-sub-3ce07762-4nnp6\" (UID: \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"
Nov 03 02:03:16.912498 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:16.912427    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ce07762-4nnp6\" (UID: \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"
Nov 03 02:03:16.912895 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:16.912879    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:03:16.958686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:16.958660    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:03:16.962304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:16.962283    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"]
Nov 03 02:03:17.013764 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.013738    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf5w6\" (UniqueName: \"kubernetes.io/projected/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-kube-api-access-tf5w6\") pod \"maas-api-revoke-sub-3ce07762-4nnp6\" (UID: \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"
Nov 03 02:03:17.013921 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.013782    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ce07762-4nnp6\" (UID: \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"
Nov 03 02:03:17.013921 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.013862    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ce07762-4nnp6\" (UID: \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"
Nov 03 02:03:17.014558 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.014536    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ce07762-4nnp6\" (UID: \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"
Nov 03 02:03:17.016120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.016098    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ce07762-4nnp6\" (UID: \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"
Nov 03 02:03:17.071025 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.070999    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf5w6\" (UniqueName: \"kubernetes.io/projected/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-kube-api-access-tf5w6\") pod \"maas-api-revoke-sub-3ce07762-4nnp6\" (UID: \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"
Nov 03 02:03:17.176728 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.176687    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"
Nov 03 02:03:17.473805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.473713    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"]
Nov 03 02:03:17.478799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.478776    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"
Nov 03 02:03:17.537436 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.537400    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"]
Nov 03 02:03:17.592569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.592433    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"]
Nov 03 02:03:17.598860 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:17.598825    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f50a3ae_3ee5_4c4f_8980_c030d3ec006e.slice/crio-19dcd1262e26151eb9f765f8879357e91da14851f3b826a4bd163bf590623dd1 WatchSource:0}: Error finding container 19dcd1262e26151eb9f765f8879357e91da14851f3b826a4bd163bf590623dd1: Status 404 returned error can't find the container with id 19dcd1262e26151eb9f765f8879357e91da14851f3b826a4bd163bf590623dd1
Nov 03 02:03:17.620037 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.620008    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsmbk\" (UniqueName: \"kubernetes.io/projected/58b20a14-5b1c-408d-9298-7ca5cefa428a-kube-api-access-zsmbk\") pod \"maas-api-revoke-sub-b8e0f312-v8r9q\" (UID: \"58b20a14-5b1c-408d-9298-7ca5cefa428a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"
Nov 03 02:03:17.620200 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.620172    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8e0f312-v8r9q\" (UID: \"58b20a14-5b1c-408d-9298-7ca5cefa428a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"
Nov 03 02:03:17.620272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.620216    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8e0f312-v8r9q\" (UID: \"58b20a14-5b1c-408d-9298-7ca5cefa428a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"
Nov 03 02:03:17.721897 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.721863    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsmbk\" (UniqueName: \"kubernetes.io/projected/58b20a14-5b1c-408d-9298-7ca5cefa428a-kube-api-access-zsmbk\") pod \"maas-api-revoke-sub-b8e0f312-v8r9q\" (UID: \"58b20a14-5b1c-408d-9298-7ca5cefa428a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"
Nov 03 02:03:17.722081 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.721961    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8e0f312-v8r9q\" (UID: \"58b20a14-5b1c-408d-9298-7ca5cefa428a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"
Nov 03 02:03:17.722081 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.721988    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8e0f312-v8r9q\" (UID: \"58b20a14-5b1c-408d-9298-7ca5cefa428a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"
Nov 03 02:03:17.722686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.722659    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8e0f312-v8r9q\" (UID: \"58b20a14-5b1c-408d-9298-7ca5cefa428a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"
Nov 03 02:03:17.724396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.724324    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8e0f312-v8r9q\" (UID: \"58b20a14-5b1c-408d-9298-7ca5cefa428a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"
Nov 03 02:03:17.779430 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.779403    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsmbk\" (UniqueName: \"kubernetes.io/projected/58b20a14-5b1c-408d-9298-7ca5cefa428a-kube-api-access-zsmbk\") pod \"maas-api-revoke-sub-b8e0f312-v8r9q\" (UID: \"58b20a14-5b1c-408d-9298-7ca5cefa428a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"
Nov 03 02:03:17.794357 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:17.794327    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"
Nov 03 02:03:18.069565 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:18.069472    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6" event={"ID":"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e","Type":"ContainerStarted","Data":"f6c7d72055073ac6a3ea647c036b7bf2d6f2b62c98c3a63e6c42ed587c83c876"}
Nov 03 02:03:18.069565 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:18.069532    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6" event={"ID":"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e","Type":"ContainerStarted","Data":"19dcd1262e26151eb9f765f8879357e91da14851f3b826a4bd163bf590623dd1"}
Nov 03 02:03:18.171656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:18.171613    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"]
Nov 03 02:03:18.173465 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:18.173422    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b20a14_5b1c_408d_9298_7ca5cefa428a.slice/crio-2694c4e7d84848aebc82623687c9b2b4a39a9d54e1a5fc66e0f3727af92519d6 WatchSource:0}: Error finding container 2694c4e7d84848aebc82623687c9b2b4a39a9d54e1a5fc66e0f3727af92519d6: Status 404 returned error can't find the container with id 2694c4e7d84848aebc82623687c9b2b4a39a9d54e1a5fc66e0f3727af92519d6
Nov 03 02:03:18.179180 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:18.179136    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6" podStartSLOduration=2.179121787 podStartE2EDuration="2.179121787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:18.177517729 +0000 UTC m=+2677.603124195" watchObservedRunningTime="2026-11-03 02:03:18.179121787 +0000 UTC m=+2677.604728257"
Nov 03 02:03:19.077808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:19.077773    5508 generic.go:356] "Generic (PLEG): container finished" podID="3f50a3ae-3ee5-4c4f-8980-c030d3ec006e" containerID="f6c7d72055073ac6a3ea647c036b7bf2d6f2b62c98c3a63e6c42ed587c83c876" exitCode=0
Nov 03 02:03:19.078281 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:19.077861    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6" event={"ID":"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e","Type":"ContainerDied","Data":"f6c7d72055073ac6a3ea647c036b7bf2d6f2b62c98c3a63e6c42ed587c83c876"}
Nov 03 02:03:19.079714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:19.079691    5508 generic.go:356] "Generic (PLEG): container finished" podID="58b20a14-5b1c-408d-9298-7ca5cefa428a" containerID="e1297b3c8ea00f0443f21a46a1e6646029871791e228cc117080df2dadbca69c" exitCode=0
Nov 03 02:03:19.079833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:19.079720    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q" event={"ID":"58b20a14-5b1c-408d-9298-7ca5cefa428a","Type":"ContainerDied","Data":"e1297b3c8ea00f0443f21a46a1e6646029871791e228cc117080df2dadbca69c"}
Nov 03 02:03:19.079833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:19.079762    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q" event={"ID":"58b20a14-5b1c-408d-9298-7ca5cefa428a","Type":"ContainerStarted","Data":"2694c4e7d84848aebc82623687c9b2b4a39a9d54e1a5fc66e0f3727af92519d6"}
Nov 03 02:03:20.252175 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.252151    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"
Nov 03 02:03:20.255184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.255159    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"
Nov 03 02:03:20.350667 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.350563    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-openshift-ca\") pod \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\" (UID: \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\") "
Nov 03 02:03:20.350667 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.350650    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b20a14-5b1c-408d-9298-7ca5cefa428a-kube-api-access-zsmbk\" (UniqueName: \"kubernetes.io/projected/58b20a14-5b1c-408d-9298-7ca5cefa428a-kube-api-access-zsmbk\") pod \"58b20a14-5b1c-408d-9298-7ca5cefa428a\" (UID: \"58b20a14-5b1c-408d-9298-7ca5cefa428a\") "
Nov 03 02:03:20.350926 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.350808    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-xks-ca\") pod \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\" (UID: \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\") "
Nov 03 02:03:20.350926 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.350869    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-xks-ca\") pod \"58b20a14-5b1c-408d-9298-7ca5cefa428a\" (UID: \"58b20a14-5b1c-408d-9298-7ca5cefa428a\") "
Nov 03 02:03:20.350926 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.350917    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-openshift-ca\") pod \"58b20a14-5b1c-408d-9298-7ca5cefa428a\" (UID: \"58b20a14-5b1c-408d-9298-7ca5cefa428a\") "
Nov 03 02:03:20.351078 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.350950    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-openshift-ca" pod "3f50a3ae-3ee5-4c4f-8980-c030d3ec006e" (UID: "3f50a3ae-3ee5-4c4f-8980-c030d3ec006e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:03:20.351078 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.351031    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-kube-api-access-tf5w6\" (UniqueName: \"kubernetes.io/projected/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-kube-api-access-tf5w6\") pod \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\" (UID: \"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e\") "
Nov 03 02:03:20.351416 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.351386    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-openshift-ca" pod "58b20a14-5b1c-408d-9298-7ca5cefa428a" (UID: "58b20a14-5b1c-408d-9298-7ca5cefa428a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:03:20.351554 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.351534    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:20.351618 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.351558    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:20.353039 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.352992    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-xks-ca" pod "3f50a3ae-3ee5-4c4f-8980-c030d3ec006e" (UID: "3f50a3ae-3ee5-4c4f-8980-c030d3ec006e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:03:20.353163 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.353073    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-kube-api-access-tf5w6" pod "3f50a3ae-3ee5-4c4f-8980-c030d3ec006e" (UID: "3f50a3ae-3ee5-4c4f-8980-c030d3ec006e"). InnerVolumeSpecName "kube-api-access-tf5w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:03:20.353297 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.353281    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b20a14-5b1c-408d-9298-7ca5cefa428a-kube-api-access-zsmbk" pod "58b20a14-5b1c-408d-9298-7ca5cefa428a" (UID: "58b20a14-5b1c-408d-9298-7ca5cefa428a"). InnerVolumeSpecName "kube-api-access-zsmbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:03:20.353508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.353492    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-xks-ca" pod "58b20a14-5b1c-408d-9298-7ca5cefa428a" (UID: "58b20a14-5b1c-408d-9298-7ca5cefa428a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:03:20.452881 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.452830    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsmbk\" (UniqueName: \"kubernetes.io/projected/58b20a14-5b1c-408d-9298-7ca5cefa428a-kube-api-access-zsmbk\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:20.452881 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.452879    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:20.452881 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.452895    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58b20a14-5b1c-408d-9298-7ca5cefa428a-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:20.453221 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:20.452909    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf5w6\" (UniqueName: \"kubernetes.io/projected/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e-kube-api-access-tf5w6\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:21.095156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:21.094936    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"
Nov 03 02:03:21.095156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:21.094953    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q" event={"ID":"58b20a14-5b1c-408d-9298-7ca5cefa428a","Type":"ContainerDied","Data":"2694c4e7d84848aebc82623687c9b2b4a39a9d54e1a5fc66e0f3727af92519d6"}
Nov 03 02:03:21.095156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:21.095006    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2694c4e7d84848aebc82623687c9b2b4a39a9d54e1a5fc66e0f3727af92519d6"
Nov 03 02:03:21.096534 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:21.096517    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"
Nov 03 02:03:21.096652 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:21.096566    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6" event={"ID":"3f50a3ae-3ee5-4c4f-8980-c030d3ec006e","Type":"ContainerDied","Data":"19dcd1262e26151eb9f765f8879357e91da14851f3b826a4bd163bf590623dd1"}
Nov 03 02:03:21.096652 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:21.096605    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19dcd1262e26151eb9f765f8879357e91da14851f3b826a4bd163bf590623dd1"
Nov 03 02:03:22.906758 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.906728    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"]
Nov 03 02:03:22.907168 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.907156    5508 cpu_manager.go:414] "containerMap: removing container" podUID="58b20a14-5b1c-408d-9298-7ca5cefa428a" containerName="revoke-keys"
Nov 03 02:03:22.907208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.907169    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b20a14-5b1c-408d-9298-7ca5cefa428a" containerName="revoke-keys"
Nov 03 02:03:22.907208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.907200    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3f50a3ae-3ee5-4c4f-8980-c030d3ec006e" containerName="revoke-keys"
Nov 03 02:03:22.907208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.907206    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f50a3ae-3ee5-4c4f-8980-c030d3ec006e" containerName="revoke-keys"
Nov 03 02:03:22.907311 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.907277    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f50a3ae-3ee5-4c4f-8980-c030d3ec006e" containerName="revoke-keys"
Nov 03 02:03:22.907311 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.907289    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b20a14-5b1c-408d-9298-7ca5cefa428a" containerName="revoke-keys"
Nov 03 02:03:22.910632 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.910614    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:22.986840 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.986802    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"]
Nov 03 02:03:22.989010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.988982    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ss8f\" (UniqueName: \"kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-kube-api-access-8ss8f\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:22.989169 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.989018    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:22.989169 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.989039    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:22.989285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.989228    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:22.989333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.989295    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:22.989429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.989399    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da498724-5787-4ae8-82ed-0e3355fb496e-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:22.989499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.989443    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da498724-5787-4ae8-82ed-0e3355fb496e-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:22.989499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.989481    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:22.989616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:22.989599    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.090682 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.090640    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.090892 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.090717    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ss8f\" (UniqueName: \"kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-kube-api-access-8ss8f\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.090892 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.090749    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.090892 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.090782    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.090892 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.090858    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.091128 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.090893    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.091128 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.090918    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da498724-5787-4ae8-82ed-0e3355fb496e-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.091128 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.090942    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da498724-5787-4ae8-82ed-0e3355fb496e-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.091128 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.090976    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.091373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.091133    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.091443 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.091415    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.091499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.091448    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.091931 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.091903    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.092085 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.092062    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da498724-5787-4ae8-82ed-0e3355fb496e-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.094478 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.094441    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.094634 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.094532    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da498724-5787-4ae8-82ed-0e3355fb496e-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.148360 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.148313    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.148501 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.148409    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ss8f\" (UniqueName: \"kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-kube-api-access-8ss8f\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.223961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.223880    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:23.660681 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.660649    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"]
Nov 03 02:03:23.667168 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:23.667138    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda498724_5787_4ae8_82ed_0e3355fb496e.slice/crio-d152736f520e17e2e78ffafaedaf87ed7b0ce09fc02be82213e7fe545ea9b97c WatchSource:0}: Error finding container d152736f520e17e2e78ffafaedaf87ed7b0ce09fc02be82213e7fe545ea9b97c: Status 404 returned error can't find the container with id d152736f520e17e2e78ffafaedaf87ed7b0ce09fc02be82213e7fe545ea9b97c
Nov 03 02:03:23.669797 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.669768    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:23.669883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.669836    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:23.669883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.669873    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:23.716396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.716374    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"]
Nov 03 02:03:23.720982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.720966    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"
Nov 03 02:03:23.769686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.769640    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:03:23.769686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.769660    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:03:23.779472 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.778911    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"]
Nov 03 02:03:23.800110 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.800069    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e013773-348f-48d7-806b-eef735149093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41128b9f-jpsfd\" (UID: \"4e013773-348f-48d7-806b-eef735149093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"
Nov 03 02:03:23.800293 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.800186    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e013773-348f-48d7-806b-eef735149093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41128b9f-jpsfd\" (UID: \"4e013773-348f-48d7-806b-eef735149093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"
Nov 03 02:03:23.800293 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.800229    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vprl\" (UniqueName: \"kubernetes.io/projected/4e013773-348f-48d7-806b-eef735149093-kube-api-access-7vprl\") pod \"maas-api-revoke-sub-41128b9f-jpsfd\" (UID: \"4e013773-348f-48d7-806b-eef735149093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"
Nov 03 02:03:23.901195 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.901152    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e013773-348f-48d7-806b-eef735149093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41128b9f-jpsfd\" (UID: \"4e013773-348f-48d7-806b-eef735149093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"
Nov 03 02:03:23.901493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.901218    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e013773-348f-48d7-806b-eef735149093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41128b9f-jpsfd\" (UID: \"4e013773-348f-48d7-806b-eef735149093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"
Nov 03 02:03:23.901493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.901261    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vprl\" (UniqueName: \"kubernetes.io/projected/4e013773-348f-48d7-806b-eef735149093-kube-api-access-7vprl\") pod \"maas-api-revoke-sub-41128b9f-jpsfd\" (UID: \"4e013773-348f-48d7-806b-eef735149093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"
Nov 03 02:03:23.902036 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.902010    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e013773-348f-48d7-806b-eef735149093-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41128b9f-jpsfd\" (UID: \"4e013773-348f-48d7-806b-eef735149093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"
Nov 03 02:03:23.903507 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.903485    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e013773-348f-48d7-806b-eef735149093-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41128b9f-jpsfd\" (UID: \"4e013773-348f-48d7-806b-eef735149093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"
Nov 03 02:03:23.958621 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:23.958547    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vprl\" (UniqueName: \"kubernetes.io/projected/4e013773-348f-48d7-806b-eef735149093-kube-api-access-7vprl\") pod \"maas-api-revoke-sub-41128b9f-jpsfd\" (UID: \"4e013773-348f-48d7-806b-eef735149093\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"
Nov 03 02:03:24.033366 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:24.033323    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"
Nov 03 02:03:24.124019 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:24.123982    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh" event={"ID":"da498724-5787-4ae8-82ed-0e3355fb496e","Type":"ContainerStarted","Data":"810759daef347d85f4b6c31860e46a758e989e472539e20c14341d963680dba6"}
Nov 03 02:03:24.124184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:24.124042    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh" event={"ID":"da498724-5787-4ae8-82ed-0e3355fb496e","Type":"ContainerStarted","Data":"d152736f520e17e2e78ffafaedaf87ed7b0ce09fc02be82213e7fe545ea9b97c"}
Nov 03 02:03:24.224839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:24.224754    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:24.226072 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:24.226046    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 02:03:24.226154 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:24.226095    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh" podUID="da498724-5787-4ae8-82ed-0e3355fb496e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 02:03:24.239526 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:24.239478    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh" podStartSLOduration=2.23946619 podStartE2EDuration="2.23946619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:24.2369287 +0000 UTC m=+2683.662535163" watchObservedRunningTime="2026-11-03 02:03:24.23946619 +0000 UTC m=+2683.665072654"
Nov 03 02:03:24.420231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:24.420196    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"]
Nov 03 02:03:24.426217 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:24.426189    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e013773_348f_48d7_806b_eef735149093.slice/crio-27ed0c7b2973bba8db28c276ffc400ecd6a018db0c3933e52740f2e209ac137b WatchSource:0}: Error finding container 27ed0c7b2973bba8db28c276ffc400ecd6a018db0c3933e52740f2e209ac137b: Status 404 returned error can't find the container with id 27ed0c7b2973bba8db28c276ffc400ecd6a018db0c3933e52740f2e209ac137b
Nov 03 02:03:25.132186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:25.132081    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd" event={"ID":"4e013773-348f-48d7-806b-eef735149093","Type":"ContainerStarted","Data":"d66ccdd25da4932862be187a725e5d5f1aaec870e4a24f68008b4504121c8603"}
Nov 03 02:03:25.132186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:25.132156    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd" event={"ID":"4e013773-348f-48d7-806b-eef735149093","Type":"ContainerStarted","Data":"27ed0c7b2973bba8db28c276ffc400ecd6a018db0c3933e52740f2e209ac137b"}
Nov 03 02:03:25.225178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:25.225145    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 02:03:25.225393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:25.225229    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh" podUID="da498724-5787-4ae8-82ed-0e3355fb496e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 02:03:25.240567 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:25.240524    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd" podStartSLOduration=2.240510469 podStartE2EDuration="2.240510469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:25.239434334 +0000 UTC m=+2684.665040816" watchObservedRunningTime="2026-11-03 02:03:25.240510469 +0000 UTC m=+2684.666116972"
Nov 03 02:03:26.140514 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:26.140482    5508 generic.go:356] "Generic (PLEG): container finished" podID="4e013773-348f-48d7-806b-eef735149093" containerID="d66ccdd25da4932862be187a725e5d5f1aaec870e4a24f68008b4504121c8603" exitCode=0
Nov 03 02:03:26.140976 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:26.140564    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd" event={"ID":"4e013773-348f-48d7-806b-eef735149093","Type":"ContainerDied","Data":"d66ccdd25da4932862be187a725e5d5f1aaec870e4a24f68008b4504121c8603"}
Nov 03 02:03:26.225125 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:26.225092    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 02:03:26.225326 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:26.225143    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh" podUID="da498724-5787-4ae8-82ed-0e3355fb496e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 02:03:26.579030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:26.578980    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:03:27.225184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:27.225022    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 02:03:27.225184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:27.225121    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh" podUID="da498724-5787-4ae8-82ed-0e3355fb496e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 02:03:27.428477 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:27.428454    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"
Nov 03 02:03:27.597372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:27.590749    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e013773-348f-48d7-806b-eef735149093-kube-api-access-7vprl\" (UniqueName: \"kubernetes.io/projected/4e013773-348f-48d7-806b-eef735149093-kube-api-access-7vprl\") pod \"4e013773-348f-48d7-806b-eef735149093\" (UID: \"4e013773-348f-48d7-806b-eef735149093\") "
Nov 03 02:03:27.597372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:27.590933    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e013773-348f-48d7-806b-eef735149093-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e013773-348f-48d7-806b-eef735149093-maas-api-xks-ca\") pod \"4e013773-348f-48d7-806b-eef735149093\" (UID: \"4e013773-348f-48d7-806b-eef735149093\") "
Nov 03 02:03:27.597372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:27.590976    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e013773-348f-48d7-806b-eef735149093-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e013773-348f-48d7-806b-eef735149093-maas-api-openshift-ca\") pod \"4e013773-348f-48d7-806b-eef735149093\" (UID: \"4e013773-348f-48d7-806b-eef735149093\") "
Nov 03 02:03:27.597372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:27.595535    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e013773-348f-48d7-806b-eef735149093-maas-api-openshift-ca" pod "4e013773-348f-48d7-806b-eef735149093" (UID: "4e013773-348f-48d7-806b-eef735149093"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:03:27.622311 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:27.622276    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e013773-348f-48d7-806b-eef735149093-maas-api-xks-ca" pod "4e013773-348f-48d7-806b-eef735149093" (UID: "4e013773-348f-48d7-806b-eef735149093"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:03:27.671626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:27.671524    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e013773-348f-48d7-806b-eef735149093-kube-api-access-7vprl" pod "4e013773-348f-48d7-806b-eef735149093" (UID: "4e013773-348f-48d7-806b-eef735149093"). InnerVolumeSpecName "kube-api-access-7vprl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:03:27.694081 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:27.694036    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e013773-348f-48d7-806b-eef735149093-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:27.694081 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:27.694074    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e013773-348f-48d7-806b-eef735149093-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:27.694081 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:27.694088    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vprl\" (UniqueName: \"kubernetes.io/projected/4e013773-348f-48d7-806b-eef735149093-kube-api-access-7vprl\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:28.174495 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:28.174459    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"
Nov 03 02:03:28.203374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:28.174463    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd" event={"ID":"4e013773-348f-48d7-806b-eef735149093","Type":"ContainerDied","Data":"27ed0c7b2973bba8db28c276ffc400ecd6a018db0c3933e52740f2e209ac137b"}
Nov 03 02:03:28.203374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:28.182557    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27ed0c7b2973bba8db28c276ffc400ecd6a018db0c3933e52740f2e209ac137b"
Nov 03 02:03:28.225124 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:28.225092    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 02:03:28.225314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:28.225155    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh" podUID="da498724-5787-4ae8-82ed-0e3355fb496e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 02:03:29.224791 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:29.224753    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 02:03:29.225064 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:29.225030    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh" podUID="da498724-5787-4ae8-82ed-0e3355fb496e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 02:03:30.224343 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:30.224301    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Nov 03 02:03:30.224757 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:30.224385    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh" podUID="da498724-5787-4ae8-82ed-0e3355fb496e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Nov 03 02:03:30.921014 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:30.920979    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"]
Nov 03 02:03:30.921570 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:30.921544    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4e013773-348f-48d7-806b-eef735149093" containerName="revoke-keys"
Nov 03 02:03:30.921570 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:30.921569    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e013773-348f-48d7-806b-eef735149093" containerName="revoke-keys"
Nov 03 02:03:30.921783 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:30.921731    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e013773-348f-48d7-806b-eef735149093" containerName="revoke-keys"
Nov 03 02:03:30.925217 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:30.925194    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"
Nov 03 02:03:30.976779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:30.976744    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:03:31.024328 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.024282    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:03:31.024911 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.024883    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-710b85d6-cnds6\" (UID: \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"
Nov 03 02:03:31.025055 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.025037    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-710b85d6-cnds6\" (UID: \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"
Nov 03 02:03:31.025128 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.025085    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5wzj\" (UniqueName: \"kubernetes.io/projected/f004de45-ebc3-4678-8311-ab50a7cbaa9e-kube-api-access-g5wzj\") pod \"maas-api-revoke-sub-710b85d6-cnds6\" (UID: \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"
Nov 03 02:03:31.028763 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.028689    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"]
Nov 03 02:03:31.126376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.126317    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-710b85d6-cnds6\" (UID: \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"
Nov 03 02:03:31.126662 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.126645    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-710b85d6-cnds6\" (UID: \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"
Nov 03 02:03:31.127144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.127125    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5wzj\" (UniqueName: \"kubernetes.io/projected/f004de45-ebc3-4678-8311-ab50a7cbaa9e-kube-api-access-g5wzj\") pod \"maas-api-revoke-sub-710b85d6-cnds6\" (UID: \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"
Nov 03 02:03:31.127329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.127213    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-710b85d6-cnds6\" (UID: \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"
Nov 03 02:03:31.132824 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.132760    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-710b85d6-cnds6\" (UID: \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"
Nov 03 02:03:31.194798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.194754    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5wzj\" (UniqueName: \"kubernetes.io/projected/f004de45-ebc3-4678-8311-ab50a7cbaa9e-kube-api-access-g5wzj\") pod \"maas-api-revoke-sub-710b85d6-cnds6\" (UID: \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"
Nov 03 02:03:31.238837 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.238803    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"
Nov 03 02:03:31.706003 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.705964    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"]
Nov 03 02:03:31.712165 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:31.712137    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf004de45_ebc3_4678_8311_ab50a7cbaa9e.slice/crio-32a6ed9540108b6b0c890b3025e511f072258ce03f255ee2ea64ac117e1f3f60 WatchSource:0}: Error finding container 32a6ed9540108b6b0c890b3025e511f072258ce03f255ee2ea64ac117e1f3f60: Status 404 returned error can't find the container with id 32a6ed9540108b6b0c890b3025e511f072258ce03f255ee2ea64ac117e1f3f60
Nov 03 02:03:31.740214 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.740171    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:31.740938 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.740875    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:31.742093 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:31.742067    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:03:32.002410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.001991    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"]
Nov 03 02:03:32.010983 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.010959    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"
Nov 03 02:03:32.032194 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.032169    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"]
Nov 03 02:03:32.041434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.040560    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r9lm\" (UniqueName: \"kubernetes.io/projected/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-kube-api-access-7r9lm\") pod \"maas-api-revoke-sub-09a324ea-mcrdn\" (UID: \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"
Nov 03 02:03:32.041434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.040790    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a324ea-mcrdn\" (UID: \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"
Nov 03 02:03:32.041434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.040871    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a324ea-mcrdn\" (UID: \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"
Nov 03 02:03:32.142484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.142454    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a324ea-mcrdn\" (UID: \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"
Nov 03 02:03:32.142652 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.142508    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a324ea-mcrdn\" (UID: \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"
Nov 03 02:03:32.142652 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.142529    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r9lm\" (UniqueName: \"kubernetes.io/projected/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-kube-api-access-7r9lm\") pod \"maas-api-revoke-sub-09a324ea-mcrdn\" (UID: \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"
Nov 03 02:03:32.143304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.143286    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a324ea-mcrdn\" (UID: \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"
Nov 03 02:03:32.145557 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.145531    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a324ea-mcrdn\" (UID: \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"
Nov 03 02:03:32.201541 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.200980    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r9lm\" (UniqueName: \"kubernetes.io/projected/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-kube-api-access-7r9lm\") pod \"maas-api-revoke-sub-09a324ea-mcrdn\" (UID: \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"
Nov 03 02:03:32.234266 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.234232    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6" event={"ID":"f004de45-ebc3-4678-8311-ab50a7cbaa9e","Type":"ContainerStarted","Data":"ce88cdab1a4e712950e8c5d6182a2cb8b2b12257270889b2d403f1c27baa4f82"}
Nov 03 02:03:32.234472 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.234287    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6" event={"ID":"f004de45-ebc3-4678-8311-ab50a7cbaa9e","Type":"ContainerStarted","Data":"32a6ed9540108b6b0c890b3025e511f072258ce03f255ee2ea64ac117e1f3f60"}
Nov 03 02:03:32.327592 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.327504    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"
Nov 03 02:03:32.353368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.350065    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6" podStartSLOduration=2.350045447 podStartE2EDuration="2.350045447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:32.344797029 +0000 UTC m=+2691.770403542" watchObservedRunningTime="2026-11-03 02:03:32.350045447 +0000 UTC m=+2691.775651910"
Nov 03 02:03:32.777382 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:32.772702    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"]
Nov 03 02:03:32.839483 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:32.837975    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29d7f6d_e5b0_42b7_afe5_06baa89f1a35.slice/crio-b50d068f7c7d3356b7d1b3dc82485aee5ed323c9e54218bf8fa6944b9fcd2ca1 WatchSource:0}: Error finding container b50d068f7c7d3356b7d1b3dc82485aee5ed323c9e54218bf8fa6944b9fcd2ca1: Status 404 returned error can't find the container with id b50d068f7c7d3356b7d1b3dc82485aee5ed323c9e54218bf8fa6944b9fcd2ca1
Nov 03 02:03:33.242459 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:33.242415    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn" event={"ID":"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35","Type":"ContainerStarted","Data":"b50d068f7c7d3356b7d1b3dc82485aee5ed323c9e54218bf8fa6944b9fcd2ca1"}
Nov 03 02:03:33.243658 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:33.243631    5508 generic.go:356] "Generic (PLEG): container finished" podID="f004de45-ebc3-4678-8311-ab50a7cbaa9e" containerID="ce88cdab1a4e712950e8c5d6182a2cb8b2b12257270889b2d403f1c27baa4f82" exitCode=0
Nov 03 02:03:33.244505 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:33.244481    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6" event={"ID":"f004de45-ebc3-4678-8311-ab50a7cbaa9e","Type":"ContainerDied","Data":"ce88cdab1a4e712950e8c5d6182a2cb8b2b12257270889b2d403f1c27baa4f82"}
Nov 03 02:03:34.275008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:34.274965    5508 generic.go:356] "Generic (PLEG): container finished" podID="f29d7f6d-e5b0-42b7-afe5-06baa89f1a35" containerID="91e3c8c251af4e4dacecd981bacf8c379aec7d31ea790b10eac20bdef6efcea0" exitCode=0
Nov 03 02:03:34.276274 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:34.276244    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn" event={"ID":"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35","Type":"ContainerDied","Data":"91e3c8c251af4e4dacecd981bacf8c379aec7d31ea790b10eac20bdef6efcea0"}
Nov 03 02:03:34.458208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:34.452158    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"
Nov 03 02:03:34.610237 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:34.610205    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-openshift-ca\") pod \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\" (UID: \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\") "
Nov 03 02:03:34.610410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:34.610308    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f004de45-ebc3-4678-8311-ab50a7cbaa9e-kube-api-access-g5wzj\" (UniqueName: \"kubernetes.io/projected/f004de45-ebc3-4678-8311-ab50a7cbaa9e-kube-api-access-g5wzj\") pod \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\" (UID: \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\") "
Nov 03 02:03:34.610410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:34.610387    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-xks-ca\") pod \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\" (UID: \"f004de45-ebc3-4678-8311-ab50a7cbaa9e\") "
Nov 03 02:03:34.610636 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:34.610610    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-openshift-ca" pod "f004de45-ebc3-4678-8311-ab50a7cbaa9e" (UID: "f004de45-ebc3-4678-8311-ab50a7cbaa9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:03:34.610810 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:34.610787    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:34.612339 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:34.612315    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f004de45-ebc3-4678-8311-ab50a7cbaa9e-kube-api-access-g5wzj" pod "f004de45-ebc3-4678-8311-ab50a7cbaa9e" (UID: "f004de45-ebc3-4678-8311-ab50a7cbaa9e"). InnerVolumeSpecName "kube-api-access-g5wzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:03:34.612467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:34.612450    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-xks-ca" pod "f004de45-ebc3-4678-8311-ab50a7cbaa9e" (UID: "f004de45-ebc3-4678-8311-ab50a7cbaa9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:03:34.712035 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:34.712004    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5wzj\" (UniqueName: \"kubernetes.io/projected/f004de45-ebc3-4678-8311-ab50a7cbaa9e-kube-api-access-g5wzj\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:34.712035 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:34.712033    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f004de45-ebc3-4678-8311-ab50a7cbaa9e-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:35.285988 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:35.285957    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"
Nov 03 02:03:35.286425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:35.285991    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6" event={"ID":"f004de45-ebc3-4678-8311-ab50a7cbaa9e","Type":"ContainerDied","Data":"32a6ed9540108b6b0c890b3025e511f072258ce03f255ee2ea64ac117e1f3f60"}
Nov 03 02:03:35.286425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:35.286052    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32a6ed9540108b6b0c890b3025e511f072258ce03f255ee2ea64ac117e1f3f60"
Nov 03 02:03:35.449006 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:35.448978    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"
Nov 03 02:03:35.521564 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:35.521531    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-xks-ca\") pod \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\" (UID: \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\") "
Nov 03 02:03:35.521760 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:35.521582    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-kube-api-access-7r9lm\" (UniqueName: \"kubernetes.io/projected/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-kube-api-access-7r9lm\") pod \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\" (UID: \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\") "
Nov 03 02:03:35.521760 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:35.521747    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-openshift-ca\") pod \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\" (UID: \"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35\") "
Nov 03 02:03:35.522161 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:35.522128    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-openshift-ca" pod "f29d7f6d-e5b0-42b7-afe5-06baa89f1a35" (UID: "f29d7f6d-e5b0-42b7-afe5-06baa89f1a35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:03:35.523872 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:35.523849    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-xks-ca" pod "f29d7f6d-e5b0-42b7-afe5-06baa89f1a35" (UID: "f29d7f6d-e5b0-42b7-afe5-06baa89f1a35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:03:35.523987 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:35.523962    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-kube-api-access-7r9lm" pod "f29d7f6d-e5b0-42b7-afe5-06baa89f1a35" (UID: "f29d7f6d-e5b0-42b7-afe5-06baa89f1a35"). InnerVolumeSpecName "kube-api-access-7r9lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:03:35.623528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:35.623442    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:35.623528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:35.623476    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r9lm\" (UniqueName: \"kubernetes.io/projected/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-kube-api-access-7r9lm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:35.623528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:35.623493    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:36.294257 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:36.294227    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"
Nov 03 02:03:36.294760 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:36.294265    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn" event={"ID":"f29d7f6d-e5b0-42b7-afe5-06baa89f1a35","Type":"ContainerDied","Data":"b50d068f7c7d3356b7d1b3dc82485aee5ed323c9e54218bf8fa6944b9fcd2ca1"}
Nov 03 02:03:36.294760 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:36.294313    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b50d068f7c7d3356b7d1b3dc82485aee5ed323c9e54218bf8fa6944b9fcd2ca1"
Nov 03 02:03:38.319197 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:38.319162    5508 generic.go:356] "Generic (PLEG): container finished" podID="65a28645-ef66-46b3-8a68-a999694dadc3" containerID="917f79b61998840d91a024d2078e9bd3bdcf7022fb67ef672712812e61a81939" exitCode=0
Nov 03 02:03:38.319768 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:38.319237    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5" event={"ID":"65a28645-ef66-46b3-8a68-a999694dadc3","Type":"ContainerDied","Data":"917f79b61998840d91a024d2078e9bd3bdcf7022fb67ef672712812e61a81939"}
Nov 03 02:03:38.319768 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:38.319738    5508 scope.go:122] "RemoveContainer" containerID="917f79b61998840d91a024d2078e9bd3bdcf7022fb67ef672712812e61a81939"
Nov 03 02:03:38.321211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:38.321176    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:38.321331 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:38.321267    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:38.321331 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:38.321317    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:39.329565 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.329529    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5" event={"ID":"65a28645-ef66-46b3-8a68-a999694dadc3","Type":"ContainerStarted","Data":"0df9011d5f6dbc1cd4f44b25e5ede5c8df040268d3f28fe8ffc0b62b4fa1b06d"}
Nov 03 02:03:39.538360 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.538310    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"]
Nov 03 02:03:39.538849 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.538829    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f29d7f6d-e5b0-42b7-afe5-06baa89f1a35" containerName="revoke-keys"
Nov 03 02:03:39.538849 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.538848    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29d7f6d-e5b0-42b7-afe5-06baa89f1a35" containerName="revoke-keys"
Nov 03 02:03:39.539011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.538869    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f004de45-ebc3-4678-8311-ab50a7cbaa9e" containerName="revoke-keys"
Nov 03 02:03:39.539011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.538875    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f004de45-ebc3-4678-8311-ab50a7cbaa9e" containerName="revoke-keys"
Nov 03 02:03:39.539011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.538985    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29d7f6d-e5b0-42b7-afe5-06baa89f1a35" containerName="revoke-keys"
Nov 03 02:03:39.539011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.538997    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f004de45-ebc3-4678-8311-ab50a7cbaa9e" containerName="revoke-keys"
Nov 03 02:03:39.542821 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.542797    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"
Nov 03 02:03:39.591596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.591510    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:03:39.591748 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.591598    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:03:39.599490 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.599463    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"]
Nov 03 02:03:39.683145 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.683109    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqs74\" (UniqueName: \"kubernetes.io/projected/af9c885c-06e4-4367-8a68-a68481eb857e-kube-api-access-zqs74\") pod \"maas-api-revoke-sub-36a17fde-td9rf\" (UID: \"af9c885c-06e4-4367-8a68-a68481eb857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"
Nov 03 02:03:39.683464 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.683443    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a17fde-td9rf\" (UID: \"af9c885c-06e4-4367-8a68-a68481eb857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"
Nov 03 02:03:39.683673 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.683655    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a17fde-td9rf\" (UID: \"af9c885c-06e4-4367-8a68-a68481eb857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"
Nov 03 02:03:39.785151 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.785117    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqs74\" (UniqueName: \"kubernetes.io/projected/af9c885c-06e4-4367-8a68-a68481eb857e-kube-api-access-zqs74\") pod \"maas-api-revoke-sub-36a17fde-td9rf\" (UID: \"af9c885c-06e4-4367-8a68-a68481eb857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"
Nov 03 02:03:39.785407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.785171    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a17fde-td9rf\" (UID: \"af9c885c-06e4-4367-8a68-a68481eb857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"
Nov 03 02:03:39.785407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.785299    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a17fde-td9rf\" (UID: \"af9c885c-06e4-4367-8a68-a68481eb857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"
Nov 03 02:03:39.785937 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.785909    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a17fde-td9rf\" (UID: \"af9c885c-06e4-4367-8a68-a68481eb857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"
Nov 03 02:03:39.787656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.787636    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a17fde-td9rf\" (UID: \"af9c885c-06e4-4367-8a68-a68481eb857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"
Nov 03 02:03:39.843087 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.843009    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqs74\" (UniqueName: \"kubernetes.io/projected/af9c885c-06e4-4367-8a68-a68481eb857e-kube-api-access-zqs74\") pod \"maas-api-revoke-sub-36a17fde-td9rf\" (UID: \"af9c885c-06e4-4367-8a68-a68481eb857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"
Nov 03 02:03:39.888806 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:39.888765    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"
Nov 03 02:03:40.031360 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:40.031323    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:03:40.032442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:40.032419    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Nov 03 02:03:40.032561 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:40.032467    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5" podUID="65a28645-ef66-46b3-8a68-a999694dadc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Nov 03 02:03:40.283061 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:40.282981    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"]
Nov 03 02:03:40.289080 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:40.289048    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9c885c_06e4_4367_8a68_a68481eb857e.slice/crio-6752176fad70a4bc421436b4fb87aa80ead15a0c223432b5deacb70d75838fd3 WatchSource:0}: Error finding container 6752176fad70a4bc421436b4fb87aa80ead15a0c223432b5deacb70d75838fd3: Status 404 returned error can't find the container with id 6752176fad70a4bc421436b4fb87aa80ead15a0c223432b5deacb70d75838fd3
Nov 03 02:03:40.338005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:40.337977    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf" event={"ID":"af9c885c-06e4-4367-8a68-a68481eb857e","Type":"ContainerStarted","Data":"6752176fad70a4bc421436b4fb87aa80ead15a0c223432b5deacb70d75838fd3"}
Nov 03 02:03:41.032069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:41.032030    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Nov 03 02:03:41.032293 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:41.032101    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5" podUID="65a28645-ef66-46b3-8a68-a999694dadc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Nov 03 02:03:41.217409 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:41.217382    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:03:41.296038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:41.295953    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:03:41.345277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:41.345248    5508 generic.go:356] "Generic (PLEG): container finished" podID="af9c885c-06e4-4367-8a68-a68481eb857e" containerID="cec33e7c2e9424ebcc1b6527b0f1b2bdf9cc9ed2d8e6ba98ee26983d85f2e731" exitCode=0
Nov 03 02:03:41.345620 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:41.345331    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf" event={"ID":"af9c885c-06e4-4367-8a68-a68481eb857e","Type":"ContainerDied","Data":"cec33e7c2e9424ebcc1b6527b0f1b2bdf9cc9ed2d8e6ba98ee26983d85f2e731"}
Nov 03 02:03:42.031908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.031871    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Nov 03 02:03:42.032091 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.031931    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5" podUID="65a28645-ef66-46b3-8a68-a999694dadc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Nov 03 02:03:42.586705 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.586676    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"
Nov 03 02:03:42.727974 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.727939    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9c885c-06e4-4367-8a68-a68481eb857e-kube-api-access-zqs74\" (UniqueName: \"kubernetes.io/projected/af9c885c-06e4-4367-8a68-a68481eb857e-kube-api-access-zqs74\") pod \"af9c885c-06e4-4367-8a68-a68481eb857e\" (UID: \"af9c885c-06e4-4367-8a68-a68481eb857e\") "
Nov 03 02:03:42.728200 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.728005    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-xks-ca\") pod \"af9c885c-06e4-4367-8a68-a68481eb857e\" (UID: \"af9c885c-06e4-4367-8a68-a68481eb857e\") "
Nov 03 02:03:42.728200 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.728070    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-openshift-ca\") pod \"af9c885c-06e4-4367-8a68-a68481eb857e\" (UID: \"af9c885c-06e4-4367-8a68-a68481eb857e\") "
Nov 03 02:03:42.728922 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.728896    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-openshift-ca" pod "af9c885c-06e4-4367-8a68-a68481eb857e" (UID: "af9c885c-06e4-4367-8a68-a68481eb857e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:03:42.730935 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.730906    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9c885c-06e4-4367-8a68-a68481eb857e-kube-api-access-zqs74" pod "af9c885c-06e4-4367-8a68-a68481eb857e" (UID: "af9c885c-06e4-4367-8a68-a68481eb857e"). InnerVolumeSpecName "kube-api-access-zqs74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:03:42.731738 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.731710    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-xks-ca" pod "af9c885c-06e4-4367-8a68-a68481eb857e" (UID: "af9c885c-06e4-4367-8a68-a68481eb857e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:03:42.797393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.797334    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"]
Nov 03 02:03:42.798097 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.798074    5508 cpu_manager.go:414] "containerMap: removing container" podUID="af9c885c-06e4-4367-8a68-a68481eb857e" containerName="revoke-keys"
Nov 03 02:03:42.798097 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.798095    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9c885c-06e4-4367-8a68-a68481eb857e" containerName="revoke-keys"
Nov 03 02:03:42.798324 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.798179    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9c885c-06e4-4367-8a68-a68481eb857e" containerName="revoke-keys"
Nov 03 02:03:42.801666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.801646    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"
Nov 03 02:03:42.830012 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.829981    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqs74\" (UniqueName: \"kubernetes.io/projected/af9c885c-06e4-4367-8a68-a68481eb857e-kube-api-access-zqs74\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:42.830012 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.830010    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:42.830227 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.830019    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af9c885c-06e4-4367-8a68-a68481eb857e-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:42.890374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.889768    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"]
Nov 03 02:03:42.931052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.931023    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76935266-gndqc\" (UID: \"3ad473b1-5cef-4668-b38a-145e2d7af270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"
Nov 03 02:03:42.931221 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.931137    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76935266-gndqc\" (UID: \"3ad473b1-5cef-4668-b38a-145e2d7af270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"
Nov 03 02:03:42.931221 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:42.931158    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkzx7\" (UniqueName: \"kubernetes.io/projected/3ad473b1-5cef-4668-b38a-145e2d7af270-kube-api-access-tkzx7\") pod \"maas-api-revoke-sub-76935266-gndqc\" (UID: \"3ad473b1-5cef-4668-b38a-145e2d7af270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"
Nov 03 02:03:43.031608 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.031519    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Nov 03 02:03:43.031608 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.031585    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5" podUID="65a28645-ef66-46b3-8a68-a999694dadc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Nov 03 02:03:43.031875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.031798    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76935266-gndqc\" (UID: \"3ad473b1-5cef-4668-b38a-145e2d7af270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"
Nov 03 02:03:43.031875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.031844    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkzx7\" (UniqueName: \"kubernetes.io/projected/3ad473b1-5cef-4668-b38a-145e2d7af270-kube-api-access-tkzx7\") pod \"maas-api-revoke-sub-76935266-gndqc\" (UID: \"3ad473b1-5cef-4668-b38a-145e2d7af270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"
Nov 03 02:03:43.031986 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.031953    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76935266-gndqc\" (UID: \"3ad473b1-5cef-4668-b38a-145e2d7af270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"
Nov 03 02:03:43.032609 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.032589    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76935266-gndqc\" (UID: \"3ad473b1-5cef-4668-b38a-145e2d7af270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"
Nov 03 02:03:43.034140 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.034120    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76935266-gndqc\" (UID: \"3ad473b1-5cef-4668-b38a-145e2d7af270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"
Nov 03 02:03:43.090254 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.090211    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkzx7\" (UniqueName: \"kubernetes.io/projected/3ad473b1-5cef-4668-b38a-145e2d7af270-kube-api-access-tkzx7\") pod \"maas-api-revoke-sub-76935266-gndqc\" (UID: \"3ad473b1-5cef-4668-b38a-145e2d7af270\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"
Nov 03 02:03:43.115184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.115151    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"
Nov 03 02:03:43.387905 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.387813    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"
Nov 03 02:03:43.387905 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.387841    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf" event={"ID":"af9c885c-06e4-4367-8a68-a68481eb857e","Type":"ContainerDied","Data":"6752176fad70a4bc421436b4fb87aa80ead15a0c223432b5deacb70d75838fd3"}
Nov 03 02:03:43.387905 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.387884    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6752176fad70a4bc421436b4fb87aa80ead15a0c223432b5deacb70d75838fd3"
Nov 03 02:03:43.531985 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.531953    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"]
Nov 03 02:03:43.538666 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:43.538639    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad473b1_5cef_4668_b38a_145e2d7af270.slice/crio-c00511bd562789e73b6d6a38b8235a02c42c3850ccc7e31c9268acbb9e7d47c9 WatchSource:0}: Error finding container c00511bd562789e73b6d6a38b8235a02c42c3850ccc7e31c9268acbb9e7d47c9: Status 404 returned error can't find the container with id c00511bd562789e73b6d6a38b8235a02c42c3850ccc7e31c9268acbb9e7d47c9
Nov 03 02:03:43.579787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:43.579741    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:03:43.815134 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:03:43.815092    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:03:44.031782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:44.031751    5508 patch_prober.go:29] interesting pod/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Nov 03 02:03:44.031967 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:44.031803    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5" podUID="65a28645-ef66-46b3-8a68-a999694dadc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Nov 03 02:03:44.396374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:44.396310    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc" event={"ID":"3ad473b1-5cef-4668-b38a-145e2d7af270","Type":"ContainerStarted","Data":"d8ced8e85c1f27bbc4d5834ddc053b515641a3cd2d32dc5838b759a6d967ac73"}
Nov 03 02:03:44.396568 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:44.396389    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc" event={"ID":"3ad473b1-5cef-4668-b38a-145e2d7af270","Type":"ContainerStarted","Data":"c00511bd562789e73b6d6a38b8235a02c42c3850ccc7e31c9268acbb9e7d47c9"}
Nov 03 02:03:44.505640 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:44.505577    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc" podStartSLOduration=2.505562265 podStartE2EDuration="2.505562265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:44.503972055 +0000 UTC m=+2703.929578520" watchObservedRunningTime="2026-11-03 02:03:44.505562265 +0000 UTC m=+2703.931168779"
Nov 03 02:03:45.040287 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.040244    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:03:45.041539 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.041521    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:03:45.404922 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.404892    5508 generic.go:356] "Generic (PLEG): container finished" podID="3ad473b1-5cef-4668-b38a-145e2d7af270" containerID="d8ced8e85c1f27bbc4d5834ddc053b515641a3cd2d32dc5838b759a6d967ac73" exitCode=0
Nov 03 02:03:45.405082 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.404976    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc" event={"ID":"3ad473b1-5cef-4668-b38a-145e2d7af270","Type":"ContainerDied","Data":"d8ced8e85c1f27bbc4d5834ddc053b515641a3cd2d32dc5838b759a6d967ac73"}
Nov 03 02:03:45.854540 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.854423    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"]
Nov 03 02:03:45.860427 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.860407    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:45.916380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.916324    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-059b23-openshift-default-dockercfg-t7jkg\""
Nov 03 02:03:45.949148 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.949109    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"]
Nov 03 02:03:45.965839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.965796    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-podinfo\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:45.966105 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.966086    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-data\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:45.966253 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.966240    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-token\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:45.966460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.966442    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-socket\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:45.966696 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.966677    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqn78\" (UniqueName: \"kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-kube-api-access-vqn78\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:45.966823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.966809    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-certs\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:45.966955 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.966941    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istiod-ca-cert\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:45.967086 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.967070    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-credential-socket\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:45.967204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:45.967190    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-envoy\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.068171 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.068139    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-podinfo\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.068171 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.068180    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-data\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.068788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.068208    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-token\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.068788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.068253    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-socket\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.068788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.068326    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqn78\" (UniqueName: \"kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-kube-api-access-vqn78\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.068788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.068362    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-certs\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.068788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.068399    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istiod-ca-cert\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.068788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.068434    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-credential-socket\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.068788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.068460    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-envoy\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.068788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.068664    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-data\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.068788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.068782    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-certs\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.069233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.069109    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-socket\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.069233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.069149    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-credential-socket\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.069329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.069310    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istiod-ca-cert\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.070974 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.070942    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-envoy\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.071216 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.071200    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-podinfo\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.125359 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.125283    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-token\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.125359 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.125326    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqn78\" (UniqueName: \"kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-kube-api-access-vqn78\") pod \"e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") " pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.252248 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.252211    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:46.534238 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.534215    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"
Nov 03 02:03:46.676020 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.675939    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad473b1-5cef-4668-b38a-145e2d7af270-kube-api-access-tkzx7\" (UniqueName: \"kubernetes.io/projected/3ad473b1-5cef-4668-b38a-145e2d7af270-kube-api-access-tkzx7\") pod \"3ad473b1-5cef-4668-b38a-145e2d7af270\" (UID: \"3ad473b1-5cef-4668-b38a-145e2d7af270\") "
Nov 03 02:03:46.676204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.676059    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-openshift-ca\") pod \"3ad473b1-5cef-4668-b38a-145e2d7af270\" (UID: \"3ad473b1-5cef-4668-b38a-145e2d7af270\") "
Nov 03 02:03:46.676204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.676111    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-xks-ca\") pod \"3ad473b1-5cef-4668-b38a-145e2d7af270\" (UID: \"3ad473b1-5cef-4668-b38a-145e2d7af270\") "
Nov 03 02:03:46.676479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.676450    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-openshift-ca" pod "3ad473b1-5cef-4668-b38a-145e2d7af270" (UID: "3ad473b1-5cef-4668-b38a-145e2d7af270"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:03:46.678171 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.678145    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-xks-ca" pod "3ad473b1-5cef-4668-b38a-145e2d7af270" (UID: "3ad473b1-5cef-4668-b38a-145e2d7af270"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:03:46.678286 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.678220    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad473b1-5cef-4668-b38a-145e2d7af270-kube-api-access-tkzx7" pod "3ad473b1-5cef-4668-b38a-145e2d7af270" (UID: "3ad473b1-5cef-4668-b38a-145e2d7af270"). InnerVolumeSpecName "kube-api-access-tkzx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:03:46.696057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.696035    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"]
Nov 03 02:03:46.697662 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:46.697639    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b83f3c5_8dbb_4187_ae4d_37b745bde034.slice/crio-7d7baade3bacc8493cf2ed09d5f9c127a65529130f29e8474d53b684932c6932 WatchSource:0}: Error finding container 7d7baade3bacc8493cf2ed09d5f9c127a65529130f29e8474d53b684932c6932: Status 404 returned error can't find the container with id 7d7baade3bacc8493cf2ed09d5f9c127a65529130f29e8474d53b684932c6932
Nov 03 02:03:46.699947 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.699915    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:46.700042 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.699991    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:46.700042 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.700035    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:46.777060 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.777032    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:46.777060 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.777059    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkzx7\" (UniqueName: \"kubernetes.io/projected/3ad473b1-5cef-4668-b38a-145e2d7af270-kube-api-access-tkzx7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:46.777269 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:46.777070    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ad473b1-5cef-4668-b38a-145e2d7af270-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:03:47.421675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:47.421644    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7" event={"ID":"0b83f3c5-8dbb-4187-ae4d-37b745bde034","Type":"ContainerStarted","Data":"2505c88ff6f85b73aa5244b4732d0d1ad2f39b708a91f63803b2f7f7bd07e778"}
Nov 03 02:03:47.422134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:47.421703    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7" event={"ID":"0b83f3c5-8dbb-4187-ae4d-37b745bde034","Type":"ContainerStarted","Data":"7d7baade3bacc8493cf2ed09d5f9c127a65529130f29e8474d53b684932c6932"}
Nov 03 02:03:47.423184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:47.423160    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc" event={"ID":"3ad473b1-5cef-4668-b38a-145e2d7af270","Type":"ContainerDied","Data":"c00511bd562789e73b6d6a38b8235a02c42c3850ccc7e31c9268acbb9e7d47c9"}
Nov 03 02:03:47.423281 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:47.423192    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"
Nov 03 02:03:47.423281 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:47.423197    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c00511bd562789e73b6d6a38b8235a02c42c3850ccc7e31c9268acbb9e7d47c9"
Nov 03 02:03:47.538782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:47.538574    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7" podStartSLOduration=2.538559684 podStartE2EDuration="2.538559684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:47.534046321 +0000 UTC m=+2706.959652787" watchObservedRunningTime="2026-11-03 02:03:47.538559684 +0000 UTC m=+2706.964166149"
Nov 03 02:03:48.253085 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:48.253055    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:48.259449 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:48.259423    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:48.430188 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:48.430153    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:48.431235 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:48.431210    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:03:49.128798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.128762    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"]
Nov 03 02:03:49.129518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.129497    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3ad473b1-5cef-4668-b38a-145e2d7af270" containerName="revoke-keys"
Nov 03 02:03:49.129578 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.129522    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad473b1-5cef-4668-b38a-145e2d7af270" containerName="revoke-keys"
Nov 03 02:03:49.129640 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.129629    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad473b1-5cef-4668-b38a-145e2d7af270" containerName="revoke-keys"
Nov 03 02:03:49.133879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.133854    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.221435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.205193    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"]
Nov 03 02:03:49.221435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.209675    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.221435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.211685    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-data\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.221435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.211843    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-podinfo\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.221435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.211979    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-credential-socket\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.221435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.212014    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-envoy\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.221435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.212144    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81bb277-1988-46e5-833e-c48ebe58a4a1-istiod-ca-cert\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.221435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.212216    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjjnq\" (UniqueName: \"kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-kube-api-access-pjjnq\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.221435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.212286    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-certs\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.221435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.212315    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-socket\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.221435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.212461    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-token\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.234951 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.232964    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f7f61d-openshift-default-dockercfg-trk4t\""
Nov 03 02:03:49.246311 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.239675    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"]
Nov 03 02:03:49.317406 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317036    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-podinfo\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.317406 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317158    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-credential-socket\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.317406 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317202    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-envoy\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.317406 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317267    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81bb277-1988-46e5-833e-c48ebe58a4a1-istiod-ca-cert\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.317406 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317295    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.317406 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317322    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b927e0b6-c156-4323-819b-300546f8d1cc-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.317406 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317374    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.317406 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317410    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjjnq\" (UniqueName: \"kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-kube-api-access-pjjnq\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.317999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317455    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-certs\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.317999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317482    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-socket\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.317999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317517    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.317999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317566    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.317999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317616    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.317999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317658    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-token\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.317999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317696    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b927e0b6-c156-4323-819b-300546f8d1cc-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.317999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317742    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-data\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.317999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317770    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.317999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.317840    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj627\" (UniqueName: \"kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-kube-api-access-nj627\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.350896 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.329987    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-certs\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.350896 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.330251    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-credential-socket\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.395426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.359880    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81bb277-1988-46e5-833e-c48ebe58a4a1-istiod-ca-cert\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.364329    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-socket\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.364901    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-data\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.365098    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"]
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.366825    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-podinfo\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.420542    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.427697    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b927e0b6-c156-4323-819b-300546f8d1cc-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.427748    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.427792    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj627\" (UniqueName: \"kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-kube-api-access-nj627\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.427950    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.427982    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b927e0b6-c156-4323-819b-300546f8d1cc-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.428018    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.428082    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.428122    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.428548    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.424304    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.429551    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.430275    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.430938    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b927e0b6-c156-4323-819b-300546f8d1cc-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.432383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.431191    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.436595 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.436138    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b927e0b6-c156-4323-819b-300546f8d1cc-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.436595 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.436298    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-envoy\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.453364 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.449981    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-token\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.472294 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.469764    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjjnq\" (UniqueName: \"kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-kube-api-access-pjjnq\") pod \"e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") " pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.497951 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.497864    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj627\" (UniqueName: \"kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-kube-api-access-nj627\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.498615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.498587    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.525946 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.525717    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:49.749726 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.749693    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:49.963700 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.963659    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"]
Nov 03 02:03:49.968964 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:49.968933    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb927e0b6_c156_4323_819b_300546f8d1cc.slice/crio-411dac136abf3527adec19871dd151c7a79092313706eb713f1c6815ec3fd047 WatchSource:0}: Error finding container 411dac136abf3527adec19871dd151c7a79092313706eb713f1c6815ec3fd047: Status 404 returned error can't find the container with id 411dac136abf3527adec19871dd151c7a79092313706eb713f1c6815ec3fd047
Nov 03 02:03:49.971907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.971863    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:49.972024 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.971953    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:49.972024 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:49.971997    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:50.133809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:50.133767    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"]
Nov 03 02:03:50.139181 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:50.139154    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81bb277_1988_46e5_833e_c48ebe58a4a1.slice/crio-a287cad25b2cc6b98aac63831717d217131e755626e48605c8f8e750837f30af WatchSource:0}: Error finding container a287cad25b2cc6b98aac63831717d217131e755626e48605c8f8e750837f30af: Status 404 returned error can't find the container with id a287cad25b2cc6b98aac63831717d217131e755626e48605c8f8e750837f30af
Nov 03 02:03:50.141924 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:50.141894    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:50.142027 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:50.141959    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:50.142027 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:50.141993    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:50.488041 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:50.488007    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd" event={"ID":"b81bb277-1988-46e5-833e-c48ebe58a4a1","Type":"ContainerStarted","Data":"c99d6df9768c419c51252268157643548f3f8a800c88e03534a95279b2dfbfc6"}
Nov 03 02:03:50.488641 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:50.488592    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd" event={"ID":"b81bb277-1988-46e5-833e-c48ebe58a4a1","Type":"ContainerStarted","Data":"a287cad25b2cc6b98aac63831717d217131e755626e48605c8f8e750837f30af"}
Nov 03 02:03:50.489972 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:50.489935    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c" event={"ID":"b927e0b6-c156-4323-819b-300546f8d1cc","Type":"ContainerStarted","Data":"8ca0cc77229ec48597d8575f10aba03c76cd16593e9a087000da7900e42d8d3a"}
Nov 03 02:03:50.490172 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:50.489985    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c" event={"ID":"b927e0b6-c156-4323-819b-300546f8d1cc","Type":"ContainerStarted","Data":"411dac136abf3527adec19871dd151c7a79092313706eb713f1c6815ec3fd047"}
Nov 03 02:03:50.526665 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:50.526630    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:50.607131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:50.607064    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd" podStartSLOduration=1.607048094 podStartE2EDuration="1.607048094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:50.602724486 +0000 UTC m=+2710.028330951" watchObservedRunningTime="2026-11-03 02:03:50.607048094 +0000 UTC m=+2710.032654558"
Nov 03 02:03:50.755419 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:50.750467    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:51.528659 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:51.528626    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 03 02:03:51.529072 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:51.528675    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c" podUID="b927e0b6-c156-4323-819b-300546f8d1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 03 02:03:51.782427 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:51.779663    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:51.895140 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:51.895080    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c" podStartSLOduration=2.895064278 podStartE2EDuration="2.895064278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:50.723796951 +0000 UTC m=+2710.149403417" watchObservedRunningTime="2026-11-03 02:03:51.895064278 +0000 UTC m=+2711.320670743"
Nov 03 02:03:52.228812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.228777    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"]
Nov 03 02:03:52.232980 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.232959    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.323374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.311854    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-a54cab-openshift-default-dockercfg-sp4k2\""
Nov 03 02:03:52.514444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.480860    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-data\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.514444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.481060    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcx4m\" (UniqueName: \"kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-kube-api-access-rcx4m\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.514444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.481112    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-token\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.514444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.481156    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-socket\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.514444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.481180    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-certs\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.514444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.481231    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-envoy\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.514444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.481276    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istiod-ca-cert\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.514444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.481318    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-credential-socket\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.514444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.481435    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-podinfo\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.514444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.491554    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"]
Nov 03 02:03:52.530065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.530026    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 03 02:03:52.530567 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.530100    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c" podUID="b927e0b6-c156-4323-819b-300546f8d1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 03 02:03:52.539430 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.539375    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:52.540586 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.540564    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:03:52.582779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.582325    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-podinfo\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.583046 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.583020    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-data\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.583374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.583329    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcx4m\" (UniqueName: \"kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-kube-api-access-rcx4m\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.583693 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.583608    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-token\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.583879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.583859    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-socket\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.584017 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.584000    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-certs\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.584180 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.584161    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-envoy\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.584907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.584883    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istiod-ca-cert\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.585104 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.585085    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-credential-socket\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.585200 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.585178    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-podinfo\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.587386 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.585729    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-credential-socket\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.587386 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.585890    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-socket\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.587386 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.586929    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istiod-ca-cert\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.587888 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.587868    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-certs\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.588071 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.588050    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-data\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.588595 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.588572    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-envoy\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.643173 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.643146    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcx4m\" (UniqueName: \"kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-kube-api-access-rcx4m\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.643370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.643336    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-token\") pod \"e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") " pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:52.846424 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:52.846312    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:53.309913 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:53.309873    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"]
Nov 03 02:03:53.315864 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:03:53.315836    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed3911d_2e2f_463d_93aa_cd8ef0bf9053.slice/crio-a99d597dabc42e3797ae1c092bad2731f0c622911aa579f991af3a4b5023db9d WatchSource:0}: Error finding container a99d597dabc42e3797ae1c092bad2731f0c622911aa579f991af3a4b5023db9d: Status 404 returned error can't find the container with id a99d597dabc42e3797ae1c092bad2731f0c622911aa579f991af3a4b5023db9d
Nov 03 02:03:53.318176 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:53.318146    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:53.318257 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:53.318208    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:53.318257 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:53.318236    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:03:53.526634 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:53.526421    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 03 02:03:53.526634 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:53.526464    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c" podUID="b927e0b6-c156-4323-819b-300546f8d1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 03 02:03:53.573426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:53.559737    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" event={"ID":"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053","Type":"ContainerStarted","Data":"6f9451f1166b46677e76f7963db18b3894b036b3f8c8020265697d8937d4f7bf"}
Nov 03 02:03:53.573426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:53.559797    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" event={"ID":"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053","Type":"ContainerStarted","Data":"a99d597dabc42e3797ae1c092bad2731f0c622911aa579f991af3a4b5023db9d"}
Nov 03 02:03:53.857429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:53.855457    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:53.990600 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:53.990529    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" podStartSLOduration=1.990508913 podStartE2EDuration="1.990508913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:53.674613912 +0000 UTC m=+2713.100220378" watchObservedRunningTime="2026-11-03 02:03:53.990508913 +0000 UTC m=+2713.416115379"
Nov 03 02:03:53.992739 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:53.992708    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"]
Nov 03 02:03:53.999410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:53.998941    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"
Nov 03 02:03:54.015295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:54.015270    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph2gt\" (UniqueName: \"kubernetes.io/projected/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-kube-api-access-ph2gt\") pod \"payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4\" (UID: \"acb627be-2d98-4b57-86e6-99ed6ff1e3b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"
Nov 03 02:03:54.015435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:54.015303    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4\" (UID: \"acb627be-2d98-4b57-86e6-99ed6ff1e3b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"
Nov 03 02:03:54.030382 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:54.030344    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"]
Nov 03 02:03:54.058371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:54.054245    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-059b23\""
Nov 03 02:03:54.129424 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:54.127138    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph2gt\" (UniqueName: \"kubernetes.io/projected/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-kube-api-access-ph2gt\") pod \"payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4\" (UID: \"acb627be-2d98-4b57-86e6-99ed6ff1e3b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"
Nov 03 02:03:54.129424 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:54.127193    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4\" (UID: \"acb627be-2d98-4b57-86e6-99ed6ff1e3b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"
Nov 03 02:03:54.129424 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:54.128244    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4\" (UID: \"acb627be-2d98-4b57-86e6-99ed6ff1e3b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"
Nov 03 02:03:54.224921 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:54.221634    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph2gt\" (UniqueName: \"kubernetes.io/projected/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-kube-api-access-ph2gt\") pod \"payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4\" (UID: \"acb627be-2d98-4b57-86e6-99ed6ff1e3b2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"
Nov 03 02:03:54.315788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:54.315761    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"
Nov 03 02:03:54.527100 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:54.527065    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 03 02:03:54.527277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:54.527119    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c" podUID="b927e0b6-c156-4323-819b-300546f8d1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 03 02:03:54.717285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:54.717252    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"]
Nov 03 02:03:54.875859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:54.875830    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:55.439373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.435538    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"]
Nov 03 02:03:55.443369 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.440669    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"
Nov 03 02:03:55.500368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.499653    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:03:55.500368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.499838    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:03:55.508369 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.504684    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"]
Nov 03 02:03:55.508369 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.505389    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfdgj\" (UniqueName: \"kubernetes.io/projected/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-kube-api-access-gfdgj\") pod \"maas-api-revoke-sub-37301876-gt5fq\" (UID: \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"
Nov 03 02:03:55.508369 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.505448    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37301876-gt5fq\" (UID: \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"
Nov 03 02:03:55.508369 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.505571    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37301876-gt5fq\" (UID: \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"
Nov 03 02:03:55.527583 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.527433    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 03 02:03:55.527583 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.527500    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c" podUID="b927e0b6-c156-4323-819b-300546f8d1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 03 02:03:55.604316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.604189    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4" event={"ID":"acb627be-2d98-4b57-86e6-99ed6ff1e3b2","Type":"ContainerStarted","Data":"453c243334650e5112c2d23a36196c481903de801f5d87f0aef7545584988541"}
Nov 03 02:03:55.604316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.604250    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4" event={"ID":"acb627be-2d98-4b57-86e6-99ed6ff1e3b2","Type":"ContainerStarted","Data":"7405caedd967ba455c15ae657b3b1a9c9e28f6a4dec2d670248498fa06bb87f6"}
Nov 03 02:03:55.604779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.604700    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"
Nov 03 02:03:55.604779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.604744    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:55.607772 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.605945    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:03:55.607772 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.606274    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfdgj\" (UniqueName: \"kubernetes.io/projected/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-kube-api-access-gfdgj\") pod \"maas-api-revoke-sub-37301876-gt5fq\" (UID: \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"
Nov 03 02:03:55.607772 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.606320    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37301876-gt5fq\" (UID: \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"
Nov 03 02:03:55.607772 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.606377    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37301876-gt5fq\" (UID: \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"
Nov 03 02:03:55.607772 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.607488    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37301876-gt5fq\" (UID: \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"
Nov 03 02:03:55.613365 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.611182    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37301876-gt5fq\" (UID: \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"
Nov 03 02:03:55.695425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.683809    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfdgj\" (UniqueName: \"kubernetes.io/projected/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-kube-api-access-gfdgj\") pod \"maas-api-revoke-sub-37301876-gt5fq\" (UID: \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"
Nov 03 02:03:55.739791 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.739722    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4" podStartSLOduration=2.739708736 podStartE2EDuration="2.739708736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:55.737838557 +0000 UTC m=+2715.163445023" watchObservedRunningTime="2026-11-03 02:03:55.739708736 +0000 UTC m=+2715.165315249"
Nov 03 02:03:55.776371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:55.775725    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"
Nov 03 02:03:56.551996 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:56.551915    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"]
Nov 03 02:03:56.563685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:56.560446    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 03 02:03:56.563685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:56.560498    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c" podUID="b927e0b6-c156-4323-819b-300546f8d1cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 03 02:03:56.760199 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:56.760143    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq" event={"ID":"7fa7ff01-4dcf-4b94-8480-bc2274ef9851","Type":"ContainerStarted","Data":"bc3419661d1710115ec0de43607bce85331566aa4b18847417df9e021c9e7d0a"}
Nov 03 02:03:57.724492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:57.724453    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq" event={"ID":"7fa7ff01-4dcf-4b94-8480-bc2274ef9851","Type":"ContainerStarted","Data":"e8f66b1340724ed188f09ddc02ab0b3c3cc32ee73a860fb44a8720d059724df4"}
Nov 03 02:03:57.838377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:57.835508    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq" podStartSLOduration=2.8354871729999997 podStartE2EDuration="2.835487173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:03:57.832367107 +0000 UTC m=+2717.257973589" watchObservedRunningTime="2026-11-03 02:03:57.835487173 +0000 UTC m=+2717.261093641"
Nov 03 02:03:57.960900 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:57.960866    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:57.961534 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:57.961390    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:57.963762 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:57.963519    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:03:58.780370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:58.779337    5508 generic.go:356] "Generic (PLEG): container finished" podID="7fa7ff01-4dcf-4b94-8480-bc2274ef9851" containerID="e8f66b1340724ed188f09ddc02ab0b3c3cc32ee73a860fb44a8720d059724df4" exitCode=0
Nov 03 02:03:58.787402 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:58.785885    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq" event={"ID":"7fa7ff01-4dcf-4b94-8480-bc2274ef9851","Type":"ContainerDied","Data":"e8f66b1340724ed188f09ddc02ab0b3c3cc32ee73a860fb44a8720d059724df4"}
Nov 03 02:03:59.938455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:03:59.938432    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"
Nov 03 02:04:00.082999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.082918    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-openshift-ca\") pod \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\" (UID: \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\") "
Nov 03 02:04:00.083182 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.083005    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-kube-api-access-gfdgj\" (UniqueName: \"kubernetes.io/projected/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-kube-api-access-gfdgj\") pod \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\" (UID: \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\") "
Nov 03 02:04:00.083182 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.083044    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-xks-ca\") pod \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\" (UID: \"7fa7ff01-4dcf-4b94-8480-bc2274ef9851\") "
Nov 03 02:04:00.083337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.083300    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-openshift-ca" pod "7fa7ff01-4dcf-4b94-8480-bc2274ef9851" (UID: "7fa7ff01-4dcf-4b94-8480-bc2274ef9851"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:04:00.083679 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.083655    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:00.085242 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.085206    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-xks-ca" pod "7fa7ff01-4dcf-4b94-8480-bc2274ef9851" (UID: "7fa7ff01-4dcf-4b94-8480-bc2274ef9851"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:04:00.085546 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.085525    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-kube-api-access-gfdgj" pod "7fa7ff01-4dcf-4b94-8480-bc2274ef9851" (UID: "7fa7ff01-4dcf-4b94-8480-bc2274ef9851"). InnerVolumeSpecName "kube-api-access-gfdgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:04:00.185478 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.185444    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfdgj\" (UniqueName: \"kubernetes.io/projected/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-kube-api-access-gfdgj\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:00.185478 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.185477    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa7ff01-4dcf-4b94-8480-bc2274ef9851-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:00.388421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.388178    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"]
Nov 03 02:04:00.388969 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.388936    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7fa7ff01-4dcf-4b94-8480-bc2274ef9851" containerName="revoke-keys"
Nov 03 02:04:00.388969 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.388968    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa7ff01-4dcf-4b94-8480-bc2274ef9851" containerName="revoke-keys"
Nov 03 02:04:00.389178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.389156    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa7ff01-4dcf-4b94-8480-bc2274ef9851" containerName="revoke-keys"
Nov 03 02:04:00.392831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.392814    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"
Nov 03 02:04:00.449599 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.449569    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"]
Nov 03 02:04:00.490428 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.490385    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b62078-fl989\" (UID: \"8ba7ffb5-2c69-4578-8106-24ac424d1469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"
Nov 03 02:04:00.490615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.490523    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmm7q\" (UniqueName: \"kubernetes.io/projected/8ba7ffb5-2c69-4578-8106-24ac424d1469-kube-api-access-hmm7q\") pod \"maas-api-revoke-sub-41b62078-fl989\" (UID: \"8ba7ffb5-2c69-4578-8106-24ac424d1469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"
Nov 03 02:04:00.490696 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.490632    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b62078-fl989\" (UID: \"8ba7ffb5-2c69-4578-8106-24ac424d1469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"
Nov 03 02:04:00.592308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.592273    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b62078-fl989\" (UID: \"8ba7ffb5-2c69-4578-8106-24ac424d1469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"
Nov 03 02:04:00.592499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.592379    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmm7q\" (UniqueName: \"kubernetes.io/projected/8ba7ffb5-2c69-4578-8106-24ac424d1469-kube-api-access-hmm7q\") pod \"maas-api-revoke-sub-41b62078-fl989\" (UID: \"8ba7ffb5-2c69-4578-8106-24ac424d1469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"
Nov 03 02:04:00.592499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.592444    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b62078-fl989\" (UID: \"8ba7ffb5-2c69-4578-8106-24ac424d1469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"
Nov 03 02:04:00.593056 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.593020    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b62078-fl989\" (UID: \"8ba7ffb5-2c69-4578-8106-24ac424d1469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"
Nov 03 02:04:00.594908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.594882    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b62078-fl989\" (UID: \"8ba7ffb5-2c69-4578-8106-24ac424d1469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"
Nov 03 02:04:00.649452 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.649424    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmm7q\" (UniqueName: \"kubernetes.io/projected/8ba7ffb5-2c69-4578-8106-24ac424d1469-kube-api-access-hmm7q\") pod \"maas-api-revoke-sub-41b62078-fl989\" (UID: \"8ba7ffb5-2c69-4578-8106-24ac424d1469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"
Nov 03 02:04:00.707279 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.707258    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"
Nov 03 02:04:00.798869 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.798840    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq" event={"ID":"7fa7ff01-4dcf-4b94-8480-bc2274ef9851","Type":"ContainerDied","Data":"bc3419661d1710115ec0de43607bce85331566aa4b18847417df9e021c9e7d0a"}
Nov 03 02:04:00.799011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.798885    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc3419661d1710115ec0de43607bce85331566aa4b18847417df9e021c9e7d0a"
Nov 03 02:04:00.799011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:00.798937    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"
Nov 03 02:04:01.087698 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:01.087458    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"]
Nov 03 02:04:01.093922 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:04:01.093891    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba7ffb5_2c69_4578_8106_24ac424d1469.slice/crio-461c9885eff5d2a0ec38aa54282ee5d608a455f0bc11895edb8351e6e01d2abe WatchSource:0}: Error finding container 461c9885eff5d2a0ec38aa54282ee5d608a455f0bc11895edb8351e6e01d2abe: Status 404 returned error can't find the container with id 461c9885eff5d2a0ec38aa54282ee5d608a455f0bc11895edb8351e6e01d2abe
Nov 03 02:04:01.807519 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:01.807486    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989" event={"ID":"8ba7ffb5-2c69-4578-8106-24ac424d1469","Type":"ContainerStarted","Data":"946fccc9fceadbec7405c93a6da985530d89677e037d72650bdc7652347c8fcc"}
Nov 03 02:04:01.807716 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:01.807537    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989" event={"ID":"8ba7ffb5-2c69-4578-8106-24ac424d1469","Type":"ContainerStarted","Data":"461c9885eff5d2a0ec38aa54282ee5d608a455f0bc11895edb8351e6e01d2abe"}
Nov 03 02:04:01.917280 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:01.917220    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989" podStartSLOduration=1.917197222 podStartE2EDuration="1.917197222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:01.916011496 +0000 UTC m=+2721.341617960" watchObservedRunningTime="2026-11-03 02:04:01.917197222 +0000 UTC m=+2721.342803691"
Nov 03 02:04:02.816111 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:02.816070    5508 generic.go:356] "Generic (PLEG): container finished" podID="8ba7ffb5-2c69-4578-8106-24ac424d1469" containerID="946fccc9fceadbec7405c93a6da985530d89677e037d72650bdc7652347c8fcc" exitCode=0
Nov 03 02:04:02.816542 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:02.816156    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989" event={"ID":"8ba7ffb5-2c69-4578-8106-24ac424d1469","Type":"ContainerDied","Data":"946fccc9fceadbec7405c93a6da985530d89677e037d72650bdc7652347c8fcc"}
Nov 03 02:04:03.459466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.459329    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"]
Nov 03 02:04:03.463814 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.463791    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"
Nov 03 02:04:03.528874 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.528842    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-a54cab\""
Nov 03 02:04:03.531497 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.531470    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-a54cab-dockercfg-rwfpz\""
Nov 03 02:04:03.581871 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.581818    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwrqk\" (UniqueName: \"kubernetes.io/projected/d0cfd5a0-8059-492b-b503-aa3644196ec4-kube-api-access-cwrqk\") pod \"payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2\" (UID: \"d0cfd5a0-8059-492b-b503-aa3644196ec4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"
Nov 03 02:04:03.582321 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.582299    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0cfd5a0-8059-492b-b503-aa3644196ec4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2\" (UID: \"d0cfd5a0-8059-492b-b503-aa3644196ec4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"
Nov 03 02:04:03.627139 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.627105    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"]
Nov 03 02:04:03.644394 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.644365    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"]
Nov 03 02:04:03.651030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.650984    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"
Nov 03 02:04:03.683535 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.683503    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwrqk\" (UniqueName: \"kubernetes.io/projected/d0cfd5a0-8059-492b-b503-aa3644196ec4-kube-api-access-cwrqk\") pod \"payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2\" (UID: \"d0cfd5a0-8059-492b-b503-aa3644196ec4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"
Nov 03 02:04:03.683726 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.683621    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0cfd5a0-8059-492b-b503-aa3644196ec4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2\" (UID: \"d0cfd5a0-8059-492b-b503-aa3644196ec4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"
Nov 03 02:04:03.684329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.684304    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0cfd5a0-8059-492b-b503-aa3644196ec4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2\" (UID: \"d0cfd5a0-8059-492b-b503-aa3644196ec4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"
Nov 03 02:04:03.703266 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.703238    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"]
Nov 03 02:04:03.762570 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.762497    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwrqk\" (UniqueName: \"kubernetes.io/projected/d0cfd5a0-8059-492b-b503-aa3644196ec4-kube-api-access-cwrqk\") pod \"payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2\" (UID: \"d0cfd5a0-8059-492b-b503-aa3644196ec4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"
Nov 03 02:04:03.777288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.777261    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"
Nov 03 02:04:03.784290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.784265    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-479zv\" (UniqueName: \"kubernetes.io/projected/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-kube-api-access-479zv\") pod \"payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h\" (UID: \"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"
Nov 03 02:04:03.784476 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.784449    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h\" (UID: \"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"
Nov 03 02:04:03.887552 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.886110    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h\" (UID: \"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"
Nov 03 02:04:03.887552 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.886282    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479zv\" (UniqueName: \"kubernetes.io/projected/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-kube-api-access-479zv\") pod \"payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h\" (UID: \"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"
Nov 03 02:04:03.887973 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.887582    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h\" (UID: \"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"
Nov 03 02:04:03.944619 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.944588    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-479zv\" (UniqueName: \"kubernetes.io/projected/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-kube-api-access-479zv\") pod \"payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h\" (UID: \"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"
Nov 03 02:04:03.968839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:03.968804    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"
Nov 03 02:04:04.005060 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.005040    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"
Nov 03 02:04:04.087983 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.087954    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-openshift-ca\") pod \"8ba7ffb5-2c69-4578-8106-24ac424d1469\" (UID: \"8ba7ffb5-2c69-4578-8106-24ac424d1469\") "
Nov 03 02:04:04.088167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.088032    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-xks-ca\") pod \"8ba7ffb5-2c69-4578-8106-24ac424d1469\" (UID: \"8ba7ffb5-2c69-4578-8106-24ac424d1469\") "
Nov 03 02:04:04.088167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.088068    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba7ffb5-2c69-4578-8106-24ac424d1469-kube-api-access-hmm7q\" (UniqueName: \"kubernetes.io/projected/8ba7ffb5-2c69-4578-8106-24ac424d1469-kube-api-access-hmm7q\") pod \"8ba7ffb5-2c69-4578-8106-24ac424d1469\" (UID: \"8ba7ffb5-2c69-4578-8106-24ac424d1469\") "
Nov 03 02:04:04.088338 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.088317    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-openshift-ca" pod "8ba7ffb5-2c69-4578-8106-24ac424d1469" (UID: "8ba7ffb5-2c69-4578-8106-24ac424d1469"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:04:04.088519 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.088502    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:04.090458 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.090430    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba7ffb5-2c69-4578-8106-24ac424d1469-kube-api-access-hmm7q" pod "8ba7ffb5-2c69-4578-8106-24ac424d1469" (UID: "8ba7ffb5-2c69-4578-8106-24ac424d1469"). InnerVolumeSpecName "kube-api-access-hmm7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:04:04.091484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.091462    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-xks-ca" pod "8ba7ffb5-2c69-4578-8106-24ac424d1469" (UID: "8ba7ffb5-2c69-4578-8106-24ac424d1469"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:04:04.164435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.164401    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"]
Nov 03 02:04:04.169943 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:04:04.169915    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0cfd5a0_8059_492b_b503_aa3644196ec4.slice/crio-b6cb3d31fa301dd678f3acdbff8d5e58b1874cab9f49f866318b144a3450c5c9 WatchSource:0}: Error finding container b6cb3d31fa301dd678f3acdbff8d5e58b1874cab9f49f866318b144a3450c5c9: Status 404 returned error can't find the container with id b6cb3d31fa301dd678f3acdbff8d5e58b1874cab9f49f866318b144a3450c5c9
Nov 03 02:04:04.190206 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.190179    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba7ffb5-2c69-4578-8106-24ac424d1469-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:04.190300 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.190212    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmm7q\" (UniqueName: \"kubernetes.io/projected/8ba7ffb5-2c69-4578-8106-24ac424d1469-kube-api-access-hmm7q\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:04:04.214026 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.214003    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"]
Nov 03 02:04:04.215292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.215264    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8ba7ffb5-2c69-4578-8106-24ac424d1469" containerName="revoke-keys"
Nov 03 02:04:04.215292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.215287    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba7ffb5-2c69-4578-8106-24ac424d1469" containerName="revoke-keys"
Nov 03 02:04:04.215573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.215444    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba7ffb5-2c69-4578-8106-24ac424d1469" containerName="revoke-keys"
Nov 03 02:04:04.219636 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.219614    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.286305 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.286226    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"]
Nov 03 02:04:04.291650 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.291627    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.291789 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.291671    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b9sg\" (UniqueName: \"kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-kube-api-access-2b9sg\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.291789 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.291756    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.291863 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.291834    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.291900 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.291864    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.291900 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.291888    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.291968 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.291902    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.291968 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.291925    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.291968 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.291945    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.349513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.349483    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"]
Nov 03 02:04:04.355003 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:04:04.354976    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1b2ae6_4a5f_4d90_90a6_24a434c0f5d7.slice/crio-5002592bde4b324ca849730fc7e43af2062eae0bcdaaba9baeee5dcdc1dde269 WatchSource:0}: Error finding container 5002592bde4b324ca849730fc7e43af2062eae0bcdaaba9baeee5dcdc1dde269: Status 404 returned error can't find the container with id 5002592bde4b324ca849730fc7e43af2062eae0bcdaaba9baeee5dcdc1dde269
Nov 03 02:04:04.393418 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.393393    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.393575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.393436    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b9sg\" (UniqueName: \"kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-kube-api-access-2b9sg\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.393575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.393480    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.393719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.393588    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.393719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.393628    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.393719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.393661    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.393719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.393686    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.393953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.393717    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.393953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.393751    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.394106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.394081    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.394165 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.394141    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.394282 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.394246    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.394594 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.394571    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.394676 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.394571    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.395724 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.395707    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.396947 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.396928    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.450630 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.450605    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.450865 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.450844    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b9sg\" (UniqueName: \"kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-kube-api-access-2b9sg\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.534449 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.534419    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:04.834912 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.834830    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"
Nov 03 02:04:04.834912 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.834857    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989" event={"ID":"8ba7ffb5-2c69-4578-8106-24ac424d1469","Type":"ContainerDied","Data":"461c9885eff5d2a0ec38aa54282ee5d608a455f0bc11895edb8351e6e01d2abe"}
Nov 03 02:04:04.835122 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.834914    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="461c9885eff5d2a0ec38aa54282ee5d608a455f0bc11895edb8351e6e01d2abe"
Nov 03 02:04:04.836921 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.836887    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2" event={"ID":"d0cfd5a0-8059-492b-b503-aa3644196ec4","Type":"ContainerStarted","Data":"4111aec62f8726db30af8cdb95818a5c61cd88be477689c8303af1de2d905d5b"}
Nov 03 02:04:04.837102 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.836935    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2" event={"ID":"d0cfd5a0-8059-492b-b503-aa3644196ec4","Type":"ContainerStarted","Data":"b6cb3d31fa301dd678f3acdbff8d5e58b1874cab9f49f866318b144a3450c5c9"}
Nov 03 02:04:04.837102 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.837076    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"
Nov 03 02:04:04.838886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.838861    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h" event={"ID":"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7","Type":"ContainerStarted","Data":"a327cb27e682415180bda9b716590bef3d10aa1afd12ffd6a66558b4f1e52f44"}
Nov 03 02:04:04.839002 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.838900    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h" event={"ID":"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7","Type":"ContainerStarted","Data":"5002592bde4b324ca849730fc7e43af2062eae0bcdaaba9baeee5dcdc1dde269"}
Nov 03 02:04:04.839002 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.838921    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"
Nov 03 02:04:04.949264 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.949203    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2" podStartSLOduration=1.9491883479999998 podStartE2EDuration="1.949188348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:04.94494423 +0000 UTC m=+2724.370550715" watchObservedRunningTime="2026-11-03 02:04:04.949188348 +0000 UTC m=+2724.374794804"
Nov 03 02:04:04.964788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.964752    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"]
Nov 03 02:04:04.969187 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:04:04.969151    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8addfd_f2cf_4a72_af4c_f803c5990dae.slice/crio-7fb9b1f4622b77fe987bfb5efffb2eb5cf71c15de2aa27c7bf26ad213896abe2 WatchSource:0}: Error finding container 7fb9b1f4622b77fe987bfb5efffb2eb5cf71c15de2aa27c7bf26ad213896abe2: Status 404 returned error can't find the container with id 7fb9b1f4622b77fe987bfb5efffb2eb5cf71c15de2aa27c7bf26ad213896abe2
Nov 03 02:04:04.972177 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.972142    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:04.972281 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.972218    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:04.972281 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:04.972275    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:05.104159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:05.104115    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h" podStartSLOduration=2.104101897 podStartE2EDuration="2.104101897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:05.102861495 +0000 UTC m=+2724.528467979" watchObservedRunningTime="2026-11-03 02:04:05.104101897 +0000 UTC m=+2724.529708361"
Nov 03 02:04:05.848185 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:05.848141    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb" event={"ID":"9e8addfd-f2cf-4a72-af4c-f803c5990dae","Type":"ContainerStarted","Data":"bfb38958c612113c4dc86911125e5933fe5683c8d8ed330acd2cb4a51c73c051"}
Nov 03 02:04:05.848373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:05.848241    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb" event={"ID":"9e8addfd-f2cf-4a72-af4c-f803c5990dae","Type":"ContainerStarted","Data":"7fb9b1f4622b77fe987bfb5efffb2eb5cf71c15de2aa27c7bf26ad213896abe2"}
Nov 03 02:04:05.976933 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:05.976857    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb" podStartSLOduration=1.9768333930000002 podStartE2EDuration="1.976833393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:05.966731554 +0000 UTC m=+2725.392338034" watchObservedRunningTime="2026-11-03 02:04:05.976833393 +0000 UTC m=+2725.402439859"
Nov 03 02:04:06.534704 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:06.534668    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:06.536407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:06.536383    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Nov 03 02:04:06.536522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:06.536438    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb" podUID="9e8addfd-f2cf-4a72-af4c-f803c5990dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Nov 03 02:04:06.719618 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:06.719591    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"
Nov 03 02:04:07.371529 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.371493    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"]
Nov 03 02:04:07.384120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.384094    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.489372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.479054    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98f1701-ad67-41e4-a528-23eb264203f8-istio-podinfo\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.489372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.479162    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98f1701-ad67-41e4-a528-23eb264203f8-istiod-ca-cert\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.489372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.479203    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-socket\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.489372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.479284    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-envoy\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.489372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.479318    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-credential-socket\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.489372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.479372    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-certs\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.489372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.479442    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-istio-token\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.489372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.479494    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89dvf\" (UniqueName: \"kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-kube-api-access-89dvf\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.489372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.479570    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-data\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.541371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.540611    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-c34879-openshift-default-dockercfg-m95fd\""
Nov 03 02:04:07.585367 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.568146    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Nov 03 02:04:07.585367 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.568247    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb" podUID="9e8addfd-f2cf-4a72-af4c-f803c5990dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Nov 03 02:04:07.585367 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.580401    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"]
Nov 03 02:04:07.611384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.595274    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-data\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.611384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.595373    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98f1701-ad67-41e4-a528-23eb264203f8-istio-podinfo\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.611384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.595458    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98f1701-ad67-41e4-a528-23eb264203f8-istiod-ca-cert\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.611384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.595497    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-socket\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.611384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.595581    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-envoy\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.611384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.595619    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-credential-socket\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.611384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.595650    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-certs\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.611384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.595733    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-istio-token\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.611384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.595787    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89dvf\" (UniqueName: \"kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-kube-api-access-89dvf\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.611384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.596697    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-data\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.611384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.610657    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98f1701-ad67-41e4-a528-23eb264203f8-istiod-ca-cert\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.611384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.610937    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-certs\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.611384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.611187    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-credential-socket\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.612239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.611505    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-socket\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.643611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.637461    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-envoy\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.643611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.638516    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98f1701-ad67-41e4-a528-23eb264203f8-istio-podinfo\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.669697 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.669584    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89dvf\" (UniqueName: \"kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-kube-api-access-89dvf\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.673931 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.673863    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-istio-token\") pod \"e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") " pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:07.909976 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:07.909944    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:08.391026 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:08.390933    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"]
Nov 03 02:04:08.395867 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:04:08.395841    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98f1701_ad67_41e4_a528_23eb264203f8.slice/crio-e58738f3a3e57f4a860c8428f5b7993050976ff609d15a9852825c8430d9c7bd WatchSource:0}: Error finding container e58738f3a3e57f4a860c8428f5b7993050976ff609d15a9852825c8430d9c7bd: Status 404 returned error can't find the container with id e58738f3a3e57f4a860c8428f5b7993050976ff609d15a9852825c8430d9c7bd
Nov 03 02:04:08.398558 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:08.398525    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:08.398653 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:08.398591    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:08.398653 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:08.398619    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:08.534715 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:08.534679    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Nov 03 02:04:08.534886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:08.534725    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb" podUID="9e8addfd-f2cf-4a72-af4c-f803c5990dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Nov 03 02:04:08.911058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:08.911019    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" event={"ID":"c98f1701-ad67-41e4-a528-23eb264203f8","Type":"ContainerStarted","Data":"e08da6c9e680148b6d99ba863e2f1e4192dee750278c97bff05dc7522adfd450"}
Nov 03 02:04:08.911272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:08.911107    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" event={"ID":"c98f1701-ad67-41e4-a528-23eb264203f8","Type":"ContainerStarted","Data":"e58738f3a3e57f4a860c8428f5b7993050976ff609d15a9852825c8430d9c7bd"}
Nov 03 02:04:09.033225 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:09.033162    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" podStartSLOduration=2.033148822 podStartE2EDuration="2.033148822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:09.028332561 +0000 UTC m=+2728.453939037" watchObservedRunningTime="2026-11-03 02:04:09.033148822 +0000 UTC m=+2728.458755287"
Nov 03 02:04:09.534816 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:09.534780    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Nov 03 02:04:09.535215 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:09.534831    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb" podUID="9e8addfd-f2cf-4a72-af4c-f803c5990dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Nov 03 02:04:09.910256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:09.910219    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:09.916484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:09.916459    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:09.919155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:09.919135    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:09.919857 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:09.919836    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:04:10.535608 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:10.535573    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Nov 03 02:04:10.536135 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:10.535637    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb" podUID="9e8addfd-f2cf-4a72-af4c-f803c5990dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Nov 03 02:04:11.534814 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:11.534778    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Nov 03 02:04:11.534998 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:11.534867    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb" podUID="9e8addfd-f2cf-4a72-af4c-f803c5990dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Nov 03 02:04:12.619719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:12.619685    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:12.620549 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:12.620528    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:12.622367 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:12.622334    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:04:13.811812 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:04:13.811772    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:04:15.851937 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:15.851252    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"
Nov 03 02:04:15.851937 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:15.851341    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"
Nov 03 02:04:20.203180 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:20.203141    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:04:22.257372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:22.256604    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"]
Nov 03 02:04:22.323372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:22.316447    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"
Nov 03 02:04:22.372375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:22.352020    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"]
Nov 03 02:04:22.372375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:22.372209    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-98a394\""
Nov 03 02:04:22.389373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:22.376533    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fac1e0b-84e4-4edb-9abe-9537a002c2db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh\" (UID: \"7fac1e0b-84e4-4edb-9abe-9537a002c2db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"
Nov 03 02:04:22.389373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:22.376767    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr4bg\" (UniqueName: \"kubernetes.io/projected/7fac1e0b-84e4-4edb-9abe-9537a002c2db-kube-api-access-cr4bg\") pod \"payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh\" (UID: \"7fac1e0b-84e4-4edb-9abe-9537a002c2db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"
Nov 03 02:04:22.389373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:22.384935    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-98a394-dockercfg-686s7\""
Nov 03 02:04:22.600864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:22.502387    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr4bg\" (UniqueName: \"kubernetes.io/projected/7fac1e0b-84e4-4edb-9abe-9537a002c2db-kube-api-access-cr4bg\") pod \"payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh\" (UID: \"7fac1e0b-84e4-4edb-9abe-9537a002c2db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"
Nov 03 02:04:22.604536 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:22.601258    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fac1e0b-84e4-4edb-9abe-9537a002c2db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh\" (UID: \"7fac1e0b-84e4-4edb-9abe-9537a002c2db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"
Nov 03 02:04:22.604536 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:22.602917    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fac1e0b-84e4-4edb-9abe-9537a002c2db-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh\" (UID: \"7fac1e0b-84e4-4edb-9abe-9537a002c2db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"
Nov 03 02:04:22.647141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:22.647023    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr4bg\" (UniqueName: \"kubernetes.io/projected/7fac1e0b-84e4-4edb-9abe-9537a002c2db-kube-api-access-cr4bg\") pod \"payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh\" (UID: \"7fac1e0b-84e4-4edb-9abe-9537a002c2db\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"
Nov 03 02:04:22.817740 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:22.817452    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"
Nov 03 02:04:23.388837 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.385472    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"]
Nov 03 02:04:23.392431 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.392409    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.464612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.464578    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.464860 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.464840    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn9dd\" (UniqueName: \"kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-kube-api-access-wn9dd\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.464993 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.464976    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.465121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.465090    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.465229 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.465207    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.465378 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.465363    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.465886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.465625    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.465886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.465775    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.466069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.465905    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.498024 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.497987    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"]
Nov 03 02:04:23.567263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.567172    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.567457 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.567297    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.567457 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.567327    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn9dd\" (UniqueName: \"kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-kube-api-access-wn9dd\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.567457 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.567402    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.567457 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.567438    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.567625 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.567482    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.567625 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.567543    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.567625 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.567619    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.567745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.567662    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.567745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.567691    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.567745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.567714    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-credential-socket\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.567953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.567933    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-certs\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.568035 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.568011    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-data\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.568544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.568520    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istiod-ca-cert\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.570124 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.570102    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-envoy\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.570634 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.570610    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-podinfo\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.624791 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.624752    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-token\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.624964 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.624857    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn9dd\" (UniqueName: \"kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-kube-api-access-wn9dd\") pod \"e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") " pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:23.691890 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.691853    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"]
Nov 03 02:04:23.735081 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:23.735055    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:24.187162 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:24.187122    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"]
Nov 03 02:04:24.192681 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:04:24.192653    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef31a9a_62f8_49a2_9a0b_d20cabfa823e.slice/crio-e970889dfca4e232592fbf949a119026500c88472af91e0c4cf68134d13e28d0 WatchSource:0}: Error finding container e970889dfca4e232592fbf949a119026500c88472af91e0c4cf68134d13e28d0: Status 404 returned error can't find the container with id e970889dfca4e232592fbf949a119026500c88472af91e0c4cf68134d13e28d0
Nov 03 02:04:24.195663 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:24.195630    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:24.195765 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:24.195698    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:24.195765 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:24.195725    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:24.304876 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:24.304828    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh" event={"ID":"7fac1e0b-84e4-4edb-9abe-9537a002c2db","Type":"ContainerStarted","Data":"cb0592d2da7e3505b707ef7abc22a27b3535ca2e108daffac2fd1b6ddbad27e8"}
Nov 03 02:04:24.305119 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:24.304907    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh" event={"ID":"7fac1e0b-84e4-4edb-9abe-9537a002c2db","Type":"ContainerStarted","Data":"a5e07758da8f643275806796d1910949998641053473507a0ac74f85926b5005"}
Nov 03 02:04:24.305119 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:24.305012    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"
Nov 03 02:04:24.307186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:24.307161    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk" event={"ID":"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e","Type":"ContainerStarted","Data":"3972a35dd5d8c08eb457fb2f5e30ff451721e7a7d98b3b96c50a3e9ee7458910"}
Nov 03 02:04:24.307302 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:24.307208    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk" event={"ID":"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e","Type":"ContainerStarted","Data":"e970889dfca4e232592fbf949a119026500c88472af91e0c4cf68134d13e28d0"}
Nov 03 02:04:24.413883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:24.413828    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh" podStartSLOduration=2.413813903 podStartE2EDuration="2.413813903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:24.412292098 +0000 UTC m=+2743.837898564" watchObservedRunningTime="2026-11-03 02:04:24.413813903 +0000 UTC m=+2743.839420368"
Nov 03 02:04:24.562435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:24.560608    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk" podStartSLOduration=1.56058584 podStartE2EDuration="1.56058584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:24.524400714 +0000 UTC m=+2743.950007260" watchObservedRunningTime="2026-11-03 02:04:24.56058584 +0000 UTC m=+2743.986192306"
Nov 03 02:04:24.736120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:24.736086    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:25.737002 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:25.736969    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 02:04:25.737441 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:25.737034    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk" podUID="3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 02:04:26.736258 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:26.736225    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 02:04:26.736461 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:26.736273    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk" podUID="3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 02:04:27.736246 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:27.736209    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 02:04:27.736651 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:27.736261    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk" podUID="3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 02:04:28.508195 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.508161    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"]
Nov 03 02:04:28.512470 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.512454    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:28.565428 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.565396    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-059b23-serving-cert\""
Nov 03 02:04:28.620958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.620923    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"]
Nov 03 02:04:28.647306 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.647271    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6b9575-4487-4e4d-92c5-55a93baff2f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:28.647520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.647316    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:28.647520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.647386    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q66m\" (UniqueName: \"kubernetes.io/projected/eb6b9575-4487-4e4d-92c5-55a93baff2f4-kube-api-access-8q66m\") pod \"maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:28.647520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.647472    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:28.736054 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.736020    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 02:04:28.736231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.736083    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk" podUID="3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 02:04:28.755374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.749628    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6b9575-4487-4e4d-92c5-55a93baff2f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:28.755973 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.755944    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:28.756132 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.756114    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q66m\" (UniqueName: \"kubernetes.io/projected/eb6b9575-4487-4e4d-92c5-55a93baff2f4-kube-api-access-8q66m\") pod \"maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:28.756325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.756310    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:28.757278 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:04:28.757256    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-serving-cert: secret "maas-api-e2e-shared-a-w3-059b23-serving-cert" not found
Nov 03 02:04:28.757501 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:04:28.757487    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-maas-api-tls podName:eb6b9575-4487-4e4d-92c5-55a93baff2f4 nodeName:}" failed. No retries permitted until 2026-11-03 02:04:29.257465128 +0000 UTC m=+2748.683071588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-maas-api-tls") pod "maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5" (UID: "eb6b9575-4487-4e4d-92c5-55a93baff2f4") : secret "maas-api-e2e-shared-a-w3-059b23-serving-cert" not found
Nov 03 02:04:28.758484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.758432    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6b9575-4487-4e4d-92c5-55a93baff2f4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:28.815371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.811987    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:28.847204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:28.846209    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q66m\" (UniqueName: \"kubernetes.io/projected/eb6b9575-4487-4e4d-92c5-55a93baff2f4-kube-api-access-8q66m\") pod \"maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:29.278891 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:29.278856    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:29.281280 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:29.281258    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:29.427058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:29.427026    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:29.735800 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:29.735770    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 02:04:29.735965 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:29.735822    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk" podUID="3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 02:04:29.843177 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:29.843141    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"]
Nov 03 02:04:29.865967 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:04:29.852498    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6b9575_4487_4e4d_92c5_55a93baff2f4.slice/crio-b58e3cebfb9b9098cb10622498ac844a37a1673360c53e6554e8478876bece04 WatchSource:0}: Error finding container b58e3cebfb9b9098cb10622498ac844a37a1673360c53e6554e8478876bece04: Status 404 returned error can't find the container with id b58e3cebfb9b9098cb10622498ac844a37a1673360c53e6554e8478876bece04
Nov 03 02:04:30.378836 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:30.378797    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5" event={"ID":"eb6b9575-4487-4e4d-92c5-55a93baff2f4","Type":"ContainerStarted","Data":"b58e3cebfb9b9098cb10622498ac844a37a1673360c53e6554e8478876bece04"}
Nov 03 02:04:30.735658 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:30.735621    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused" start-of-body=
Nov 03 02:04:30.735833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:30.735677    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk" podUID="3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.13:15021/healthz/ready\": dial tcp 10.133.1.13:15021: connect: connection refused"
Nov 03 02:04:31.390328 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:31.390231    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5" event={"ID":"eb6b9575-4487-4e4d-92c5-55a93baff2f4","Type":"ContainerStarted","Data":"c191e5dd171f7f2ccb8c073b57789e824db2421f307a772a80e59f85845eac5a"}
Nov 03 02:04:31.390328 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:31.390321    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:31.503562 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:31.503508    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5" podStartSLOduration=2.412815036 podStartE2EDuration="3.503491995s" totalImagesPullingTime="1.090676959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:31.501747256 +0000 UTC m=+2750.927353743" watchObservedRunningTime="2026-11-03 02:04:31.503491995 +0000 UTC m=+2750.929098459"
Nov 03 02:04:32.098935 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:32.098892    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:32.099574 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:32.099547    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:32.100556 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:32.100477    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:04:35.317309 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:35.317273    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"
Nov 03 02:04:37.406387 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:37.406342    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:04:43.811776 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:04:43.811740    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:04:49.214168 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.214133    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"]
Nov 03 02:04:49.222133 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.222092    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.310001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.309962    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.310001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.310006    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.310269 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.310135    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.310269 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.310243    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.310448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.310419    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.310612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.310462    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.310612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.310580    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.310742 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.310617    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.310800 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.310758    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmphd\" (UniqueName: \"kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-kube-api-access-wmphd\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.318142 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.318122    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"]
Nov 03 02:04:49.412877 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.412846    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.412877 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.412881    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.413244 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.413214    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-certs\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.413369 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.413280    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.413369 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.413307    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.413509 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.413433    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.413509 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.413466    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.413509 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.413505    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.413692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.413526    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.413692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.413586    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmphd\" (UniqueName: \"kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-kube-api-access-wmphd\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.413805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.413685    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istiod-ca-cert\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.413805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.413719    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-data\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.413986 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.413962    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-credential-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.414264 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.414245    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-socket\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.416049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.416026    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-envoy\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.416197 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.416179    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-podinfo\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.496437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.496342    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmphd\" (UniqueName: \"kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-kube-api-access-wmphd\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.503321 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.503258    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-token\") pod \"e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") " pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:49.545700 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:49.545670    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:50.113913 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:50.113876    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"]
Nov 03 02:04:50.146101 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:04:50.146071    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2002ee43_2f1a_4ea5_98a8_b5afa7a88305.slice/crio-4d670eee92a60022979b0b32207c5e08b2a9f5a57f7970cd0f81aa7af0c6f1b1 WatchSource:0}: Error finding container 4d670eee92a60022979b0b32207c5e08b2a9f5a57f7970cd0f81aa7af0c6f1b1: Status 404 returned error can't find the container with id 4d670eee92a60022979b0b32207c5e08b2a9f5a57f7970cd0f81aa7af0c6f1b1
Nov 03 02:04:50.166368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:50.165750    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:50.166368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:50.165852    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:50.166368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:50.165899    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:50.536063 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:50.536027    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh" event={"ID":"2002ee43-2f1a-4ea5-98a8-b5afa7a88305","Type":"ContainerStarted","Data":"4ecd66cab7395a2921757c28d228f357556b392b61155d9c1322e07aed912128"}
Nov 03 02:04:50.536497 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:50.536102    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh" event={"ID":"2002ee43-2f1a-4ea5-98a8-b5afa7a88305","Type":"ContainerStarted","Data":"4d670eee92a60022979b0b32207c5e08b2a9f5a57f7970cd0f81aa7af0c6f1b1"}
Nov 03 02:04:50.546724 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:50.546700    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:50.666372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:50.666066    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh" podStartSLOduration=1.666047312 podStartE2EDuration="1.666047312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:04:50.663091437 +0000 UTC m=+2770.088697902" watchObservedRunningTime="2026-11-03 02:04:50.666047312 +0000 UTC m=+2770.091653778"
Nov 03 02:04:51.547869 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:51.547841    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 02:04:51.548379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:51.547888    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh" podUID="2002ee43-2f1a-4ea5-98a8-b5afa7a88305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 02:04:52.546196 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:52.546159    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 02:04:52.546414 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:52.546243    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh" podUID="2002ee43-2f1a-4ea5-98a8-b5afa7a88305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 02:04:53.547934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:53.547892    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 02:04:53.548461 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:53.547954    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh" podUID="2002ee43-2f1a-4ea5-98a8-b5afa7a88305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 02:04:54.546992 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:54.546912    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 02:04:54.546992 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:54.546970    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh" podUID="2002ee43-2f1a-4ea5-98a8-b5afa7a88305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 02:04:55.546889 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:55.546845    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 02:04:55.547422 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:55.546913    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh" podUID="2002ee43-2f1a-4ea5-98a8-b5afa7a88305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 02:04:56.546418 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:56.546381    5508 patch_prober.go:29] interesting pod/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 02:04:56.546605 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:56.546445    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh" podUID="2002ee43-2f1a-4ea5-98a8-b5afa7a88305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 02:04:58.016971 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:58.016927    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:58.018935 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:58.018911    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:58.019987 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:58.019963    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:04:59.845164 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:59.845131    5508 generic.go:356] "Generic (PLEG): container finished" podID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerID="2505c88ff6f85b73aa5244b4732d0d1ad2f39b708a91f63803b2f7f7bd07e778" exitCode=0
Nov 03 02:04:59.845702 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:59.845211    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7" event={"ID":"0b83f3c5-8dbb-4187-ae4d-37b745bde034","Type":"ContainerDied","Data":"2505c88ff6f85b73aa5244b4732d0d1ad2f39b708a91f63803b2f7f7bd07e778"}
Nov 03 02:04:59.845776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:59.845734    5508 scope.go:122] "RemoveContainer" containerID="2505c88ff6f85b73aa5244b4732d0d1ad2f39b708a91f63803b2f7f7bd07e778"
Nov 03 02:04:59.847198 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:59.847177    5508 generic.go:356] "Generic (PLEG): container finished" podID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerID="c99d6df9768c419c51252268157643548f3f8a800c88e03534a95279b2dfbfc6" exitCode=0
Nov 03 02:04:59.847330 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:59.847247    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd" event={"ID":"b81bb277-1988-46e5-833e-c48ebe58a4a1","Type":"ContainerDied","Data":"c99d6df9768c419c51252268157643548f3f8a800c88e03534a95279b2dfbfc6"}
Nov 03 02:04:59.847478 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:59.847451    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:59.847553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:59.847520    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:59.847609 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:59.847554    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:59.847750 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:59.847734    5508 scope.go:122] "RemoveContainer" containerID="c99d6df9768c419c51252268157643548f3f8a800c88e03534a95279b2dfbfc6"
Nov 03 02:04:59.849007 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:59.848982    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:59.849095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:59.849044    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:04:59.849095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:04:59.849076    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:05:00.903391 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:00.902846    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7" event={"ID":"0b83f3c5-8dbb-4187-ae4d-37b745bde034","Type":"ContainerStarted","Data":"b93fcc21544e9a20eeca86f6603e518709fdd690407b301a802a6979ab3ed11c"}
Nov 03 02:05:00.905561 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:00.905536    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd" event={"ID":"b81bb277-1988-46e5-833e-c48ebe58a4a1","Type":"ContainerStarted","Data":"728496c3ccf36770c1e878cc6012c57a91144cd483548a5b899fd85ed311c33b"}
Nov 03 02:05:01.260812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:01.260718    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:05:01.268535 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:01.268502    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 03 02:05:01.268813 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:01.268784    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7" podUID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 03 02:05:01.753720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:01.753692    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:05:01.754107 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:01.754072    5508 patch_prober.go:29] interesting pod/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Nov 03 02:05:01.754286 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:01.754263    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd" podUID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Nov 03 02:05:02.252815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:02.252749    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 03 02:05:02.253488 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:02.253454    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7" podUID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 03 02:05:02.772372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:02.769454    5508 patch_prober.go:29] interesting pod/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Nov 03 02:05:02.772372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:02.769517    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd" podUID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Nov 03 02:05:03.096536 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:03.096415    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"]
Nov 03 02:05:03.109625 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:03.109590    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa188f32-tnfs4"]
Nov 03 02:05:03.255375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:03.252718    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 03 02:05:03.255375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:03.252780    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7" podUID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 03 02:05:03.443369 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:03.436590    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:05:03.756431 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:03.754674    5508 patch_prober.go:29] interesting pod/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Nov 03 02:05:03.756431 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:03.754742    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd" podUID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Nov 03 02:05:04.253374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:04.252742    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 03 02:05:04.253374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:04.252805    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7" podUID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 03 02:05:04.713395 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:04.712995    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6628479-581a-41b5-8aac-8a1cf766f7f1" path="/var/lib/kubelet/pods/f6628479-581a-41b5-8aac-8a1cf766f7f1/volumes"
Nov 03 02:05:04.766376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:04.763129    5508 patch_prober.go:29] interesting pod/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Nov 03 02:05:04.766376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:04.763195    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd" podUID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Nov 03 02:05:05.258370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:05.255875    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Nov 03 02:05:05.258370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:05.255962    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7" podUID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Nov 03 02:05:05.752375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:05.750061    5508 patch_prober.go:29] interesting pod/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Nov 03 02:05:05.752375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:05.750133    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd" podUID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Nov 03 02:05:06.257638 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:06.257608    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:05:06.258816 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:06.258795    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:05:06.802738 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:06.802707    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:05:06.808103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:06.808081    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:05:06.810441 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:06.810403    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:05:13.813252 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:05:13.813201    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:05:18.782534 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:18.782435    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:05:25.920084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:25.920034    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:05:25.920649 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:25.920100    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:05:29.200666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:29.200627    5508 generic.go:356] "Generic (PLEG): container finished" podID="c98f1701-ad67-41e4-a528-23eb264203f8" containerID="e08da6c9e680148b6d99ba863e2f1e4192dee750278c97bff05dc7522adfd450" exitCode=0
Nov 03 02:05:29.201166 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:29.200702    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" event={"ID":"c98f1701-ad67-41e4-a528-23eb264203f8","Type":"ContainerDied","Data":"e08da6c9e680148b6d99ba863e2f1e4192dee750278c97bff05dc7522adfd450"}
Nov 03 02:05:29.201261 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:29.201244    5508 scope.go:122] "RemoveContainer" containerID="e08da6c9e680148b6d99ba863e2f1e4192dee750278c97bff05dc7522adfd450"
Nov 03 02:05:29.203068 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:29.203037    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:05:29.203178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:29.203108    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:05:29.203178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:29.203142    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:05:30.209419 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:30.209383    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" event={"ID":"c98f1701-ad67-41e4-a528-23eb264203f8","Type":"ContainerStarted","Data":"c225b96e741306753183bb09104726783a5dd3a2cb4e22741ff9f0c8f389e315"}
Nov 03 02:05:30.910775 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:30.910738    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:05:30.911017 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:30.910997    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Nov 03 02:05:30.911066 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:30.911042    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Nov 03 02:05:31.911419 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:31.911336    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Nov 03 02:05:31.911845 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:31.911462    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Nov 03 02:05:32.910940 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:32.910902    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Nov 03 02:05:32.911142 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:32.910965    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Nov 03 02:05:33.701445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.701375    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"]
Nov 03 02:05:33.709216 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.707746    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.743525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.734602    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-podinfo\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.743525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.734731    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f380f36-d73c-4e52-afb0-ef1f167df815-istiod-ca-cert\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.743525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.734778    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-certs\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.743525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.734833    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-token\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.743525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.734871    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-credential-socket\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.743525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.734922    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptm8v\" (UniqueName: \"kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-kube-api-access-ptm8v\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.743525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.734968    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-socket\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.743525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.735042    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-data\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.743525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.735105    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-envoy\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.791866 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.791808    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-434b6a-openshift-default-dockercfg-bz2gf\""
Nov 03 02:05:33.926090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.880138    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-socket\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.926090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.880242    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-data\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.926090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.880306    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-envoy\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.926090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.880404    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-podinfo\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.926090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.880503    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f380f36-d73c-4e52-afb0-ef1f167df815-istiod-ca-cert\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.926090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.880545    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-certs\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.926090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.880594    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-token\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.926090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.880631    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-credential-socket\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.926090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.880685    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptm8v\" (UniqueName: \"kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-kube-api-access-ptm8v\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.926090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.881507    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-socket\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:33.926090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.881775    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-data\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:34.009467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.987951    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Nov 03 02:05:34.009467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.988044    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Nov 03 02:05:34.009467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:33.988876    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-certs\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:34.034603 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.034564    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f380f36-d73c-4e52-afb0-ef1f167df815-istiod-ca-cert\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:34.068659 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.068620    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-credential-socket\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:34.103115 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.103045    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-token\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:34.110609 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.110572    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"]
Nov 03 02:05:34.114695 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.114661    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-envoy\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:34.118050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.117995    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptm8v\" (UniqueName: \"kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-kube-api-access-ptm8v\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:34.119366 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.119301    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-podinfo\") pod \"e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") " pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:34.346537 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.346444    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:34.813329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.813281    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"]
Nov 03 02:05:34.819546 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:05:34.819519    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f380f36_d73c_4e52_afb0_ef1f167df815.slice/crio-b6d6bb06936718ef3a0384fff4bfed3a6a0d49934779d971b6be3fe0efc96a29 WatchSource:0}: Error finding container b6d6bb06936718ef3a0384fff4bfed3a6a0d49934779d971b6be3fe0efc96a29: Status 404 returned error can't find the container with id b6d6bb06936718ef3a0384fff4bfed3a6a0d49934779d971b6be3fe0efc96a29
Nov 03 02:05:34.822548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.822520    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:05:34.822623 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.822585    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:05:34.822623 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.822616    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:05:34.910800 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.910769    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Nov 03 02:05:34.910967 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:34.910835    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Nov 03 02:05:35.278300 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:35.278260    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" event={"ID":"2f380f36-d73c-4e52-afb0-ef1f167df815","Type":"ContainerStarted","Data":"58cabfff660dc8b72cf33bc3d24f132b84591bffc01db2ddf6556770d16fce05"}
Nov 03 02:05:35.278538 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:35.278330    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" event={"ID":"2f380f36-d73c-4e52-afb0-ef1f167df815","Type":"ContainerStarted","Data":"b6d6bb06936718ef3a0384fff4bfed3a6a0d49934779d971b6be3fe0efc96a29"}
Nov 03 02:05:35.347013 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:35.346983    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:35.354615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:35.354482    5508 patch_prober.go:29] interesting pod/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 03 02:05:35.354615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:35.354553    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 03 02:05:35.395410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:35.395328    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" podStartSLOduration=2.395310278 podStartE2EDuration="2.395310278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:35.390970486 +0000 UTC m=+2814.816576965" watchObservedRunningTime="2026-11-03 02:05:35.395310278 +0000 UTC m=+2814.820916745"
Nov 03 02:05:35.914218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:35.914184    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:05:35.914683 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:35.914665    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:05:35.915300 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:35.915281    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:05:36.352281 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:36.352193    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:37.293838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:37.293805    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:37.294822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:37.294804    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:05:42.814524 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:42.814480    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"]
Nov 03 02:05:42.819691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:42.819668    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"
Nov 03 02:05:42.846958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:42.846931    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtfs\" (UniqueName: \"kubernetes.io/projected/c6acc642-397b-45d9-9d62-fae5f01cb830-kube-api-access-cmtfs\") pod \"payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz\" (UID: \"c6acc642-397b-45d9-9d62-fae5f01cb830\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"
Nov 03 02:05:42.847097 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:42.847064    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6acc642-397b-45d9-9d62-fae5f01cb830-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz\" (UID: \"c6acc642-397b-45d9-9d62-fae5f01cb830\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"
Nov 03 02:05:42.862142 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:42.862113    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"]
Nov 03 02:05:42.913256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:42.913213    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-434b6a-dockercfg-wsx5d\""
Nov 03 02:05:42.917323 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:42.914211    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-434b6a\""
Nov 03 02:05:42.961373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:42.951209    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6acc642-397b-45d9-9d62-fae5f01cb830-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz\" (UID: \"c6acc642-397b-45d9-9d62-fae5f01cb830\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"
Nov 03 02:05:42.961373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:42.951450    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtfs\" (UniqueName: \"kubernetes.io/projected/c6acc642-397b-45d9-9d62-fae5f01cb830-kube-api-access-cmtfs\") pod \"payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz\" (UID: \"c6acc642-397b-45d9-9d62-fae5f01cb830\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"
Nov 03 02:05:42.961373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:42.952805    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6acc642-397b-45d9-9d62-fae5f01cb830-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz\" (UID: \"c6acc642-397b-45d9-9d62-fae5f01cb830\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"
Nov 03 02:05:43.021453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.021423    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"]
Nov 03 02:05:43.055740 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.055709    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"]
Nov 03 02:05:43.056083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.056065    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"
Nov 03 02:05:43.080455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.080365    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtfs\" (UniqueName: \"kubernetes.io/projected/c6acc642-397b-45d9-9d62-fae5f01cb830-kube-api-access-cmtfs\") pod \"payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz\" (UID: \"c6acc642-397b-45d9-9d62-fae5f01cb830\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"
Nov 03 02:05:43.133701 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.133663    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"
Nov 03 02:05:43.188406 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.188332    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944f1443-f33d-440c-964a-a600fd22b000-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l\" (UID: \"944f1443-f33d-440c-964a-a600fd22b000\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"
Nov 03 02:05:43.188552 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.188511    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb49c\" (UniqueName: \"kubernetes.io/projected/944f1443-f33d-440c-964a-a600fd22b000-kube-api-access-cb49c\") pod \"payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l\" (UID: \"944f1443-f33d-440c-964a-a600fd22b000\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"
Nov 03 02:05:43.289767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.289733    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb49c\" (UniqueName: \"kubernetes.io/projected/944f1443-f33d-440c-964a-a600fd22b000-kube-api-access-cb49c\") pod \"payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l\" (UID: \"944f1443-f33d-440c-964a-a600fd22b000\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"
Nov 03 02:05:43.289964 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.289891    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944f1443-f33d-440c-964a-a600fd22b000-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l\" (UID: \"944f1443-f33d-440c-964a-a600fd22b000\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"
Nov 03 02:05:43.290602 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.290582    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944f1443-f33d-440c-964a-a600fd22b000-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l\" (UID: \"944f1443-f33d-440c-964a-a600fd22b000\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"
Nov 03 02:05:43.349908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.349829    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb49c\" (UniqueName: \"kubernetes.io/projected/944f1443-f33d-440c-964a-a600fd22b000-kube-api-access-cb49c\") pod \"payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l\" (UID: \"944f1443-f33d-440c-964a-a600fd22b000\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"
Nov 03 02:05:43.401175 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.401144    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"
Nov 03 02:05:43.522074 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.522040    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"]
Nov 03 02:05:43.527988 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:05:43.527957    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6acc642_397b_45d9_9d62_fae5f01cb830.slice/crio-527013d4f6d882a73067041a7d98d7e5097884223a819158cc9ce7648ee31f1e WatchSource:0}: Error finding container 527013d4f6d882a73067041a7d98d7e5097884223a819158cc9ce7648ee31f1e: Status 404 returned error can't find the container with id 527013d4f6d882a73067041a7d98d7e5097884223a819158cc9ce7648ee31f1e
Nov 03 02:05:43.782965 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:43.782929    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"]
Nov 03 02:05:43.789360 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:05:43.789312    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944f1443_f33d_440c_964a_a600fd22b000.slice/crio-9f2dce96f8de5a5685802b9a36f8199211cc486d7d527af66da3ced6ceb11bfb WatchSource:0}: Error finding container 9f2dce96f8de5a5685802b9a36f8199211cc486d7d527af66da3ced6ceb11bfb: Status 404 returned error can't find the container with id 9f2dce96f8de5a5685802b9a36f8199211cc486d7d527af66da3ced6ceb11bfb
Nov 03 02:05:43.817721 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:05:43.817688    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:05:44.350038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:44.350000    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l" event={"ID":"944f1443-f33d-440c-964a-a600fd22b000","Type":"ContainerStarted","Data":"df1afc219e21f8e30420e9356a6c060b2bc7a392eaa16b7b65a16ab5a609dd6c"}
Nov 03 02:05:44.350253 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:44.350066    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l" event={"ID":"944f1443-f33d-440c-964a-a600fd22b000","Type":"ContainerStarted","Data":"9f2dce96f8de5a5685802b9a36f8199211cc486d7d527af66da3ced6ceb11bfb"}
Nov 03 02:05:44.350253 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:44.350097    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"
Nov 03 02:05:44.352308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:44.352280    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz" event={"ID":"c6acc642-397b-45d9-9d62-fae5f01cb830","Type":"ContainerStarted","Data":"4e5e1dc2dc9dfe56da59b0ec2760eb9b9cbd3a47f0e86e21c1ffa19d85aa0796"}
Nov 03 02:05:44.352461 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:44.352330    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz" event={"ID":"c6acc642-397b-45d9-9d62-fae5f01cb830","Type":"ContainerStarted","Data":"527013d4f6d882a73067041a7d98d7e5097884223a819158cc9ce7648ee31f1e"}
Nov 03 02:05:44.352461 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:44.352384    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"
Nov 03 02:05:44.462988 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:44.462939    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l" podStartSLOduration=2.462924576 podStartE2EDuration="2.462924576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:44.458504591 +0000 UTC m=+2823.884111057" watchObservedRunningTime="2026-11-03 02:05:44.462924576 +0000 UTC m=+2823.888531053"
Nov 03 02:05:44.567440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:44.567382    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz" podStartSLOduration=2.567338253 podStartE2EDuration="2.567338253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:05:44.566076595 +0000 UTC m=+2823.991683077" watchObservedRunningTime="2026-11-03 02:05:44.567338253 +0000 UTC m=+2823.992944721"
Nov 03 02:05:45.083864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:45.083830    5508 scope.go:122] "RemoveContainer" containerID="d70fe736a58b6425a00c75b0225e5541d4fb656edcccb0689a29838a6df20045"
Nov 03 02:05:46.447509 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:46.447468    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:05:55.365218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:55.365187    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"
Nov 03 02:05:55.365831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:55.365279    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"
Nov 03 02:05:56.229872 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:56.229835    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": read tcp 10.133.0.2:58806->10.133.1.4:15021: read: connection reset by peer" start-of-body=
Nov 03 02:05:56.230019 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:56.229924    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": read tcp 10.133.0.2:58806->10.133.1.4:15021: read: connection reset by peer"
Nov 03 02:05:56.540675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:56.540588    5508 generic.go:356] "Generic (PLEG): container finished" podID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerID="6f9451f1166b46677e76f7963db18b3894b036b3f8c8020265697d8937d4f7bf" exitCode=0
Nov 03 02:05:56.540675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:56.540659    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" event={"ID":"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053","Type":"ContainerDied","Data":"6f9451f1166b46677e76f7963db18b3894b036b3f8c8020265697d8937d4f7bf"}
Nov 03 02:05:56.541107 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:56.541083    5508 scope.go:122] "RemoveContainer" containerID="6f9451f1166b46677e76f7963db18b3894b036b3f8c8020265697d8937d4f7bf"
Nov 03 02:05:56.542728 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:56.542704    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:05:56.542805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:56.542767    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:05:56.542805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:56.542795    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:05:57.579878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:57.579847    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" event={"ID":"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053","Type":"ContainerStarted","Data":"f743c1dee3a59b38f511f65a798b1f62e9f00306512cfebbb18cc8c4f197b335"}
Nov 03 02:05:57.847835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:57.847756    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:05:57.848017 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:57.847918    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 03 02:05:57.848017 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:57.847970    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 03 02:05:58.832174 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:58.832139    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"]
Nov 03 02:05:58.860216 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:58.859720    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 03 02:05:58.860216 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:58.859802    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 03 02:05:58.869675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:58.869642    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"
Nov 03 02:05:58.897024 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:58.896989    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"]
Nov 03 02:05:58.928876 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:58.928844    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-1c9b4e\""
Nov 03 02:05:58.929948 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:58.929922    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-1c9b4e-dockercfg-mh6qd\""
Nov 03 02:05:59.143968 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.143871    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2ljl\" (UniqueName: \"kubernetes.io/projected/7aaaad46-1501-477a-b1e4-7cd7d758e92f-kube-api-access-r2ljl\") pod \"payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr\" (UID: \"7aaaad46-1501-477a-b1e4-7cd7d758e92f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"
Nov 03 02:05:59.172675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.172634    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaaad46-1501-477a-b1e4-7cd7d758e92f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr\" (UID: \"7aaaad46-1501-477a-b1e4-7cd7d758e92f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"
Nov 03 02:05:59.173200 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.156417    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"]
Nov 03 02:05:59.199875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.199843    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"]
Nov 03 02:05:59.200174 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.200159    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"
Nov 03 02:05:59.278615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.278481    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhxxd\" (UniqueName: \"kubernetes.io/projected/911e2aa1-8c98-428d-91fc-988b3131d2de-kube-api-access-fhxxd\") pod \"payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb\" (UID: \"911e2aa1-8c98-428d-91fc-988b3131d2de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"
Nov 03 02:05:59.278853 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.278647    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911e2aa1-8c98-428d-91fc-988b3131d2de-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb\" (UID: \"911e2aa1-8c98-428d-91fc-988b3131d2de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"
Nov 03 02:05:59.278853 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.278715    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2ljl\" (UniqueName: \"kubernetes.io/projected/7aaaad46-1501-477a-b1e4-7cd7d758e92f-kube-api-access-r2ljl\") pod \"payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr\" (UID: \"7aaaad46-1501-477a-b1e4-7cd7d758e92f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"
Nov 03 02:05:59.278999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.278926    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaaad46-1501-477a-b1e4-7cd7d758e92f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr\" (UID: \"7aaaad46-1501-477a-b1e4-7cd7d758e92f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"
Nov 03 02:05:59.279887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.279863    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaaad46-1501-477a-b1e4-7cd7d758e92f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr\" (UID: \"7aaaad46-1501-477a-b1e4-7cd7d758e92f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"
Nov 03 02:05:59.336991 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.336956    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2ljl\" (UniqueName: \"kubernetes.io/projected/7aaaad46-1501-477a-b1e4-7cd7d758e92f-kube-api-access-r2ljl\") pod \"payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr\" (UID: \"7aaaad46-1501-477a-b1e4-7cd7d758e92f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"
Nov 03 02:05:59.380086 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.380041    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhxxd\" (UniqueName: \"kubernetes.io/projected/911e2aa1-8c98-428d-91fc-988b3131d2de-kube-api-access-fhxxd\") pod \"payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb\" (UID: \"911e2aa1-8c98-428d-91fc-988b3131d2de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"
Nov 03 02:05:59.380086 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.380094    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911e2aa1-8c98-428d-91fc-988b3131d2de-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb\" (UID: \"911e2aa1-8c98-428d-91fc-988b3131d2de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"
Nov 03 02:05:59.380860 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.380840    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911e2aa1-8c98-428d-91fc-988b3131d2de-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb\" (UID: \"911e2aa1-8c98-428d-91fc-988b3131d2de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"
Nov 03 02:05:59.437810 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.437773    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhxxd\" (UniqueName: \"kubernetes.io/projected/911e2aa1-8c98-428d-91fc-988b3131d2de-kube-api-access-fhxxd\") pod \"payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb\" (UID: \"911e2aa1-8c98-428d-91fc-988b3131d2de\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"
Nov 03 02:05:59.516548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.516516    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"
Nov 03 02:05:59.557371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.556706    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"
Nov 03 02:05:59.847499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.847411    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 03 02:05:59.847499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.847471    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 03 02:05:59.919275 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:05:59.919236    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"]
Nov 03 02:05:59.924372 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:05:59.924322    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911e2aa1_8c98_428d_91fc_988b3131d2de.slice/crio-2c183cc2530e8a23aa06f8022ebc03e7baba6bb44b89f99311a1e450166b5159 WatchSource:0}: Error finding container 2c183cc2530e8a23aa06f8022ebc03e7baba6bb44b89f99311a1e450166b5159: Status 404 returned error can't find the container with id 2c183cc2530e8a23aa06f8022ebc03e7baba6bb44b89f99311a1e450166b5159
Nov 03 02:06:00.037260 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:00.037219    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"]
Nov 03 02:06:00.047273 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:06:00.047245    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aaaad46_1501_477a_b1e4_7cd7d758e92f.slice/crio-5515dbbca6daec7e87b1d42cfa67215f1398bd19b08e2940ccfb46da032226b8 WatchSource:0}: Error finding container 5515dbbca6daec7e87b1d42cfa67215f1398bd19b08e2940ccfb46da032226b8: Status 404 returned error can't find the container with id 5515dbbca6daec7e87b1d42cfa67215f1398bd19b08e2940ccfb46da032226b8
Nov 03 02:06:00.792251 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:00.792210    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr" event={"ID":"7aaaad46-1501-477a-b1e4-7cd7d758e92f","Type":"ContainerStarted","Data":"f04faaf51f2fa73a00aa18355df45012404e558b4da28a65a29810540c3f1eff"}
Nov 03 02:06:00.792493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:00.792277    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr" event={"ID":"7aaaad46-1501-477a-b1e4-7cd7d758e92f","Type":"ContainerStarted","Data":"5515dbbca6daec7e87b1d42cfa67215f1398bd19b08e2940ccfb46da032226b8"}
Nov 03 02:06:00.792493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:00.792338    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"
Nov 03 02:06:00.794027 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:00.794002    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb" event={"ID":"911e2aa1-8c98-428d-91fc-988b3131d2de","Type":"ContainerStarted","Data":"f27879941a0fdeaa028401a48408702c0955045f988c0f2e6cb037605555dfb1"}
Nov 03 02:06:00.794152 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:00.794056    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"
Nov 03 02:06:00.794152 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:00.794081    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb" event={"ID":"911e2aa1-8c98-428d-91fc-988b3131d2de","Type":"ContainerStarted","Data":"2c183cc2530e8a23aa06f8022ebc03e7baba6bb44b89f99311a1e450166b5159"}
Nov 03 02:06:00.847566 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:00.847530    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 03 02:06:00.847971 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:00.847585    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 03 02:06:00.901301 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:00.901231    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr" podStartSLOduration=2.901216212 podStartE2EDuration="2.901216212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:00.90032458 +0000 UTC m=+2840.325931075" watchObservedRunningTime="2026-11-03 02:06:00.901216212 +0000 UTC m=+2840.326822676"
Nov 03 02:06:01.847483 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:01.847444    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 03 02:06:01.847672 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:01.847521    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 03 02:06:02.850823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:02.850794    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:06:02.851284 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:02.851149    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:06:02.851734 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:02.851716    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:06:03.011217 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:03.011171    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb" podStartSLOduration=5.011158735 podStartE2EDuration="5.011158735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:01.032844756 +0000 UTC m=+2840.458451222" watchObservedRunningTime="2026-11-03 02:06:03.011158735 +0000 UTC m=+2842.436765200"
Nov 03 02:06:11.804860 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:11.804830    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"
Nov 03 02:06:11.805278 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:11.804893    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"
Nov 03 02:06:13.819977 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:06:13.819943    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:06:18.936850 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:18.936813    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"]
Nov 03 02:06:18.941886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:18.941858    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"
Nov 03 02:06:18.972301 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:18.972271    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83bb0867-e955-4327-9790-2a78ab123f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86\" (UID: \"83bb0867-e955-4327-9790-2a78ab123f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"
Nov 03 02:06:18.972604 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:18.972580    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq9fh\" (UniqueName: \"kubernetes.io/projected/83bb0867-e955-4327-9790-2a78ab123f6e-kube-api-access-gq9fh\") pod \"payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86\" (UID: \"83bb0867-e955-4327-9790-2a78ab123f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"
Nov 03 02:06:19.006721 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:19.006686    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-d9576a\""
Nov 03 02:06:19.065371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:19.064189    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-d9576a-dockercfg-k8xlv\""
Nov 03 02:06:19.092528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:19.092481    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq9fh\" (UniqueName: \"kubernetes.io/projected/83bb0867-e955-4327-9790-2a78ab123f6e-kube-api-access-gq9fh\") pod \"payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86\" (UID: \"83bb0867-e955-4327-9790-2a78ab123f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"
Nov 03 02:06:19.112382 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:19.107535    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83bb0867-e955-4327-9790-2a78ab123f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86\" (UID: \"83bb0867-e955-4327-9790-2a78ab123f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"
Nov 03 02:06:19.112382 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:19.108644    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83bb0867-e955-4327-9790-2a78ab123f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86\" (UID: \"83bb0867-e955-4327-9790-2a78ab123f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"
Nov 03 02:06:19.152507 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:19.150811    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"]
Nov 03 02:06:19.170180 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:19.170091    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq9fh\" (UniqueName: \"kubernetes.io/projected/83bb0867-e955-4327-9790-2a78ab123f6e-kube-api-access-gq9fh\") pod \"payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86\" (UID: \"83bb0867-e955-4327-9790-2a78ab123f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"
Nov 03 02:06:19.270749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:19.270671    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"
Nov 03 02:06:19.652768 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:19.652741    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"]
Nov 03 02:06:19.656102 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:06:19.656078    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bb0867_e955_4327_9790_2a78ab123f6e.slice/crio-9cf180d74f1e7706a15242a77ed5aa0fda88c90c131c7abe24ec3a85dc128db0 WatchSource:0}: Error finding container 9cf180d74f1e7706a15242a77ed5aa0fda88c90c131c7abe24ec3a85dc128db0: Status 404 returned error can't find the container with id 9cf180d74f1e7706a15242a77ed5aa0fda88c90c131c7abe24ec3a85dc128db0
Nov 03 02:06:20.069609 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:20.069518    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86" event={"ID":"83bb0867-e955-4327-9790-2a78ab123f6e","Type":"ContainerStarted","Data":"a94929f9c4a585a45a7d2a79d6a03edc0127b946b5f76e1c38642b86915b5e1c"}
Nov 03 02:06:20.069609 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:20.069574    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86" event={"ID":"83bb0867-e955-4327-9790-2a78ab123f6e","Type":"ContainerStarted","Data":"9cf180d74f1e7706a15242a77ed5aa0fda88c90c131c7abe24ec3a85dc128db0"}
Nov 03 02:06:20.069609 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:20.069601    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"
Nov 03 02:06:20.182032 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:20.181982    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86" podStartSLOduration=2.181970206 podStartE2EDuration="2.181970206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:06:20.177393804 +0000 UTC m=+2859.603000288" watchObservedRunningTime="2026-11-03 02:06:20.181970206 +0000 UTC m=+2859.607576670"
Nov 03 02:06:28.048129 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:28.048096    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"]
Nov 03 02:06:28.099562 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:28.099520    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c5959f-954z6"]
Nov 03 02:06:28.696802 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:28.696763    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f20122-1247-46aa-88ae-a499568d0969" path="/var/lib/kubelet/pods/75f20122-1247-46aa-88ae-a499568d0969/volumes"
Nov 03 02:06:29.571941 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:29.571903    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:06:31.079791 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:31.079717    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"
Nov 03 02:06:43.813281 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:06:43.813246    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:06:44.066688 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:44.065335    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"]
Nov 03 02:06:44.093822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:44.093796    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb1653a-cnhzk"]
Nov 03 02:06:44.694488 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:44.694454    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced98a04-fd64-4ad5-b27e-4fd75fe326ee" path="/var/lib/kubelet/pods/ced98a04-fd64-4ad5-b27e-4fd75fe326ee/volumes"
Nov 03 02:06:45.112408 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:45.112301    5508 scope.go:122] "RemoveContainer" containerID="eff5a106f93938e332da14618c2430132630418fd4cb904e7101baa6fd9b33a0"
Nov 03 02:06:59.047506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:59.047432    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"]
Nov 03 02:06:59.092853 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:06:59.092827    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39160055-zgs7f"]
Nov 03 02:07:00.047433 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:00.047393    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"]
Nov 03 02:07:00.093584 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:00.093553    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdcd9448-4lv4x"]
Nov 03 02:07:00.695473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:00.695443    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83d97342-b645-4a63-8ca8-378c099e98e6" path="/var/lib/kubelet/pods/83d97342-b645-4a63-8ca8-378c099e98e6/volumes"
Nov 03 02:07:00.695920 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:00.695907    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9fb8b31-e75e-4e28-affa-303892a342a7" path="/var/lib/kubelet/pods/c9fb8b31-e75e-4e28-affa-303892a342a7/volumes"
Nov 03 02:07:02.751830 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:02.751797    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"]
Nov 03 02:07:02.759026 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:02.759002    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:02.813380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:02.809653    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-1c9b4e-serving-cert\""
Nov 03 02:07:02.830258 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:02.830221    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"]
Nov 03 02:07:02.984886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:02.982447    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:02.984886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:02.982572    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skw6w\" (UniqueName: \"kubernetes.io/projected/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-kube-api-access-skw6w\") pod \"maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:02.984886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:02.982605    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:02.984886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:02.982780    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:03.092211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:03.092124    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skw6w\" (UniqueName: \"kubernetes.io/projected/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-kube-api-access-skw6w\") pod \"maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:03.092211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:03.092166    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:03.092509 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:03.092308    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:03.092509 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:03.092381    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:03.093931 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:03.093899    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:03.095793 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:03.095769    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:03.095934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:03.095802    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:03.098708 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:03.098689    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:07:03.152404 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:03.152369    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skw6w\" (UniqueName: \"kubernetes.io/projected/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-kube-api-access-skw6w\") pod \"maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:03.375557 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:03.375466    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:03.767558 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:03.767529    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"]
Nov 03 02:07:03.772071 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:07:03.772042    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6cdc5d_e308_4560_96fb_8a1fa621c19c.slice/crio-2dd7a6f5810c5e57c8e96301c60465ab1f1bd5f5da72c9853f6f2b4c3468da2f WatchSource:0}: Error finding container 2dd7a6f5810c5e57c8e96301c60465ab1f1bd5f5da72c9853f6f2b4c3468da2f: Status 404 returned error can't find the container with id 2dd7a6f5810c5e57c8e96301c60465ab1f1bd5f5da72c9853f6f2b4c3468da2f
Nov 03 02:07:03.773058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:03.773039    5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:07:04.494493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:04.494442    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9" event={"ID":"fa6cdc5d-e308-4560-96fb-8a1fa621c19c","Type":"ContainerStarted","Data":"2dd7a6f5810c5e57c8e96301c60465ab1f1bd5f5da72c9853f6f2b4c3468da2f"}
Nov 03 02:07:05.503839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:05.503806    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9" event={"ID":"fa6cdc5d-e308-4560-96fb-8a1fa621c19c","Type":"ContainerStarted","Data":"5bb60a6fcff120bd72280ccadea7d9b4816bc1423faa202a738d37ff2fd02cc0"}
Nov 03 02:07:05.504252 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:05.503873    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:05.632769 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:05.632720    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9" podStartSLOduration=2.802090928 podStartE2EDuration="3.632703975s" totalImagesPullingTime="830.613047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:07:05.631479169 +0000 UTC m=+2905.057085653" watchObservedRunningTime="2026-11-03 02:07:05.632703975 +0000 UTC m=+2905.058310440"
Nov 03 02:07:09.067373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:09.061304    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"]
Nov 03 02:07:09.094744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:09.094712    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbdb56cc-zhwqw"]
Nov 03 02:07:10.695535 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:10.695504    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb8b73c-2f1d-4c1a-8eea-e9f04491879b" path="/var/lib/kubelet/pods/dbb8b73c-2f1d-4c1a-8eea-e9f04491879b/volumes"
Nov 03 02:07:11.575627 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:11.575597    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:07:13.630363 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:13.630251    5508 generic.go:356] "Generic (PLEG): container finished" podID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerID="58cabfff660dc8b72cf33bc3d24f132b84591bffc01db2ddf6556770d16fce05" exitCode=0
Nov 03 02:07:13.630776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:13.630325    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" event={"ID":"2f380f36-d73c-4e52-afb0-ef1f167df815","Type":"ContainerDied","Data":"58cabfff660dc8b72cf33bc3d24f132b84591bffc01db2ddf6556770d16fce05"}
Nov 03 02:07:13.630776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:13.630754    5508 scope.go:122] "RemoveContainer" containerID="58cabfff660dc8b72cf33bc3d24f132b84591bffc01db2ddf6556770d16fce05"
Nov 03 02:07:13.632268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:13.632238    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:07:13.632406 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:13.632339    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:07:13.632485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:13.632405    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:07:13.814833 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:07:13.814794    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:07:14.640708 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:14.640674    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" event={"ID":"2f380f36-d73c-4e52-afb0-ef1f167df815","Type":"ContainerStarted","Data":"6178c5ca607ab2310a61808adecee43bd36fb78c719f7f83d1e270902160ff02"}
Nov 03 02:07:15.346735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:15.346705    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:07:15.346929 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:15.346879    5508 patch_prober.go:29] interesting pod/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 03 02:07:15.346994 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:15.346921    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 03 02:07:16.047206 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:16.047173    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"]
Nov 03 02:07:16.092822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:16.092792    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bdf4d8c-7jqx5"]
Nov 03 02:07:16.347262 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:16.347169    5508 patch_prober.go:29] interesting pod/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 03 02:07:16.347262 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:16.347240    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 03 02:07:16.693957 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:16.693915    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f38340d-e9cd-4d78-8881-cf97e97fe559" path="/var/lib/kubelet/pods/3f38340d-e9cd-4d78-8881-cf97e97fe559/volumes"
Nov 03 02:07:17.347161 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:17.347126    5508 patch_prober.go:29] interesting pod/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 03 02:07:17.347603 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:17.347187    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 03 02:07:18.347397 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:18.347342    5508 patch_prober.go:29] interesting pod/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 03 02:07:18.347886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:18.347418    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 03 02:07:19.347178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:19.347142    5508 patch_prober.go:29] interesting pod/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Nov 03 02:07:19.347397 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:19.347191    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Nov 03 02:07:20.351671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:20.351636    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:07:20.352193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:20.352071    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:07:20.352835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:20.352813    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:07:26.053536 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.053500    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"]
Nov 03 02:07:26.099654 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.099621    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b281df-9blgh"]
Nov 03 02:07:26.671502 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.671460    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"]
Nov 03 02:07:26.677203 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.677183    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:26.692634 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.692612    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mwbw\" (UniqueName: \"kubernetes.io/projected/bed779ff-4b38-4116-a5bd-1e07b2e28441-kube-api-access-8mwbw\") pod \"maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:26.692744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.692655    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:26.692744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.692681    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bed779ff-4b38-4116-a5bd-1e07b2e28441-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:26.692868 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.692850    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:26.693522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.693503    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97fb68aa-4dba-43d3-9eda-d0262926d4e9" path="/var/lib/kubelet/pods/97fb68aa-4dba-43d3-9eda-d0262926d4e9/volumes"
Nov 03 02:07:26.725946 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.725922    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-d9576a-serving-cert\""
Nov 03 02:07:26.777963 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.777917    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"]
Nov 03 02:07:26.793872 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.793841    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mwbw\" (UniqueName: \"kubernetes.io/projected/bed779ff-4b38-4116-a5bd-1e07b2e28441-kube-api-access-8mwbw\") pod \"maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:26.794057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.793887    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:26.794057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.793908    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bed779ff-4b38-4116-a5bd-1e07b2e28441-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:26.794057 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:07:26.793994    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-serving-cert: secret "maas-api-e2e-shared-b-w4-d9576a-serving-cert" not found
Nov 03 02:07:26.794057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.794023    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:26.794303 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:07:26.794065    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-maas-api-tls podName:bed779ff-4b38-4116-a5bd-1e07b2e28441 nodeName:}" failed. No retries permitted until 2026-11-03 02:07:27.294049554 +0000 UTC m=+2926.719655997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-maas-api-tls") pod "maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t" (UID: "bed779ff-4b38-4116-a5bd-1e07b2e28441") : secret "maas-api-e2e-shared-b-w4-d9576a-serving-cert" not found
Nov 03 02:07:26.794723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.794698    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bed779ff-4b38-4116-a5bd-1e07b2e28441-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:26.796756 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.796730    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:26.851852 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:26.851815    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mwbw\" (UniqueName: \"kubernetes.io/projected/bed779ff-4b38-4116-a5bd-1e07b2e28441-kube-api-access-8mwbw\") pod \"maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:27.300881 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:27.300843    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:27.303929 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:27.303905    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:27.592893 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:27.592808    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:27.974715 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:27.974685    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"]
Nov 03 02:07:27.980506 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:07:27.980476    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed779ff_4b38_4116_a5bd_1e07b2e28441.slice/crio-ac5486cab1a6c1426ce61651cd2d0dcc69cf0f414d7510cb8884ba94ae466539 WatchSource:0}: Error finding container ac5486cab1a6c1426ce61651cd2d0dcc69cf0f414d7510cb8884ba94ae466539: Status 404 returned error can't find the container with id ac5486cab1a6c1426ce61651cd2d0dcc69cf0f414d7510cb8884ba94ae466539
Nov 03 02:07:28.852786 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:28.852750    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t" event={"ID":"bed779ff-4b38-4116-a5bd-1e07b2e28441","Type":"ContainerStarted","Data":"ac5486cab1a6c1426ce61651cd2d0dcc69cf0f414d7510cb8884ba94ae466539"}
Nov 03 02:07:29.862398 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:29.862327    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t" event={"ID":"bed779ff-4b38-4116-a5bd-1e07b2e28441","Type":"ContainerStarted","Data":"8495fbe805ca8b15c61bc0c510295eeae47c4301570620754afa887ae34a9fa6"}
Nov 03 02:07:29.862812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:29.862421    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:29.977208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:29.977156    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t" podStartSLOduration=3.104296153 podStartE2EDuration="3.977142988s" totalImagesPullingTime="872.846835ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:07:29.972372936 +0000 UTC m=+2929.397979394" watchObservedRunningTime="2026-11-03 02:07:29.977142988 +0000 UTC m=+2929.402749452"
Nov 03 02:07:32.048527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:32.048492    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"]
Nov 03 02:07:32.094238 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:32.094205    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a57d3de2-lvzzb"]
Nov 03 02:07:32.693824 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:32.693794    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13" path="/var/lib/kubelet/pods/5cff4f0a-4a78-420a-91e2-a9cb9dd7fb13/volumes"
Nov 03 02:07:35.047025 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:35.046987    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"]
Nov 03 02:07:35.092488 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:35.092454    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af96121-wcvcb"]
Nov 03 02:07:35.876685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:35.876658    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:07:36.694306 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:36.694270    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90e57e5e-1860-4edd-8e96-77f2e4140313" path="/var/lib/kubelet/pods/90e57e5e-1860-4edd-8e96-77f2e4140313/volumes"
Nov 03 02:07:37.052321 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:37.052251    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"]
Nov 03 02:07:37.100643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:37.100612    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b49194-nxs82"]
Nov 03 02:07:38.694266 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:38.694222    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e26672c2-1d84-4680-92c2-342140350153" path="/var/lib/kubelet/pods/e26672c2-1d84-4680-92c2-342140350153/volumes"
Nov 03 02:07:43.557213 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:43.557172    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:07:43.813854 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:07:43.813759    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:07:45.141208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:45.141177    5508 scope.go:122] "RemoveContainer" containerID="b1421ec386572f391a89498f0116020cf7dbc9eae8438dd5b1c124ac0fc2b4a1"
Nov 03 02:07:45.168659 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:45.168633    5508 scope.go:122] "RemoveContainer" containerID="28d08abb496cbd445b9a809f54fabddf0fb716ffc9ab4bf91152a9658be7158c"
Nov 03 02:07:45.180041 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:45.180023    5508 scope.go:122] "RemoveContainer" containerID="9e9132afdc94da8616e37c1eb3d95859cf72dd9037549bddc4cf60f3f010c503"
Nov 03 02:07:51.049279 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:51.049230    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"]
Nov 03 02:07:51.094809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:51.094772    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad60eed-jthg8"]
Nov 03 02:07:52.698430 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:52.698389    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da1775f2-b42d-4791-80ab-0ed77ac8acf3" path="/var/lib/kubelet/pods/da1775f2-b42d-4791-80ab-0ed77ac8acf3/volumes"
Nov 03 02:07:56.046124 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:56.046081    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"]
Nov 03 02:07:56.094675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:56.094639    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-931404d5-4v24x"]
Nov 03 02:07:56.693197 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:07:56.693168    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd52e2c-f248-4dc9-b311-6948ffbb71f4" path="/var/lib/kubelet/pods/2cd52e2c-f248-4dc9-b311-6948ffbb71f4/volumes"
Nov 03 02:08:06.048279 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:06.048240    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"]
Nov 03 02:08:06.052030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:06.051996    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-438d237c-2cv8r"]
Nov 03 02:08:06.693594 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:06.693563    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c" path="/var/lib/kubelet/pods/7a0e8437-1ab7-4902-b2df-d1dcae2b0b3c/volumes"
Nov 03 02:08:12.048080 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:12.048039    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"]
Nov 03 02:08:12.094539 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:12.094508    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84dc44ff-fgw7l"]
Nov 03 02:08:12.694526 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:12.694493    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d016891-f4ed-4759-9c09-e0e139b60aaa" path="/var/lib/kubelet/pods/3d016891-f4ed-4759-9c09-e0e139b60aaa/volumes"
Nov 03 02:08:13.813863 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:08:13.813819    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:08:14.374074 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:14.374036    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:08:21.046472 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:21.046381    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"]
Nov 03 02:08:21.092967 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:21.092938    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"]
Nov 03 02:08:21.096434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:21.096415    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8e0f312-v8r9q"]
Nov 03 02:08:21.099553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:21.099531    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ce07762-4nnp6"]
Nov 03 02:08:22.694880 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:22.694848    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f50a3ae-3ee5-4c4f-8980-c030d3ec006e" path="/var/lib/kubelet/pods/3f50a3ae-3ee5-4c4f-8980-c030d3ec006e/volumes"
Nov 03 02:08:22.695307 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:22.695293    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58b20a14-5b1c-408d-9298-7ca5cefa428a" path="/var/lib/kubelet/pods/58b20a14-5b1c-408d-9298-7ca5cefa428a/volumes"
Nov 03 02:08:28.046980 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:28.046945    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"]
Nov 03 02:08:28.093340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:28.093295    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41128b9f-jpsfd"]
Nov 03 02:08:28.695533 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:28.695502    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e013773-348f-48d7-806b-eef735149093" path="/var/lib/kubelet/pods/4e013773-348f-48d7-806b-eef735149093/volumes"
Nov 03 02:08:35.045692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:35.045656    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"]
Nov 03 02:08:35.091749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:35.091719    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-710b85d6-cnds6"]
Nov 03 02:08:36.047727 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:36.047696    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"]
Nov 03 02:08:36.094009 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:36.093974    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a324ea-mcrdn"]
Nov 03 02:08:36.693807 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:36.693778    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f004de45-ebc3-4678-8311-ab50a7cbaa9e" path="/var/lib/kubelet/pods/f004de45-ebc3-4678-8311-ab50a7cbaa9e/volumes"
Nov 03 02:08:36.694288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:36.694273    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29d7f6d-e5b0-42b7-afe5-06baa89f1a35" path="/var/lib/kubelet/pods/f29d7f6d-e5b0-42b7-afe5-06baa89f1a35/volumes"
Nov 03 02:08:41.295109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:41.295075    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:08:41.354517 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:41.354494    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:08:43.045739 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:43.045705    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"]
Nov 03 02:08:43.091356 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:43.091324    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a17fde-td9rf"]
Nov 03 02:08:43.814337 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:08:43.814303    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:08:44.694549 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:44.694515    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9c885c-06e4-4367-8a68-a68481eb857e" path="/var/lib/kubelet/pods/af9c885c-06e4-4367-8a68-a68481eb857e/volumes"
Nov 03 02:08:45.218581 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:45.218548    5508 scope.go:122] "RemoveContainer" containerID="c0256ff1851cc9c3b5d571ac6e3aea3782165d83b16d1c052f660df8b939bb5f"
Nov 03 02:08:45.232886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:45.232857    5508 scope.go:122] "RemoveContainer" containerID="bc57ece718d325b6fdbf4c01f310fc05b33610c57216ca8588f57fc970df5e88"
Nov 03 02:08:45.243080 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:45.243059    5508 scope.go:122] "RemoveContainer" containerID="0132d5a8263ede036f486ea2cc58ab64ae8ee19bc44fb5d762164c690d4fb0c9"
Nov 03 02:08:45.253921 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:45.253900    5508 scope.go:122] "RemoveContainer" containerID="c8fbf3f64496943bb83ba2358f9e9ac6e7ecc6ea42a1a5bf16627b7cb46da0ee"
Nov 03 02:08:45.263942 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:45.263921    5508 scope.go:122] "RemoveContainer" containerID="54937a97b19e4eabe1a8b10af99177807ec3c64368319250151d1edca20c8f1b"
Nov 03 02:08:45.274377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:45.274336    5508 scope.go:122] "RemoveContainer" containerID="3a71fce94a3ef29f34c520ed01c4d80ee89019f3123729575d0ca191a596ec4e"
Nov 03 02:08:45.284784 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:45.284761    5508 scope.go:122] "RemoveContainer" containerID="e6e8c3e2c4aa66e4c76419c6c36da947acede60eb8243f93e59bb0a51e5959f3"
Nov 03 02:08:45.293780 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:45.293757    5508 scope.go:122] "RemoveContainer" containerID="a2c6540eb9bea5c2c2c0b2ec5d43fcc23d3fa6b3a16e3bed27ee7cbe81deb815"
Nov 03 02:08:47.046457 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:47.046427    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"]
Nov 03 02:08:47.092478 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:47.092443    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76935266-gndqc"]
Nov 03 02:08:48.695151 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:48.695119    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad473b1-5cef-4668-b38a-145e2d7af270" path="/var/lib/kubelet/pods/3ad473b1-5cef-4668-b38a-145e2d7af270/volumes"
Nov 03 02:08:50.715426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:08:50.715393    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:09:00.051839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:00.051803    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"]
Nov 03 02:09:00.098069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:00.098038    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37301876-gt5fq"]
Nov 03 02:09:00.693299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:00.693269    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa7ff01-4dcf-4b94-8480-bc2274ef9851" path="/var/lib/kubelet/pods/7fa7ff01-4dcf-4b94-8480-bc2274ef9851/volumes"
Nov 03 02:09:00.811064 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:00.811033    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"]
Nov 03 02:09:00.817413 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:00.817391    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"
Nov 03 02:09:00.866482 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:00.866447    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:09:00.912644 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:00.912611    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:09:00.917335 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:00.917307    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"]
Nov 03 02:09:01.006464 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:01.006389    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac4c77c7-dnw9b\" (UID: \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"
Nov 03 02:09:01.006671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:01.006650    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kltt6\" (UniqueName: \"kubernetes.io/projected/2a9ea631-7c13-46c8-831a-89e4ac4c6776-kube-api-access-kltt6\") pod \"maas-api-revoke-sub-ac4c77c7-dnw9b\" (UID: \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"
Nov 03 02:09:01.006756 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:01.006741    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac4c77c7-dnw9b\" (UID: \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"
Nov 03 02:09:01.107689 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:01.107655    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kltt6\" (UniqueName: \"kubernetes.io/projected/2a9ea631-7c13-46c8-831a-89e4ac4c6776-kube-api-access-kltt6\") pod \"maas-api-revoke-sub-ac4c77c7-dnw9b\" (UID: \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"
Nov 03 02:09:01.108091 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:01.107716    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac4c77c7-dnw9b\" (UID: \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"
Nov 03 02:09:01.108091 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:01.107749    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac4c77c7-dnw9b\" (UID: \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"
Nov 03 02:09:01.108430 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:01.108413    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac4c77c7-dnw9b\" (UID: \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"
Nov 03 02:09:01.110157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:01.110135    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac4c77c7-dnw9b\" (UID: \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"
Nov 03 02:09:01.165380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:01.165321    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kltt6\" (UniqueName: \"kubernetes.io/projected/2a9ea631-7c13-46c8-831a-89e4ac4c6776-kube-api-access-kltt6\") pod \"maas-api-revoke-sub-ac4c77c7-dnw9b\" (UID: \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"
Nov 03 02:09:01.249520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:01.249489    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"]
Nov 03 02:09:01.349332 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:01.349240    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"]
Nov 03 02:09:01.431505 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:01.431462    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"
Nov 03 02:09:01.825425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:01.825388    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"]
Nov 03 02:09:01.831837 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:09:01.831811    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9ea631_7c13_46c8_831a_89e4ac4c6776.slice/crio-b0f3adbc6066468d89713d5e6087a57cd38818f67a6ea84d4a202d3c459e592f WatchSource:0}: Error finding container b0f3adbc6066468d89713d5e6087a57cd38818f67a6ea84d4a202d3c459e592f: Status 404 returned error can't find the container with id b0f3adbc6066468d89713d5e6087a57cd38818f67a6ea84d4a202d3c459e592f
Nov 03 02:09:02.793272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:02.793112    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b" event={"ID":"2a9ea631-7c13-46c8-831a-89e4ac4c6776","Type":"ContainerStarted","Data":"d465401c71cfa9e50adad42195c7d8891a066c0a186c4d9f0f887446ed40f1d3"}
Nov 03 02:09:02.810443 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:02.810399    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b" event={"ID":"2a9ea631-7c13-46c8-831a-89e4ac4c6776","Type":"ContainerStarted","Data":"b0f3adbc6066468d89713d5e6087a57cd38818f67a6ea84d4a202d3c459e592f"}
Nov 03 02:09:02.936223 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:02.918305    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b" podStartSLOduration=2.918283928 podStartE2EDuration="2.918283928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:09:02.915619816 +0000 UTC m=+3022.341226301" watchObservedRunningTime="2026-11-03 02:09:02.918283928 +0000 UTC m=+3022.343890395"
Nov 03 02:09:03.802887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:03.802846    5508 generic.go:356] "Generic (PLEG): container finished" podID="2a9ea631-7c13-46c8-831a-89e4ac4c6776" containerID="d465401c71cfa9e50adad42195c7d8891a066c0a186c4d9f0f887446ed40f1d3" exitCode=0
Nov 03 02:09:03.803299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:03.802886    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b" event={"ID":"2a9ea631-7c13-46c8-831a-89e4ac4c6776","Type":"ContainerDied","Data":"d465401c71cfa9e50adad42195c7d8891a066c0a186c4d9f0f887446ed40f1d3"}
Nov 03 02:09:04.058566 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:04.058472    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"]
Nov 03 02:09:04.062061 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:04.062034    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b62078-fl989"]
Nov 03 02:09:04.693809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:04.693777    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba7ffb5-2c69-4578-8106-24ac424d1469" path="/var/lib/kubelet/pods/8ba7ffb5-2c69-4578-8106-24ac424d1469/volumes"
Nov 03 02:09:04.974550 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:04.974527    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"
Nov 03 02:09:04.980364 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:04.980325    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9ea631-7c13-46c8-831a-89e4ac4c6776-kube-api-access-kltt6\" (UniqueName: \"kubernetes.io/projected/2a9ea631-7c13-46c8-831a-89e4ac4c6776-kube-api-access-kltt6\") pod \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\" (UID: \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\") "
Nov 03 02:09:04.980535 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:04.980519    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-openshift-ca\") pod \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\" (UID: \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\") "
Nov 03 02:09:04.980614 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:04.980597    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-xks-ca\") pod \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\" (UID: \"2a9ea631-7c13-46c8-831a-89e4ac4c6776\") "
Nov 03 02:09:04.980909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:04.980885    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-openshift-ca" pod "2a9ea631-7c13-46c8-831a-89e4ac4c6776" (UID: "2a9ea631-7c13-46c8-831a-89e4ac4c6776"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:09:04.981252 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:04.981230    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:04.984487 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:04.984461    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-xks-ca" pod "2a9ea631-7c13-46c8-831a-89e4ac4c6776" (UID: "2a9ea631-7c13-46c8-831a-89e4ac4c6776"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:09:04.984611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:04.984497    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9ea631-7c13-46c8-831a-89e4ac4c6776-kube-api-access-kltt6" pod "2a9ea631-7c13-46c8-831a-89e4ac4c6776" (UID: "2a9ea631-7c13-46c8-831a-89e4ac4c6776"). InnerVolumeSpecName "kube-api-access-kltt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:09:05.082737 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:05.082659    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kltt6\" (UniqueName: \"kubernetes.io/projected/2a9ea631-7c13-46c8-831a-89e4ac4c6776-kube-api-access-kltt6\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:05.082737 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:05.082693    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a9ea631-7c13-46c8-831a-89e4ac4c6776-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:05.819267 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:05.819240    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"
Nov 03 02:09:05.819267 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:05.819258    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b" event={"ID":"2a9ea631-7c13-46c8-831a-89e4ac4c6776","Type":"ContainerDied","Data":"b0f3adbc6066468d89713d5e6087a57cd38818f67a6ea84d4a202d3c459e592f"}
Nov 03 02:09:05.819489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:05.819304    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0f3adbc6066468d89713d5e6087a57cd38818f67a6ea84d4a202d3c459e592f"
Nov 03 02:09:06.249950 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:06.249890    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng" podUID="a5e3b594-c7a5-4f96-9c13-d9789331f2ef" containerName="payload-processing" containerID="cri-o://a940ccac965c0f4778041b81558bbdd1fbe3eb552e30e5c58a31d0c8437e1930" gracePeriod=25
Nov 03 02:09:06.349736 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:06.349676    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58" podUID="7df245af-91c3-42d2-9a61-bff747cd1701" containerName="payload-pre-processing" containerID="cri-o://ab2d4f4a41029b6a7c006b3ba920858be7759d47d6a9d97b1839dc75f2dbebee" gracePeriod=25
Nov 03 02:09:06.640746 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:06.640669    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng" podUID="a5e3b594-c7a5-4f96-9c13-d9789331f2ef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:09:06.640746 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:06.640687    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58" podUID="7df245af-91c3-42d2-9a61-bff747cd1701" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:09:13.812533 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:09:13.812493    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:09:15.991701 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:15.991667    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"]
Nov 03 02:09:15.992635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:15.991850    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9" podUID="8bbc58e2-b00c-4bc7-ba27-cabe0d546740" containerName="istio-proxy" containerID="cri-o://0329c02ba8a4fddc392fd98bb2c37033e2de7ad8403dd27b07687706870377d2" gracePeriod=30
Nov 03 02:09:16.639753 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:16.639711    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng" podUID="a5e3b594-c7a5-4f96-9c13-d9789331f2ef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:09:16.639753 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:16.639733    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58" podUID="7df245af-91c3-42d2-9a61-bff747cd1701" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:09:21.278021 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.277961    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:09:21.430271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.430240    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-certs\") pod \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") "
Nov 03 02:09:21.430496 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.430320    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-kube-api-access-wgp4f\" (UniqueName: \"kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-kube-api-access-wgp4f\") pod \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") "
Nov 03 02:09:21.430496 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.430419    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-socket\") pod \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") "
Nov 03 02:09:21.430496 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.430462    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-podinfo\") pod \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") "
Nov 03 02:09:21.430698 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.430497    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istiod-ca-cert\") pod \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") "
Nov 03 02:09:21.430698 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.430585    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-certs" pod "8bbc58e2-b00c-4bc7-ba27-cabe0d546740" (UID: "8bbc58e2-b00c-4bc7-ba27-cabe0d546740"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:21.430698 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.430639    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-credential-socket\") pod \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") "
Nov 03 02:09:21.430871 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.430702    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-envoy\") pod \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") "
Nov 03 02:09:21.430871 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.430764    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-token\" (UniqueName: \"kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-token\") pod \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") "
Nov 03 02:09:21.430871 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.430823    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-data\") pod \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\" (UID: \"8bbc58e2-b00c-4bc7-ba27-cabe0d546740\") "
Nov 03 02:09:21.431041 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.430931    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-credential-socket" pod "8bbc58e2-b00c-4bc7-ba27-cabe0d546740" (UID: "8bbc58e2-b00c-4bc7-ba27-cabe0d546740"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:21.431270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.431161    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-socket" pod "8bbc58e2-b00c-4bc7-ba27-cabe0d546740" (UID: "8bbc58e2-b00c-4bc7-ba27-cabe0d546740"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:21.431270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.431226    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istiod-ca-cert" pod "8bbc58e2-b00c-4bc7-ba27-cabe0d546740" (UID: "8bbc58e2-b00c-4bc7-ba27-cabe0d546740"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:09:21.431593 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.431412    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-data" pod "8bbc58e2-b00c-4bc7-ba27-cabe0d546740" (UID: "8bbc58e2-b00c-4bc7-ba27-cabe0d546740"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:21.431678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.431646    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:21.431770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.431676    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:21.431770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.431693    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:21.431770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.431707    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:21.431770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.431720    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:21.433700 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.433638    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-podinfo" pod "8bbc58e2-b00c-4bc7-ba27-cabe0d546740" (UID: "8bbc58e2-b00c-4bc7-ba27-cabe0d546740"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:09:21.433700 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.433642    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-token" pod "8bbc58e2-b00c-4bc7-ba27-cabe0d546740" (UID: "8bbc58e2-b00c-4bc7-ba27-cabe0d546740"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:09:21.434106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.433695    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-kube-api-access-wgp4f" pod "8bbc58e2-b00c-4bc7-ba27-cabe0d546740" (UID: "8bbc58e2-b00c-4bc7-ba27-cabe0d546740"). InnerVolumeSpecName "kube-api-access-wgp4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:09:21.434106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.434000    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-envoy" pod "8bbc58e2-b00c-4bc7-ba27-cabe0d546740" (UID: "8bbc58e2-b00c-4bc7-ba27-cabe0d546740"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:21.533551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.533473    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:21.533551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.533510    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgp4f\" (UniqueName: \"kubernetes.io/projected/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-kube-api-access-wgp4f\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:21.533551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.533529    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:21.533551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.533546    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bbc58e2-b00c-4bc7-ba27-cabe0d546740-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:21.939989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.939955    5508 generic.go:356] "Generic (PLEG): container finished" podID="8bbc58e2-b00c-4bc7-ba27-cabe0d546740" containerID="0329c02ba8a4fddc392fd98bb2c37033e2de7ad8403dd27b07687706870377d2" exitCode=0
Nov 03 02:09:21.940181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.940001    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"
Nov 03 02:09:21.940181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.940042    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9" event={"ID":"8bbc58e2-b00c-4bc7-ba27-cabe0d546740","Type":"ContainerDied","Data":"0329c02ba8a4fddc392fd98bb2c37033e2de7ad8403dd27b07687706870377d2"}
Nov 03 02:09:21.940181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.940112    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9" event={"ID":"8bbc58e2-b00c-4bc7-ba27-cabe0d546740","Type":"ContainerDied","Data":"b247b27313d9bc39ad377c915029cab81c243b80e5cfcf0ee90b4168b8c0fd1e"}
Nov 03 02:09:21.940181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.940149    5508 scope.go:122] "RemoveContainer" containerID="0329c02ba8a4fddc392fd98bb2c37033e2de7ad8403dd27b07687706870377d2"
Nov 03 02:09:21.951680 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.951647    5508 scope.go:122] "RemoveContainer" containerID="0329c02ba8a4fddc392fd98bb2c37033e2de7ad8403dd27b07687706870377d2"
Nov 03 02:09:21.951958 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:09:21.951942    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0329c02ba8a4fddc392fd98bb2c37033e2de7ad8403dd27b07687706870377d2\": container with ID starting with 0329c02ba8a4fddc392fd98bb2c37033e2de7ad8403dd27b07687706870377d2 not found: ID does not exist" containerID="0329c02ba8a4fddc392fd98bb2c37033e2de7ad8403dd27b07687706870377d2"
Nov 03 02:09:21.952029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:21.951966    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0329c02ba8a4fddc392fd98bb2c37033e2de7ad8403dd27b07687706870377d2"} err="failed to get container status \"0329c02ba8a4fddc392fd98bb2c37033e2de7ad8403dd27b07687706870377d2\": rpc error: code = NotFound desc = could not find container \"0329c02ba8a4fddc392fd98bb2c37033e2de7ad8403dd27b07687706870377d2\": container with ID starting with 0329c02ba8a4fddc392fd98bb2c37033e2de7ad8403dd27b07687706870377d2 not found: ID does not exist"
Nov 03 02:09:22.141432 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:22.141404    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"]
Nov 03 02:09:22.187608 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:22.187567    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e1793d-openshift-default-7b764d5f4f-hkbc9"]
Nov 03 02:09:22.695281 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:22.695248    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bbc58e2-b00c-4bc7-ba27-cabe0d546740" path="/var/lib/kubelet/pods/8bbc58e2-b00c-4bc7-ba27-cabe0d546740/volumes"
Nov 03 02:09:26.537417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.537384    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"
Nov 03 02:09:26.592162 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.592124    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-plugins-config-volume\") pod \"a5e3b594-c7a5-4f96-9c13-d9789331f2ef\" (UID: \"a5e3b594-c7a5-4f96-9c13-d9789331f2ef\") "
Nov 03 02:09:26.592415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.592202    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-kube-api-access-pphww\" (UniqueName: \"kubernetes.io/projected/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-kube-api-access-pphww\") pod \"a5e3b594-c7a5-4f96-9c13-d9789331f2ef\" (UID: \"a5e3b594-c7a5-4f96-9c13-d9789331f2ef\") "
Nov 03 02:09:26.592644 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.592619    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-plugins-config-volume" pod "a5e3b594-c7a5-4f96-9c13-d9789331f2ef" (UID: "a5e3b594-c7a5-4f96-9c13-d9789331f2ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:09:26.592924 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.592900    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:26.595435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.595403    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-kube-api-access-pphww" pod "a5e3b594-c7a5-4f96-9c13-d9789331f2ef" (UID: "a5e3b594-c7a5-4f96-9c13-d9789331f2ef"). InnerVolumeSpecName "kube-api-access-pphww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:09:26.629364 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.629316    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"
Nov 03 02:09:26.693410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.693382    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7df245af-91c3-42d2-9a61-bff747cd1701-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df245af-91c3-42d2-9a61-bff747cd1701-plugins-config-volume\") pod \"7df245af-91c3-42d2-9a61-bff747cd1701\" (UID: \"7df245af-91c3-42d2-9a61-bff747cd1701\") "
Nov 03 02:09:26.693657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.693435    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df245af-91c3-42d2-9a61-bff747cd1701-kube-api-access-b8bs8\" (UniqueName: \"kubernetes.io/projected/7df245af-91c3-42d2-9a61-bff747cd1701-kube-api-access-b8bs8\") pod \"7df245af-91c3-42d2-9a61-bff747cd1701\" (UID: \"7df245af-91c3-42d2-9a61-bff747cd1701\") "
Nov 03 02:09:26.693896 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.693810    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df245af-91c3-42d2-9a61-bff747cd1701-plugins-config-volume" pod "7df245af-91c3-42d2-9a61-bff747cd1701" (UID: "7df245af-91c3-42d2-9a61-bff747cd1701"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:09:26.694613 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.694587    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pphww\" (UniqueName: \"kubernetes.io/projected/a5e3b594-c7a5-4f96-9c13-d9789331f2ef-kube-api-access-pphww\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:26.694712 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.694617    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7df245af-91c3-42d2-9a61-bff747cd1701-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:26.695865 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.695834    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df245af-91c3-42d2-9a61-bff747cd1701-kube-api-access-b8bs8" pod "7df245af-91c3-42d2-9a61-bff747cd1701" (UID: "7df245af-91c3-42d2-9a61-bff747cd1701"). InnerVolumeSpecName "kube-api-access-b8bs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:09:26.796048 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.795962    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8bs8\" (UniqueName: \"kubernetes.io/projected/7df245af-91c3-42d2-9a61-bff747cd1701-kube-api-access-b8bs8\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:26.981961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.981923    5508 generic.go:356] "Generic (PLEG): container finished" podID="7df245af-91c3-42d2-9a61-bff747cd1701" containerID="ab2d4f4a41029b6a7c006b3ba920858be7759d47d6a9d97b1839dc75f2dbebee" exitCode=0
Nov 03 02:09:26.982227 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.981973    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"
Nov 03 02:09:26.982227 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.982009    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58" event={"ID":"7df245af-91c3-42d2-9a61-bff747cd1701","Type":"ContainerDied","Data":"ab2d4f4a41029b6a7c006b3ba920858be7759d47d6a9d97b1839dc75f2dbebee"}
Nov 03 02:09:26.982227 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.982081    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58" event={"ID":"7df245af-91c3-42d2-9a61-bff747cd1701","Type":"ContainerDied","Data":"ca4946d202a251bdab8854f804ffd5d25e2a2a87225666415cddb82800764ac6"}
Nov 03 02:09:26.982227 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.982114    5508 scope.go:122] "RemoveContainer" containerID="ab2d4f4a41029b6a7c006b3ba920858be7759d47d6a9d97b1839dc75f2dbebee"
Nov 03 02:09:26.983964 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.983945    5508 generic.go:356] "Generic (PLEG): container finished" podID="a5e3b594-c7a5-4f96-9c13-d9789331f2ef" containerID="a940ccac965c0f4778041b81558bbdd1fbe3eb552e30e5c58a31d0c8437e1930" exitCode=0
Nov 03 02:09:26.984083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.983994    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"
Nov 03 02:09:26.984083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.984034    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng" event={"ID":"a5e3b594-c7a5-4f96-9c13-d9789331f2ef","Type":"ContainerDied","Data":"a940ccac965c0f4778041b81558bbdd1fbe3eb552e30e5c58a31d0c8437e1930"}
Nov 03 02:09:26.984083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.984070    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng" event={"ID":"a5e3b594-c7a5-4f96-9c13-d9789331f2ef","Type":"ContainerDied","Data":"e364710c40a33ee4ed31db81e682c85a3e36b6011bc868ad5c60ea6960d7860f"}
Nov 03 02:09:26.992720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.992704    5508 scope.go:122] "RemoveContainer" containerID="ab2d4f4a41029b6a7c006b3ba920858be7759d47d6a9d97b1839dc75f2dbebee"
Nov 03 02:09:26.992991 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:09:26.992973    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2d4f4a41029b6a7c006b3ba920858be7759d47d6a9d97b1839dc75f2dbebee\": container with ID starting with ab2d4f4a41029b6a7c006b3ba920858be7759d47d6a9d97b1839dc75f2dbebee not found: ID does not exist" containerID="ab2d4f4a41029b6a7c006b3ba920858be7759d47d6a9d97b1839dc75f2dbebee"
Nov 03 02:09:26.993058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.992999    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2d4f4a41029b6a7c006b3ba920858be7759d47d6a9d97b1839dc75f2dbebee"} err="failed to get container status \"ab2d4f4a41029b6a7c006b3ba920858be7759d47d6a9d97b1839dc75f2dbebee\": rpc error: code = NotFound desc = could not find container \"ab2d4f4a41029b6a7c006b3ba920858be7759d47d6a9d97b1839dc75f2dbebee\": container with ID starting with ab2d4f4a41029b6a7c006b3ba920858be7759d47d6a9d97b1839dc75f2dbebee not found: ID does not exist"
Nov 03 02:09:26.993058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:26.993014    5508 scope.go:122] "RemoveContainer" containerID="a940ccac965c0f4778041b81558bbdd1fbe3eb552e30e5c58a31d0c8437e1930"
Nov 03 02:09:27.002247 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:27.002220    5508 scope.go:122] "RemoveContainer" containerID="a940ccac965c0f4778041b81558bbdd1fbe3eb552e30e5c58a31d0c8437e1930"
Nov 03 02:09:27.002533 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:09:27.002517    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a940ccac965c0f4778041b81558bbdd1fbe3eb552e30e5c58a31d0c8437e1930\": container with ID starting with a940ccac965c0f4778041b81558bbdd1fbe3eb552e30e5c58a31d0c8437e1930 not found: ID does not exist" containerID="a940ccac965c0f4778041b81558bbdd1fbe3eb552e30e5c58a31d0c8437e1930"
Nov 03 02:09:27.002587 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:27.002541    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a940ccac965c0f4778041b81558bbdd1fbe3eb552e30e5c58a31d0c8437e1930"} err="failed to get container status \"a940ccac965c0f4778041b81558bbdd1fbe3eb552e30e5c58a31d0c8437e1930\": rpc error: code = NotFound desc = could not find container \"a940ccac965c0f4778041b81558bbdd1fbe3eb552e30e5c58a31d0c8437e1930\": container with ID starting with a940ccac965c0f4778041b81558bbdd1fbe3eb552e30e5c58a31d0c8437e1930 not found: ID does not exist"
Nov 03 02:09:27.136299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:27.136216    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"]
Nov 03 02:09:27.140460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:27.140433    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b22fcc-bffc65b95-6jg58"]
Nov 03 02:09:27.242125 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:27.242086    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"]
Nov 03 02:09:27.246463 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:27.246429    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b22fcc-6d6b7b9487-j27ng"]
Nov 03 02:09:28.692873 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:28.692842    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df245af-91c3-42d2-9a61-bff747cd1701" path="/var/lib/kubelet/pods/7df245af-91c3-42d2-9a61-bff747cd1701/volumes"
Nov 03 02:09:28.693271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:28.693263    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e3b594-c7a5-4f96-9c13-d9789331f2ef" path="/var/lib/kubelet/pods/a5e3b594-c7a5-4f96-9c13-d9789331f2ef/volumes"
Nov 03 02:09:40.495451 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:40.495412    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:09:43.812198 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:09:43.812154    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:09:45.359449 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:45.359417    5508 scope.go:122] "RemoveContainer" containerID="202071504b638f4596fb7fc0456c6d295881c9f8524731a70521ddae584abe0e"
Nov 03 02:09:45.370595 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:45.370575    5508 scope.go:122] "RemoveContainer" containerID="3226670a10a4a5aad4ff1b0441f4d39f538c509c34580e30810431e0e987c0aa"
Nov 03 02:09:45.381017 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:45.380987    5508 scope.go:122] "RemoveContainer" containerID="cec33e7c2e9424ebcc1b6527b0f1b2bdf9cc9ed2d8e6ba98ee26983d85f2e731"
Nov 03 02:09:45.390007 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:45.389988    5508 scope.go:122] "RemoveContainer" containerID="ce88cdab1a4e712950e8c5d6182a2cb8b2b12257270889b2d403f1c27baa4f82"
Nov 03 02:09:45.398633 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:45.398615    5508 scope.go:122] "RemoveContainer" containerID="97f2529a691201fe7fee8ab5dbf7b2598691c3fae3d0e853ecde16e122aa9d49"
Nov 03 02:09:45.406906 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:45.406879    5508 scope.go:122] "RemoveContainer" containerID="d66ccdd25da4932862be187a725e5d5f1aaec870e4a24f68008b4504121c8603"
Nov 03 02:09:45.414944 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:45.414927    5508 scope.go:122] "RemoveContainer" containerID="e1297b3c8ea00f0443f21a46a1e6646029871791e228cc117080df2dadbca69c"
Nov 03 02:09:45.424766 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:45.424748    5508 scope.go:122] "RemoveContainer" containerID="f6c7d72055073ac6a3ea647c036b7bf2d6f2b62c98c3a63e6c42ed587c83c876"
Nov 03 02:09:45.433410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:45.433390    5508 scope.go:122] "RemoveContainer" containerID="d8ced8e85c1f27bbc4d5834ddc053b515641a3cd2d32dc5838b759a6d967ac73"
Nov 03 02:09:45.441588 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:45.441571    5508 scope.go:122] "RemoveContainer" containerID="dfbce68f9500614f74e440057cd8dd6026e556432ed38135e4f5bf850ddc79c3"
Nov 03 02:09:45.450053 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:45.450033    5508 scope.go:122] "RemoveContainer" containerID="91e3c8c251af4e4dacecd981bacf8c379aec7d31ea790b10eac20bdef6efcea0"
Nov 03 02:09:48.953304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.953211    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"]
Nov 03 02:09:48.954096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.954018    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7df245af-91c3-42d2-9a61-bff747cd1701" containerName="payload-pre-processing"
Nov 03 02:09:48.954096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.954038    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df245af-91c3-42d2-9a61-bff747cd1701" containerName="payload-pre-processing"
Nov 03 02:09:48.954096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.954070    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8bbc58e2-b00c-4bc7-ba27-cabe0d546740" containerName="istio-proxy"
Nov 03 02:09:48.954096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.954079    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bbc58e2-b00c-4bc7-ba27-cabe0d546740" containerName="istio-proxy"
Nov 03 02:09:48.954096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.954099    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a5e3b594-c7a5-4f96-9c13-d9789331f2ef" containerName="payload-processing"
Nov 03 02:09:48.954421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.954106    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e3b594-c7a5-4f96-9c13-d9789331f2ef" containerName="payload-processing"
Nov 03 02:09:48.954421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.954129    5508 cpu_manager.go:414] "containerMap: removing container" podUID="2a9ea631-7c13-46c8-831a-89e4ac4c6776" containerName="revoke-keys"
Nov 03 02:09:48.954421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.954136    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9ea631-7c13-46c8-831a-89e4ac4c6776" containerName="revoke-keys"
Nov 03 02:09:48.954421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.954254    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df245af-91c3-42d2-9a61-bff747cd1701" containerName="payload-pre-processing"
Nov 03 02:09:48.954421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.954271    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bbc58e2-b00c-4bc7-ba27-cabe0d546740" containerName="istio-proxy"
Nov 03 02:09:48.954421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.954281    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e3b594-c7a5-4f96-9c13-d9789331f2ef" containerName="payload-processing"
Nov 03 02:09:48.954421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.954288    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9ea631-7c13-46c8-831a-89e4ac4c6776" containerName="revoke-keys"
Nov 03 02:09:48.960065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:48.960043    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"
Nov 03 02:09:49.005726 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.005700    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb4zv\" (UniqueName: \"kubernetes.io/projected/3bd2e071-eddc-4363-827a-487d7d2a9b46-kube-api-access-gb4zv\") pod \"maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh\" (UID: \"3bd2e071-eddc-4363-827a-487d7d2a9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"
Nov 03 02:09:49.005860 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.005845    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh\" (UID: \"3bd2e071-eddc-4363-827a-487d7d2a9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"
Nov 03 02:09:49.005914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.005903    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh\" (UID: \"3bd2e071-eddc-4363-827a-487d7d2a9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"
Nov 03 02:09:49.011625 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.011586    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:09:49.011801 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.011783    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:09:49.014888 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.014868    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"]
Nov 03 02:09:49.107108 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.107080    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb4zv\" (UniqueName: \"kubernetes.io/projected/3bd2e071-eddc-4363-827a-487d7d2a9b46-kube-api-access-gb4zv\") pod \"maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh\" (UID: \"3bd2e071-eddc-4363-827a-487d7d2a9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"
Nov 03 02:09:49.107278 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.107227    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh\" (UID: \"3bd2e071-eddc-4363-827a-487d7d2a9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"
Nov 03 02:09:49.107338 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.107308    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh\" (UID: \"3bd2e071-eddc-4363-827a-487d7d2a9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"
Nov 03 02:09:49.108143 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.108118    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh\" (UID: \"3bd2e071-eddc-4363-827a-487d7d2a9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"
Nov 03 02:09:49.109561 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.109525    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh\" (UID: \"3bd2e071-eddc-4363-827a-487d7d2a9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"
Nov 03 02:09:49.167330 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.167306    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb4zv\" (UniqueName: \"kubernetes.io/projected/3bd2e071-eddc-4363-827a-487d7d2a9b46-kube-api-access-gb4zv\") pod \"maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh\" (UID: \"3bd2e071-eddc-4363-827a-487d7d2a9b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"
Nov 03 02:09:49.275398 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.275304    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"
Nov 03 02:09:49.526022 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.525914    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"]
Nov 03 02:09:49.526196 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.526170    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb" podUID="9e8addfd-f2cf-4a72-af4c-f803c5990dae" containerName="istio-proxy" containerID="cri-o://bfb38958c612113c4dc86911125e5933fe5683c8d8ed330acd2cb4a51c73c051" gracePeriod=30
Nov 03 02:09:49.572605 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.572573    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"]
Nov 03 02:09:49.572876 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.572849    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh" podUID="2002ee43-2f1a-4ea5-98a8-b5afa7a88305" containerName="istio-proxy" containerID="cri-o://4ecd66cab7395a2921757c28d228f357556b392b61155d9c1322e07aed912128" gracePeriod=30
Nov 03 02:09:49.575958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.575906    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"]
Nov 03 02:09:49.576189 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.576143    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c" podUID="b927e0b6-c156-4323-819b-300546f8d1cc" containerName="istio-proxy" containerID="cri-o://8ca0cc77229ec48597d8575f10aba03c76cd16593e9a087000da7900e42d8d3a" gracePeriod=30
Nov 03 02:09:49.693076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:49.693037    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"]
Nov 03 02:09:49.698587 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:09:49.698556    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd2e071_eddc_4363_827a_487d7d2a9b46.slice/crio-793980b73044d98d9321af72babb05477efd91855bff8cbe34c3a8b1f79bf863 WatchSource:0}: Error finding container 793980b73044d98d9321af72babb05477efd91855bff8cbe34c3a8b1f79bf863: Status 404 returned error can't find the container with id 793980b73044d98d9321af72babb05477efd91855bff8cbe34c3a8b1f79bf863
Nov 03 02:09:50.167084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:50.167047    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh" event={"ID":"3bd2e071-eddc-4363-827a-487d7d2a9b46","Type":"ContainerStarted","Data":"bbc6fab9aa7864098f1ab51342e57bd561ff47b44500ac1554077b40032078ca"}
Nov 03 02:09:50.167472 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:50.167107    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh" event={"ID":"3bd2e071-eddc-4363-827a-487d7d2a9b46","Type":"ContainerStarted","Data":"793980b73044d98d9321af72babb05477efd91855bff8cbe34c3a8b1f79bf863"}
Nov 03 02:09:50.277268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:50.277214    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh" podStartSLOduration=2.277201121 podStartE2EDuration="2.277201121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:09:50.274611326 +0000 UTC m=+3069.700217808" watchObservedRunningTime="2026-11-03 02:09:50.277201121 +0000 UTC m=+3069.702807587"
Nov 03 02:09:51.176580 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:51.176544    5508 generic.go:356] "Generic (PLEG): container finished" podID="3bd2e071-eddc-4363-827a-487d7d2a9b46" containerID="bbc6fab9aa7864098f1ab51342e57bd561ff47b44500ac1554077b40032078ca" exitCode=0
Nov 03 02:09:51.177003 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:51.176620    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh" event={"ID":"3bd2e071-eddc-4363-827a-487d7d2a9b46","Type":"ContainerDied","Data":"bbc6fab9aa7864098f1ab51342e57bd561ff47b44500ac1554077b40032078ca"}
Nov 03 02:09:52.344481 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:52.344454    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"
Nov 03 02:09:52.449134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:52.449049    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd2e071-eddc-4363-827a-487d7d2a9b46-kube-api-access-gb4zv\" (UniqueName: \"kubernetes.io/projected/3bd2e071-eddc-4363-827a-487d7d2a9b46-kube-api-access-gb4zv\") pod \"3bd2e071-eddc-4363-827a-487d7d2a9b46\" (UID: \"3bd2e071-eddc-4363-827a-487d7d2a9b46\") "
Nov 03 02:09:52.449313 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:52.449174    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-xks-ca\") pod \"3bd2e071-eddc-4363-827a-487d7d2a9b46\" (UID: \"3bd2e071-eddc-4363-827a-487d7d2a9b46\") "
Nov 03 02:09:52.449313 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:52.449298    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-openshift-ca\") pod \"3bd2e071-eddc-4363-827a-487d7d2a9b46\" (UID: \"3bd2e071-eddc-4363-827a-487d7d2a9b46\") "
Nov 03 02:09:52.449732 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:52.449701    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-openshift-ca" pod "3bd2e071-eddc-4363-827a-487d7d2a9b46" (UID: "3bd2e071-eddc-4363-827a-487d7d2a9b46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:09:52.449948 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:52.449930    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:52.451744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:52.451713    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-xks-ca" pod "3bd2e071-eddc-4363-827a-487d7d2a9b46" (UID: "3bd2e071-eddc-4363-827a-487d7d2a9b46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:09:52.451858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:52.451747    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd2e071-eddc-4363-827a-487d7d2a9b46-kube-api-access-gb4zv" pod "3bd2e071-eddc-4363-827a-487d7d2a9b46" (UID: "3bd2e071-eddc-4363-827a-487d7d2a9b46"). InnerVolumeSpecName "kube-api-access-gb4zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:09:52.561249 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:52.559563    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb4zv\" (UniqueName: \"kubernetes.io/projected/3bd2e071-eddc-4363-827a-487d7d2a9b46-kube-api-access-gb4zv\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:52.561249 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:52.559621    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd2e071-eddc-4363-827a-487d7d2a9b46-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:53.196544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:53.196513    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"
Nov 03 02:09:53.196730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:53.196517    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh" event={"ID":"3bd2e071-eddc-4363-827a-487d7d2a9b46","Type":"ContainerDied","Data":"793980b73044d98d9321af72babb05477efd91855bff8cbe34c3a8b1f79bf863"}
Nov 03 02:09:53.196730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:53.196648    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="793980b73044d98d9321af72babb05477efd91855bff8cbe34c3a8b1f79bf863"
Nov 03 02:09:54.862050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:54.860714    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:09:54.887049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:54.887018    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:09:54.926998 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:54.926975    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:09:55.001221 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001190    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-socket\") pod \"b927e0b6-c156-4323-819b-300546f8d1cc\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") "
Nov 03 02:09:55.001379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001239    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-certs\") pod \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") "
Nov 03 02:09:55.001379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001267    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-data\") pod \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") "
Nov 03 02:09:55.001379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001298    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-istio-token\" (UniqueName: \"kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-istio-token\") pod \"b927e0b6-c156-4323-819b-300546f8d1cc\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") "
Nov 03 02:09:55.001379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001325    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-envoy\") pod \"b927e0b6-c156-4323-819b-300546f8d1cc\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") "
Nov 03 02:09:55.001612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001380    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-credential-socket\") pod \"b927e0b6-c156-4323-819b-300546f8d1cc\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") "
Nov 03 02:09:55.001612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001416    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b927e0b6-c156-4323-819b-300546f8d1cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b927e0b6-c156-4323-819b-300546f8d1cc-istiod-ca-cert\") pod \"b927e0b6-c156-4323-819b-300546f8d1cc\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") "
Nov 03 02:09:55.001612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001454    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-data\") pod \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") "
Nov 03 02:09:55.001612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001497    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-data\") pod \"b927e0b6-c156-4323-819b-300546f8d1cc\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") "
Nov 03 02:09:55.001612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001507    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-certs" pod "9e8addfd-f2cf-4a72-af4c-f803c5990dae" (UID: "9e8addfd-f2cf-4a72-af4c-f803c5990dae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.001612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001509    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-socket" pod "b927e0b6-c156-4323-819b-300546f8d1cc" (UID: "b927e0b6-c156-4323-819b-300546f8d1cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.001612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001567    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-data" pod "9e8addfd-f2cf-4a72-af4c-f803c5990dae" (UID: "9e8addfd-f2cf-4a72-af4c-f803c5990dae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.001612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001578    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-certs\") pod \"b927e0b6-c156-4323-819b-300546f8d1cc\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") "
Nov 03 02:09:55.002038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001623    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-podinfo\") pod \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") "
Nov 03 02:09:55.002038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001675    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-data" pod "2002ee43-2f1a-4ea5-98a8-b5afa7a88305" (UID: "2002ee43-2f1a-4ea5-98a8-b5afa7a88305"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.002038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001679    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istiod-ca-cert\") pod \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") "
Nov 03 02:09:55.002038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001715    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-credential-socket" pod "b927e0b6-c156-4323-819b-300546f8d1cc" (UID: "b927e0b6-c156-4323-819b-300546f8d1cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.002038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001725    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-credential-socket\") pod \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") "
Nov 03 02:09:55.002038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001825    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-kube-api-access-2b9sg\" (UniqueName: \"kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-kube-api-access-2b9sg\") pod \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") "
Nov 03 02:09:55.002038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001826    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b927e0b6-c156-4323-819b-300546f8d1cc-istiod-ca-cert" pod "b927e0b6-c156-4323-819b-300546f8d1cc" (UID: "b927e0b6-c156-4323-819b-300546f8d1cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:09:55.002038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001837    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-certs" pod "b927e0b6-c156-4323-819b-300546f8d1cc" (UID: "b927e0b6-c156-4323-819b-300546f8d1cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.002038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001867    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-data" pod "b927e0b6-c156-4323-819b-300546f8d1cc" (UID: "b927e0b6-c156-4323-819b-300546f8d1cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.002038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001894    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-envoy\") pod \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") "
Nov 03 02:09:55.002038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.001954    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b927e0b6-c156-4323-819b-300546f8d1cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b927e0b6-c156-4323-819b-300546f8d1cc-istio-podinfo\") pod \"b927e0b6-c156-4323-819b-300546f8d1cc\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") "
Nov 03 02:09:55.002038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.002028    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istiod-ca-cert" pod "2002ee43-2f1a-4ea5-98a8-b5afa7a88305" (UID: "2002ee43-2f1a-4ea5-98a8-b5afa7a88305"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:09:55.002038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.002034    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-token\" (UniqueName: \"kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-token\") pod \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") "
Nov 03 02:09:55.002767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.002117    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-credential-socket\") pod \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") "
Nov 03 02:09:55.002767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.002148    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-podinfo\") pod \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") "
Nov 03 02:09:55.002767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.002211    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-kube-api-access-nj627\" (UniqueName: \"kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-kube-api-access-nj627\") pod \"b927e0b6-c156-4323-819b-300546f8d1cc\" (UID: \"b927e0b6-c156-4323-819b-300546f8d1cc\") "
Nov 03 02:09:55.002767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.002279    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-token\" (UniqueName: \"kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-token\") pod \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") "
Nov 03 02:09:55.002767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.002325    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istiod-ca-cert\") pod \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") "
Nov 03 02:09:55.002767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.002414    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-socket\") pod \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") "
Nov 03 02:09:55.002767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.002444    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-kube-api-access-wmphd\" (UniqueName: \"kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-kube-api-access-wmphd\") pod \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") "
Nov 03 02:09:55.002767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.002570    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-credential-socket" pod "2002ee43-2f1a-4ea5-98a8-b5afa7a88305" (UID: "2002ee43-2f1a-4ea5-98a8-b5afa7a88305"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.003223 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.002897    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istiod-ca-cert" pod "9e8addfd-f2cf-4a72-af4c-f803c5990dae" (UID: "9e8addfd-f2cf-4a72-af4c-f803c5990dae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:09:55.003223 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.003065    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-credential-socket" pod "9e8addfd-f2cf-4a72-af4c-f803c5990dae" (UID: "9e8addfd-f2cf-4a72-af4c-f803c5990dae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.003377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.003266    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-socket" pod "9e8addfd-f2cf-4a72-af4c-f803c5990dae" (UID: "9e8addfd-f2cf-4a72-af4c-f803c5990dae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.003377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.003305    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-envoy\") pod \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\" (UID: \"9e8addfd-f2cf-4a72-af4c-f803c5990dae\") "
Nov 03 02:09:55.003377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.003339    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-socket\") pod \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") "
Nov 03 02:09:55.003555 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.003412    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-certs\") pod \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\" (UID: \"2002ee43-2f1a-4ea5-98a8-b5afa7a88305\") "
Nov 03 02:09:55.004069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004044    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.004069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004075    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.004341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004092    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.004341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004107    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.004341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004119    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.004341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004132    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.004341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004149    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.004341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004165    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b927e0b6-c156-4323-819b-300546f8d1cc-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.004341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004180    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.004341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004194    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.004341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004208    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.004341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004220    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.004341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004235    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.006109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004616    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-socket" pod "2002ee43-2f1a-4ea5-98a8-b5afa7a88305" (UID: "2002ee43-2f1a-4ea5-98a8-b5afa7a88305"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.006109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004609    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-envoy" pod "2002ee43-2f1a-4ea5-98a8-b5afa7a88305" (UID: "2002ee43-2f1a-4ea5-98a8-b5afa7a88305"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.006109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.004683    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-certs" pod "2002ee43-2f1a-4ea5-98a8-b5afa7a88305" (UID: "2002ee43-2f1a-4ea5-98a8-b5afa7a88305"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.006109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.005173    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-podinfo" pod "2002ee43-2f1a-4ea5-98a8-b5afa7a88305" (UID: "2002ee43-2f1a-4ea5-98a8-b5afa7a88305"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:09:55.006109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.005216    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-token" pod "9e8addfd-f2cf-4a72-af4c-f803c5990dae" (UID: "9e8addfd-f2cf-4a72-af4c-f803c5990dae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:09:55.006109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.005221    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-envoy" pod "b927e0b6-c156-4323-819b-300546f8d1cc" (UID: "b927e0b6-c156-4323-819b-300546f8d1cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.006109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.005717    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-istio-token" pod "b927e0b6-c156-4323-819b-300546f8d1cc" (UID: "b927e0b6-c156-4323-819b-300546f8d1cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:09:55.006109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.005888    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-kube-api-access-2b9sg" pod "9e8addfd-f2cf-4a72-af4c-f803c5990dae" (UID: "9e8addfd-f2cf-4a72-af4c-f803c5990dae"). InnerVolumeSpecName "kube-api-access-2b9sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:09:55.006508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.006215    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-podinfo" pod "9e8addfd-f2cf-4a72-af4c-f803c5990dae" (UID: "9e8addfd-f2cf-4a72-af4c-f803c5990dae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:09:55.006508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.006305    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b927e0b6-c156-4323-819b-300546f8d1cc-istio-podinfo" pod "b927e0b6-c156-4323-819b-300546f8d1cc" (UID: "b927e0b6-c156-4323-819b-300546f8d1cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:09:55.006587 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.006537    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-token" pod "2002ee43-2f1a-4ea5-98a8-b5afa7a88305" (UID: "2002ee43-2f1a-4ea5-98a8-b5afa7a88305"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:09:55.006644 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.006623    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-kube-api-access-wmphd" pod "2002ee43-2f1a-4ea5-98a8-b5afa7a88305" (UID: "2002ee43-2f1a-4ea5-98a8-b5afa7a88305"). InnerVolumeSpecName "kube-api-access-wmphd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:09:55.006776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.006751    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-kube-api-access-nj627" pod "b927e0b6-c156-4323-819b-300546f8d1cc" (UID: "b927e0b6-c156-4323-819b-300546f8d1cc"). InnerVolumeSpecName "kube-api-access-nj627". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:09:55.006945 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.006927    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-envoy" pod "9e8addfd-f2cf-4a72-af4c-f803c5990dae" (UID: "9e8addfd-f2cf-4a72-af4c-f803c5990dae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:09:55.105954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.105876    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.105954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.105910    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b927e0b6-c156-4323-819b-300546f8d1cc-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.105954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.105920    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.105954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.105929    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b9sg\" (UniqueName: \"kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-kube-api-access-2b9sg\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.105954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.105940    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.105954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.105948    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b927e0b6-c156-4323-819b-300546f8d1cc-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.105954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.105956    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.106292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.105965    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.106292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.105974    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj627\" (UniqueName: \"kubernetes.io/projected/b927e0b6-c156-4323-819b-300546f8d1cc-kube-api-access-nj627\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.106292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.105982    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.106292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.105991    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmphd\" (UniqueName: \"kubernetes.io/projected/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-kube-api-access-wmphd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.106292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.106000    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e8addfd-f2cf-4a72-af4c-f803c5990dae-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.106292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.106009    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.106292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.106018    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2002ee43-2f1a-4ea5-98a8-b5afa7a88305-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:09:55.213102 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.213061    5508 generic.go:356] "Generic (PLEG): container finished" podID="9e8addfd-f2cf-4a72-af4c-f803c5990dae" containerID="bfb38958c612113c4dc86911125e5933fe5683c8d8ed330acd2cb4a51c73c051" exitCode=0
Nov 03 02:09:55.213310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.213127    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"
Nov 03 02:09:55.213310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.213211    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb" event={"ID":"9e8addfd-f2cf-4a72-af4c-f803c5990dae","Type":"ContainerDied","Data":"bfb38958c612113c4dc86911125e5933fe5683c8d8ed330acd2cb4a51c73c051"}
Nov 03 02:09:55.213310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.213264    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb" event={"ID":"9e8addfd-f2cf-4a72-af4c-f803c5990dae","Type":"ContainerDied","Data":"7fb9b1f4622b77fe987bfb5efffb2eb5cf71c15de2aa27c7bf26ad213896abe2"}
Nov 03 02:09:55.213310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.213298    5508 scope.go:122] "RemoveContainer" containerID="bfb38958c612113c4dc86911125e5933fe5683c8d8ed330acd2cb4a51c73c051"
Nov 03 02:09:55.215530 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.215502    5508 generic.go:356] "Generic (PLEG): container finished" podID="2002ee43-2f1a-4ea5-98a8-b5afa7a88305" containerID="4ecd66cab7395a2921757c28d228f357556b392b61155d9c1322e07aed912128" exitCode=0
Nov 03 02:09:55.215652 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.215609    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh" event={"ID":"2002ee43-2f1a-4ea5-98a8-b5afa7a88305","Type":"ContainerDied","Data":"4ecd66cab7395a2921757c28d228f357556b392b61155d9c1322e07aed912128"}
Nov 03 02:09:55.215757 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.215651    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh" event={"ID":"2002ee43-2f1a-4ea5-98a8-b5afa7a88305","Type":"ContainerDied","Data":"4d670eee92a60022979b0b32207c5e08b2a9f5a57f7970cd0f81aa7af0c6f1b1"}
Nov 03 02:09:55.215757 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.215732    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"
Nov 03 02:09:55.223071 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.223045    5508 generic.go:356] "Generic (PLEG): container finished" podID="b927e0b6-c156-4323-819b-300546f8d1cc" containerID="8ca0cc77229ec48597d8575f10aba03c76cd16593e9a087000da7900e42d8d3a" exitCode=0
Nov 03 02:09:55.223183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.223099    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"
Nov 03 02:09:55.223183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.223124    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c" event={"ID":"b927e0b6-c156-4323-819b-300546f8d1cc","Type":"ContainerDied","Data":"8ca0cc77229ec48597d8575f10aba03c76cd16593e9a087000da7900e42d8d3a"}
Nov 03 02:09:55.223183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.223167    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c" event={"ID":"b927e0b6-c156-4323-819b-300546f8d1cc","Type":"ContainerDied","Data":"411dac136abf3527adec19871dd151c7a79092313706eb713f1c6815ec3fd047"}
Nov 03 02:09:55.226833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.226814    5508 scope.go:122] "RemoveContainer" containerID="bfb38958c612113c4dc86911125e5933fe5683c8d8ed330acd2cb4a51c73c051"
Nov 03 02:09:55.227194 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:09:55.227124    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb38958c612113c4dc86911125e5933fe5683c8d8ed330acd2cb4a51c73c051\": container with ID starting with bfb38958c612113c4dc86911125e5933fe5683c8d8ed330acd2cb4a51c73c051 not found: ID does not exist" containerID="bfb38958c612113c4dc86911125e5933fe5683c8d8ed330acd2cb4a51c73c051"
Nov 03 02:09:55.227194 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.227157    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb38958c612113c4dc86911125e5933fe5683c8d8ed330acd2cb4a51c73c051"} err="failed to get container status \"bfb38958c612113c4dc86911125e5933fe5683c8d8ed330acd2cb4a51c73c051\": rpc error: code = NotFound desc = could not find container \"bfb38958c612113c4dc86911125e5933fe5683c8d8ed330acd2cb4a51c73c051\": container with ID starting with bfb38958c612113c4dc86911125e5933fe5683c8d8ed330acd2cb4a51c73c051 not found: ID does not exist"
Nov 03 02:09:55.227194 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.227181    5508 scope.go:122] "RemoveContainer" containerID="4ecd66cab7395a2921757c28d228f357556b392b61155d9c1322e07aed912128"
Nov 03 02:09:55.236828 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.236808    5508 scope.go:122] "RemoveContainer" containerID="4ecd66cab7395a2921757c28d228f357556b392b61155d9c1322e07aed912128"
Nov 03 02:09:55.237093 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:09:55.237075    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ecd66cab7395a2921757c28d228f357556b392b61155d9c1322e07aed912128\": container with ID starting with 4ecd66cab7395a2921757c28d228f357556b392b61155d9c1322e07aed912128 not found: ID does not exist" containerID="4ecd66cab7395a2921757c28d228f357556b392b61155d9c1322e07aed912128"
Nov 03 02:09:55.237159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.237100    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecd66cab7395a2921757c28d228f357556b392b61155d9c1322e07aed912128"} err="failed to get container status \"4ecd66cab7395a2921757c28d228f357556b392b61155d9c1322e07aed912128\": rpc error: code = NotFound desc = could not find container \"4ecd66cab7395a2921757c28d228f357556b392b61155d9c1322e07aed912128\": container with ID starting with 4ecd66cab7395a2921757c28d228f357556b392b61155d9c1322e07aed912128 not found: ID does not exist"
Nov 03 02:09:55.237159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.237118    5508 scope.go:122] "RemoveContainer" containerID="8ca0cc77229ec48597d8575f10aba03c76cd16593e9a087000da7900e42d8d3a"
Nov 03 02:09:55.253940 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.253919    5508 scope.go:122] "RemoveContainer" containerID="8ca0cc77229ec48597d8575f10aba03c76cd16593e9a087000da7900e42d8d3a"
Nov 03 02:09:55.254427 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:09:55.254399    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca0cc77229ec48597d8575f10aba03c76cd16593e9a087000da7900e42d8d3a\": container with ID starting with 8ca0cc77229ec48597d8575f10aba03c76cd16593e9a087000da7900e42d8d3a not found: ID does not exist" containerID="8ca0cc77229ec48597d8575f10aba03c76cd16593e9a087000da7900e42d8d3a"
Nov 03 02:09:55.254582 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.254439    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca0cc77229ec48597d8575f10aba03c76cd16593e9a087000da7900e42d8d3a"} err="failed to get container status \"8ca0cc77229ec48597d8575f10aba03c76cd16593e9a087000da7900e42d8d3a\": rpc error: code = NotFound desc = could not find container \"8ca0cc77229ec48597d8575f10aba03c76cd16593e9a087000da7900e42d8d3a\": container with ID starting with 8ca0cc77229ec48597d8575f10aba03c76cd16593e9a087000da7900e42d8d3a not found: ID does not exist"
Nov 03 02:09:55.369799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.363798    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"]
Nov 03 02:09:55.372785 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.372760    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-4jb2c"]
Nov 03 02:09:55.475921 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.475886    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"]
Nov 03 02:09:55.480957 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.480930    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-mfzlb"]
Nov 03 02:09:55.581662 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.581630    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"]
Nov 03 02:09:55.586063 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:55.586033    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-qtklh"]
Nov 03 02:09:56.695076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:56.695036    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2002ee43-2f1a-4ea5-98a8-b5afa7a88305" path="/var/lib/kubelet/pods/2002ee43-2f1a-4ea5-98a8-b5afa7a88305/volumes"
Nov 03 02:09:56.695789 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:56.695769    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e8addfd-f2cf-4a72-af4c-f803c5990dae" path="/var/lib/kubelet/pods/9e8addfd-f2cf-4a72-af4c-f803c5990dae/volumes"
Nov 03 02:09:56.696437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:09:56.696417    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b927e0b6-c156-4323-819b-300546f8d1cc" path="/var/lib/kubelet/pods/b927e0b6-c156-4323-819b-300546f8d1cc/volumes"
Nov 03 02:10:07.592656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.592619    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"]
Nov 03 02:10:07.593501 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.593475    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b927e0b6-c156-4323-819b-300546f8d1cc" containerName="istio-proxy"
Nov 03 02:10:07.593501 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.593502    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b927e0b6-c156-4323-819b-300546f8d1cc" containerName="istio-proxy"
Nov 03 02:10:07.593671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.593515    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9e8addfd-f2cf-4a72-af4c-f803c5990dae" containerName="istio-proxy"
Nov 03 02:10:07.593671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.593520    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8addfd-f2cf-4a72-af4c-f803c5990dae" containerName="istio-proxy"
Nov 03 02:10:07.593671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.593561    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3bd2e071-eddc-4363-827a-487d7d2a9b46" containerName="revoke-keys"
Nov 03 02:10:07.593671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.593567    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd2e071-eddc-4363-827a-487d7d2a9b46" containerName="revoke-keys"
Nov 03 02:10:07.593671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.593582    5508 cpu_manager.go:414] "containerMap: removing container" podUID="2002ee43-2f1a-4ea5-98a8-b5afa7a88305" containerName="istio-proxy"
Nov 03 02:10:07.593671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.593586    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2002ee43-2f1a-4ea5-98a8-b5afa7a88305" containerName="istio-proxy"
Nov 03 02:10:07.593870 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.593691    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd2e071-eddc-4363-827a-487d7d2a9b46" containerName="revoke-keys"
Nov 03 02:10:07.593870 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.593704    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="2002ee43-2f1a-4ea5-98a8-b5afa7a88305" containerName="istio-proxy"
Nov 03 02:10:07.593870 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.593715    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b927e0b6-c156-4323-819b-300546f8d1cc" containerName="istio-proxy"
Nov 03 02:10:07.593870 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.593722    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8addfd-f2cf-4a72-af4c-f803c5990dae" containerName="istio-proxy"
Nov 03 02:10:07.597396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.597338    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.646299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.646264    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-98a394\"/\"openshift-service-ca.crt\""
Nov 03 02:10:07.657437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.657412    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-dshm\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.657573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.657447    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-tmp-dir\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.657621 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.657565    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-home\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.657621 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.657610    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-model-cache\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.657705 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.657694    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972xs\" (UniqueName: \"kubernetes.io/projected/e054f5c9-0628-45b3-8c8a-956d5b305518-kube-api-access-972xs\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.657771 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.657755    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e054f5c9-0628-45b3-8c8a-956d5b305518-tls-certs\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.692109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.692086    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-98a394\"/\"kube-root-ca.crt\""
Nov 03 02:10:07.692521 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.692505    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-98a394\"/\"default-dockercfg-ztgfx\""
Nov 03 02:10:07.692604 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.692533    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-98a394\"/\"auth-test-model-98a394-kserve-self-signed-certs\""
Nov 03 02:10:07.695873 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.695848    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"]
Nov 03 02:10:07.763281 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.763222    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-dshm\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.784563 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.784531    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-tmp-dir\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.784947 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.784910    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-home\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.785098 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.785083    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-model-cache\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.785248 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.785232    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972xs\" (UniqueName: \"kubernetes.io/projected/e054f5c9-0628-45b3-8c8a-956d5b305518-kube-api-access-972xs\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.785429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.785406    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e054f5c9-0628-45b3-8c8a-956d5b305518-tls-certs\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.787336 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.787312    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-home\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.787801 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.787780    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-tmp-dir\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.791267 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.790737    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-model-cache\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.791594 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.791571    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e054f5c9-0628-45b3-8c8a-956d5b305518-tls-certs\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.796972 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.796860    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-dshm\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.848796 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.848715    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-972xs\" (UniqueName: \"kubernetes.io/projected/e054f5c9-0628-45b3-8c8a-956d5b305518-kube-api-access-972xs\") pod \"auth-test-model-98a394-kserve-75b977bcc-njnxp\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") " pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:07.910605 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:07.910567    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:08.299968 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:08.299930    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"]
Nov 03 02:10:08.306204 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:10:08.306173    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode054f5c9_0628_45b3_8c8a_956d5b305518.slice/crio-b0586454866357865aabd5c9d01abc97cc7027ed78c53fd7b491ebbc5185c97a WatchSource:0}: Error finding container b0586454866357865aabd5c9d01abc97cc7027ed78c53fd7b491ebbc5185c97a: Status 404 returned error can't find the container with id b0586454866357865aabd5c9d01abc97cc7027ed78c53fd7b491ebbc5185c97a
Nov 03 02:10:08.331748 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:08.331719    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp" event={"ID":"e054f5c9-0628-45b3-8c8a-956d5b305518","Type":"ContainerStarted","Data":"b0586454866357865aabd5c9d01abc97cc7027ed78c53fd7b491ebbc5185c97a"}
Nov 03 02:10:09.342059 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:09.342021    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp" event={"ID":"e054f5c9-0628-45b3-8c8a-956d5b305518","Type":"ContainerStarted","Data":"e0673802b1af0001bbe669eb4b1b9e94d3f9439fbe261c1d29afae930fd3c072"}
Nov 03 02:10:09.452933 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:09.452882    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp" podStartSLOduration=2.45286693 podStartE2EDuration="2.45286693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:10:09.451412789 +0000 UTC m=+3088.877019271" watchObservedRunningTime="2026-11-03 02:10:09.45286693 +0000 UTC m=+3088.878473385"
Nov 03 02:10:09.911745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:09.911715    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:10.991252 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:10.991214    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"]
Nov 03 02:10:10.991779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:10.991484    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp" podUID="41ea6070-8986-41ae-98be-6e20baf08648" containerName="istio-proxy" containerID="cri-o://98b363f396d884557f47e5d66b21e1fad20ac32bfcf5aa45a1d411ade58ebca1" gracePeriod=30
Nov 03 02:10:11.037476 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:11.037433    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"]
Nov 03 02:10:11.037719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:11.037675    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk" podUID="3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" containerName="istio-proxy" containerID="cri-o://3972a35dd5d8c08eb457fb2f5e30ff451721e7a7d98b3b96c50a3e9ee7458910" gracePeriod=30
Nov 03 02:10:13.814486 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:10:13.814448    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:10:16.312729 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.312689    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:10:16.328502 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.328467    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-certs\") pod \"41ea6070-8986-41ae-98be-6e20baf08648\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") "
Nov 03 02:10:16.328667 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.328539    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41ea6070-8986-41ae-98be-6e20baf08648-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ea6070-8986-41ae-98be-6e20baf08648-istio-podinfo\") pod \"41ea6070-8986-41ae-98be-6e20baf08648\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") "
Nov 03 02:10:16.328667 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.328574    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-data\") pod \"41ea6070-8986-41ae-98be-6e20baf08648\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") "
Nov 03 02:10:16.328667 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.328601    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-socket\") pod \"41ea6070-8986-41ae-98be-6e20baf08648\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") "
Nov 03 02:10:16.328667 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.328631    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-istio-token\" (UniqueName: \"kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-istio-token\") pod \"41ea6070-8986-41ae-98be-6e20baf08648\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") "
Nov 03 02:10:16.328667 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.328654    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-credential-socket\") pod \"41ea6070-8986-41ae-98be-6e20baf08648\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") "
Nov 03 02:10:16.328974 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.328693    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-kube-api-access-mvspd\" (UniqueName: \"kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-kube-api-access-mvspd\") pod \"41ea6070-8986-41ae-98be-6e20baf08648\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") "
Nov 03 02:10:16.328974 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.328739    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-envoy\") pod \"41ea6070-8986-41ae-98be-6e20baf08648\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") "
Nov 03 02:10:16.328974 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.328771    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-certs" pod "41ea6070-8986-41ae-98be-6e20baf08648" (UID: "41ea6070-8986-41ae-98be-6e20baf08648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:10:16.328974 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.328882    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41ea6070-8986-41ae-98be-6e20baf08648-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ea6070-8986-41ae-98be-6e20baf08648-istiod-ca-cert\") pod \"41ea6070-8986-41ae-98be-6e20baf08648\" (UID: \"41ea6070-8986-41ae-98be-6e20baf08648\") "
Nov 03 02:10:16.329690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.329528    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.329690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.329614    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-credential-socket" pod "41ea6070-8986-41ae-98be-6e20baf08648" (UID: "41ea6070-8986-41ae-98be-6e20baf08648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:10:16.329890 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.329803    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ea6070-8986-41ae-98be-6e20baf08648-istiod-ca-cert" pod "41ea6070-8986-41ae-98be-6e20baf08648" (UID: "41ea6070-8986-41ae-98be-6e20baf08648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:10:16.329963 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.329940    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-socket" pod "41ea6070-8986-41ae-98be-6e20baf08648" (UID: "41ea6070-8986-41ae-98be-6e20baf08648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:10:16.330052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.330005    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-data" pod "41ea6070-8986-41ae-98be-6e20baf08648" (UID: "41ea6070-8986-41ae-98be-6e20baf08648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:10:16.331479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.331283    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-envoy" pod "41ea6070-8986-41ae-98be-6e20baf08648" (UID: "41ea6070-8986-41ae-98be-6e20baf08648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:10:16.331479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.331439    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41ea6070-8986-41ae-98be-6e20baf08648-istio-podinfo" pod "41ea6070-8986-41ae-98be-6e20baf08648" (UID: "41ea6070-8986-41ae-98be-6e20baf08648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:10:16.331739 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.331713    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-kube-api-access-mvspd" pod "41ea6070-8986-41ae-98be-6e20baf08648" (UID: "41ea6070-8986-41ae-98be-6e20baf08648"). InnerVolumeSpecName "kube-api-access-mvspd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:10:16.331845 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.331826    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-istio-token" pod "41ea6070-8986-41ae-98be-6e20baf08648" (UID: "41ea6070-8986-41ae-98be-6e20baf08648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:10:16.352111 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.352081    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:10:16.420795 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.420760    5508 generic.go:356] "Generic (PLEG): container finished" podID="41ea6070-8986-41ae-98be-6e20baf08648" containerID="98b363f396d884557f47e5d66b21e1fad20ac32bfcf5aa45a1d411ade58ebca1" exitCode=0
Nov 03 02:10:16.421019 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.420999    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp" event={"ID":"41ea6070-8986-41ae-98be-6e20baf08648","Type":"ContainerDied","Data":"98b363f396d884557f47e5d66b21e1fad20ac32bfcf5aa45a1d411ade58ebca1"}
Nov 03 02:10:16.421106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.421092    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp" event={"ID":"41ea6070-8986-41ae-98be-6e20baf08648","Type":"ContainerDied","Data":"0424f6951fb804043623d84edddf926f18e7fd5609dfd224e63f9f0dd242b114"}
Nov 03 02:10:16.421141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.421126    5508 scope.go:122] "RemoveContainer" containerID="98b363f396d884557f47e5d66b21e1fad20ac32bfcf5aa45a1d411ade58ebca1"
Nov 03 02:10:16.421440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.421412    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"
Nov 03 02:10:16.424253 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.424224    5508 generic.go:356] "Generic (PLEG): container finished" podID="3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" containerID="3972a35dd5d8c08eb457fb2f5e30ff451721e7a7d98b3b96c50a3e9ee7458910" exitCode=0
Nov 03 02:10:16.424359 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.424271    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"
Nov 03 02:10:16.424359 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.424290    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk" event={"ID":"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e","Type":"ContainerDied","Data":"3972a35dd5d8c08eb457fb2f5e30ff451721e7a7d98b3b96c50a3e9ee7458910"}
Nov 03 02:10:16.424451 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.424341    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk" event={"ID":"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e","Type":"ContainerDied","Data":"e970889dfca4e232592fbf949a119026500c88472af91e0c4cf68134d13e28d0"}
Nov 03 02:10:16.430644 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.430619    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-credential-socket\") pod \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") "
Nov 03 02:10:16.430763 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.430681    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-certs\") pod \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") "
Nov 03 02:10:16.430763 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.430703    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-podinfo\") pod \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") "
Nov 03 02:10:16.430763 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.430721    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-data\") pod \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") "
Nov 03 02:10:16.431065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.430774    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-kube-api-access-wn9dd\" (UniqueName: \"kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-kube-api-access-wn9dd\") pod \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") "
Nov 03 02:10:16.431065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.430815    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-token\" (UniqueName: \"kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-token\") pod \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") "
Nov 03 02:10:16.431065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.430860    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-envoy\") pod \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") "
Nov 03 02:10:16.431065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.430908    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-socket\") pod \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") "
Nov 03 02:10:16.431065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.430913    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-credential-socket" pod "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" (UID: "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:10:16.431065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.431007    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istiod-ca-cert\") pod \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\" (UID: \"3ef31a9a-62f8-49a2-9a0b-d20cabfa823e\") "
Nov 03 02:10:16.431513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.431444    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-certs" pod "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" (UID: "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:10:16.431513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.431463    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ea6070-8986-41ae-98be-6e20baf08648-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.431513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.431485    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ea6070-8986-41ae-98be-6e20baf08648-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.431513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.431499    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.431513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.431512    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.431751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.431526    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.431751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.431539    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.431751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.431553    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.431751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.431569    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvspd\" (UniqueName: \"kubernetes.io/projected/41ea6070-8986-41ae-98be-6e20baf08648-kube-api-access-mvspd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.431751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.431585    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ea6070-8986-41ae-98be-6e20baf08648-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.431751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.431637    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-socket" pod "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" (UID: "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:10:16.432042 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.431843    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istiod-ca-cert" pod "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" (UID: "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:10:16.432113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.432089    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-data" pod "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" (UID: "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:10:16.433636 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.433611    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-envoy" pod "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" (UID: "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:10:16.433743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.433651    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-kube-api-access-wn9dd" pod "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" (UID: "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e"). InnerVolumeSpecName "kube-api-access-wn9dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:10:16.434017 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.433994    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-token" pod "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" (UID: "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:10:16.434076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.434025    5508 scope.go:122] "RemoveContainer" containerID="98b363f396d884557f47e5d66b21e1fad20ac32bfcf5aa45a1d411ade58ebca1"
Nov 03 02:10:16.434309 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:10:16.434290    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b363f396d884557f47e5d66b21e1fad20ac32bfcf5aa45a1d411ade58ebca1\": container with ID starting with 98b363f396d884557f47e5d66b21e1fad20ac32bfcf5aa45a1d411ade58ebca1 not found: ID does not exist" containerID="98b363f396d884557f47e5d66b21e1fad20ac32bfcf5aa45a1d411ade58ebca1"
Nov 03 02:10:16.434390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.434317    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b363f396d884557f47e5d66b21e1fad20ac32bfcf5aa45a1d411ade58ebca1"} err="failed to get container status \"98b363f396d884557f47e5d66b21e1fad20ac32bfcf5aa45a1d411ade58ebca1\": rpc error: code = NotFound desc = could not find container \"98b363f396d884557f47e5d66b21e1fad20ac32bfcf5aa45a1d411ade58ebca1\": container with ID starting with 98b363f396d884557f47e5d66b21e1fad20ac32bfcf5aa45a1d411ade58ebca1 not found: ID does not exist"
Nov 03 02:10:16.434390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.434333    5508 scope.go:122] "RemoveContainer" containerID="3972a35dd5d8c08eb457fb2f5e30ff451721e7a7d98b3b96c50a3e9ee7458910"
Nov 03 02:10:16.434850 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.434827    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-podinfo" pod "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" (UID: "3ef31a9a-62f8-49a2-9a0b-d20cabfa823e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:10:16.443548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.443528    5508 scope.go:122] "RemoveContainer" containerID="3972a35dd5d8c08eb457fb2f5e30ff451721e7a7d98b3b96c50a3e9ee7458910"
Nov 03 02:10:16.443851 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:10:16.443832    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3972a35dd5d8c08eb457fb2f5e30ff451721e7a7d98b3b96c50a3e9ee7458910\": container with ID starting with 3972a35dd5d8c08eb457fb2f5e30ff451721e7a7d98b3b96c50a3e9ee7458910 not found: ID does not exist" containerID="3972a35dd5d8c08eb457fb2f5e30ff451721e7a7d98b3b96c50a3e9ee7458910"
Nov 03 02:10:16.443915 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.443860    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3972a35dd5d8c08eb457fb2f5e30ff451721e7a7d98b3b96c50a3e9ee7458910"} err="failed to get container status \"3972a35dd5d8c08eb457fb2f5e30ff451721e7a7d98b3b96c50a3e9ee7458910\": rpc error: code = NotFound desc = could not find container \"3972a35dd5d8c08eb457fb2f5e30ff451721e7a7d98b3b96c50a3e9ee7458910\": container with ID starting with 3972a35dd5d8c08eb457fb2f5e30ff451721e7a7d98b3b96c50a3e9ee7458910 not found: ID does not exist"
Nov 03 02:10:16.533032 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.532950    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.533032 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.532981    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.533032 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.532996    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.533032 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.533008    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.533032 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.533023    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn9dd\" (UniqueName: \"kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-kube-api-access-wn9dd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.533453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.533037    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.533453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.533050    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.533453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.533061    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:10:16.743173 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.743106    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"]
Nov 03 02:10:16.748113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.748086    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp"]
Nov 03 02:10:16.850555 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.850437    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"]
Nov 03 02:10:16.855141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:16.855107    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-87mwk"]
Nov 03 02:10:17.217148 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:17.217102    5508 patch_prober.go:29] interesting pod/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:10:17.217339 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:17.217179    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-7s9rp" podUID="41ea6070-8986-41ae-98be-6e20baf08648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:10:17.910805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:17.910768    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:17.923240 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:17.923216    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:18.452220 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:18.452192    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:10:18.694811 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:18.694771    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" path="/var/lib/kubelet/pods/3ef31a9a-62f8-49a2-9a0b-d20cabfa823e/volumes"
Nov 03 02:10:18.695288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:18.695273    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ea6070-8986-41ae-98be-6e20baf08648" path="/var/lib/kubelet/pods/41ea6070-8986-41ae-98be-6e20baf08648/volumes"
Nov 03 02:10:21.882051 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:21.882018    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:10:43.815455 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:10:43.815414    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:10:45.541076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:45.541045    5508 scope.go:122] "RemoveContainer" containerID="946fccc9fceadbec7405c93a6da985530d89677e037d72650bdc7652347c8fcc"
Nov 03 02:10:45.551657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:10:45.551636    5508 scope.go:122] "RemoveContainer" containerID="e8f66b1340724ed188f09ddc02ab0b3c3cc32ee73a860fb44a8720d059724df4"
Nov 03 02:11:03.487015 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:03.486946    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"]
Nov 03 02:11:03.487607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:03.487177    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm" podUID="b0bf6da9-6b35-4113-a0b8-bf0dd61f498e" containerName="maas-api" containerID="cri-o://5ed18d8727b9ecf02a3bdc6a2326c7e1c648543e697717d4c7cdc0739178db86" gracePeriod=30
Nov 03 02:11:03.769743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:03.769655    5508 generic.go:356] "Generic (PLEG): container finished" podID="b0bf6da9-6b35-4113-a0b8-bf0dd61f498e" containerID="5ed18d8727b9ecf02a3bdc6a2326c7e1c648543e697717d4c7cdc0739178db86" exitCode=0
Nov 03 02:11:03.769743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:03.769732    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm" event={"ID":"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e","Type":"ContainerDied","Data":"5ed18d8727b9ecf02a3bdc6a2326c7e1c648543e697717d4c7cdc0739178db86"}
Nov 03 02:11:03.812520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:03.812486    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:11:03.899527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:03.899494    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-metrics-certs\") pod \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") "
Nov 03 02:11:03.899738 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:03.899539    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-maas-api-tls\") pod \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") "
Nov 03 02:11:03.899738 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:03.899626    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-kube-api-access-89nfs\" (UniqueName: \"kubernetes.io/projected/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-kube-api-access-89nfs\") pod \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") "
Nov 03 02:11:03.899883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:03.899806    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-openshift-service-ca\") pod \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\" (UID: \"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e\") "
Nov 03 02:11:03.900272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:03.900239    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-openshift-service-ca" pod "b0bf6da9-6b35-4113-a0b8-bf0dd61f498e" (UID: "b0bf6da9-6b35-4113-a0b8-bf0dd61f498e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:11:03.901765 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:03.901731    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-metrics-certs" pod "b0bf6da9-6b35-4113-a0b8-bf0dd61f498e" (UID: "b0bf6da9-6b35-4113-a0b8-bf0dd61f498e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:11:03.901970 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:03.901948    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-kube-api-access-89nfs" pod "b0bf6da9-6b35-4113-a0b8-bf0dd61f498e" (UID: "b0bf6da9-6b35-4113-a0b8-bf0dd61f498e"). InnerVolumeSpecName "kube-api-access-89nfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:11:03.902176 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:03.902147    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-maas-api-tls" pod "b0bf6da9-6b35-4113-a0b8-bf0dd61f498e" (UID: "b0bf6da9-6b35-4113-a0b8-bf0dd61f498e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:11:04.001376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:04.001324    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:04.001376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:04.001379    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:04.001588 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:04.001389    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:04.001588 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:04.001398    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89nfs\" (UniqueName: \"kubernetes.io/projected/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e-kube-api-access-89nfs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:04.778130 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:04.778100    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"
Nov 03 02:11:04.778650 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:04.778101    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm" event={"ID":"b0bf6da9-6b35-4113-a0b8-bf0dd61f498e","Type":"ContainerDied","Data":"5c3264b7cc7b79580e09aacae35f8b4ddfbe2c88b687153fab71fc229372cf28"}
Nov 03 02:11:04.778650 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:04.778249    5508 scope.go:122] "RemoveContainer" containerID="5ed18d8727b9ecf02a3bdc6a2326c7e1c648543e697717d4c7cdc0739178db86"
Nov 03 02:11:04.932587 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:04.932552    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"]
Nov 03 02:11:04.980813 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:04.980733    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b22fcc-7f54f45b5d-vprgm"]
Nov 03 02:11:05.555799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:05.555760    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"]
Nov 03 02:11:05.559124 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:05.559099    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b22fcc-fhmbh"]
Nov 03 02:11:05.795684 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:05.795651    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"]
Nov 03 02:11:05.796480 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:05.796449    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5" podUID="65a28645-ef66-46b3-8a68-a999694dadc3" containerName="istio-proxy" containerID="cri-o://0df9011d5f6dbc1cd4f44b25e5ede5c8df040268d3f28fe8ffc0b62b4fa1b06d" gracePeriod=30
Nov 03 02:11:06.694413 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:06.694375    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd2e071-eddc-4363-827a-487d7d2a9b46" path="/var/lib/kubelet/pods/3bd2e071-eddc-4363-827a-487d7d2a9b46/volumes"
Nov 03 02:11:06.695010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:06.694989    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bf6da9-6b35-4113-a0b8-bf0dd61f498e" path="/var/lib/kubelet/pods/b0bf6da9-6b35-4113-a0b8-bf0dd61f498e/volumes"
Nov 03 02:11:11.066455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.066385    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:11:11.214492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.214458    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-certs\") pod \"65a28645-ef66-46b3-8a68-a999694dadc3\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") "
Nov 03 02:11:11.214669 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.214537    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65a28645-ef66-46b3-8a68-a999694dadc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a28645-ef66-46b3-8a68-a999694dadc3-istiod-ca-cert\") pod \"65a28645-ef66-46b3-8a68-a999694dadc3\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") "
Nov 03 02:11:11.214669 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.214560    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65a28645-ef66-46b3-8a68-a999694dadc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a28645-ef66-46b3-8a68-a999694dadc3-istio-podinfo\") pod \"65a28645-ef66-46b3-8a68-a999694dadc3\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") "
Nov 03 02:11:11.214669 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.214595    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-envoy\") pod \"65a28645-ef66-46b3-8a68-a999694dadc3\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") "
Nov 03 02:11:11.214669 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.214630    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-kube-api-access-bmtvb\" (UniqueName: \"kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-kube-api-access-bmtvb\") pod \"65a28645-ef66-46b3-8a68-a999694dadc3\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") "
Nov 03 02:11:11.214907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.214669    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-data\") pod \"65a28645-ef66-46b3-8a68-a999694dadc3\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") "
Nov 03 02:11:11.214907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.214724    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-istio-token\" (UniqueName: \"kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-istio-token\") pod \"65a28645-ef66-46b3-8a68-a999694dadc3\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") "
Nov 03 02:11:11.214907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.214769    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-certs" pod "65a28645-ef66-46b3-8a68-a999694dadc3" (UID: "65a28645-ef66-46b3-8a68-a999694dadc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:11:11.214907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.214790    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-credential-socket\") pod \"65a28645-ef66-46b3-8a68-a999694dadc3\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") "
Nov 03 02:11:11.214907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.214820    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-socket\") pod \"65a28645-ef66-46b3-8a68-a999694dadc3\" (UID: \"65a28645-ef66-46b3-8a68-a999694dadc3\") "
Nov 03 02:11:11.215197 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.215015    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-data" pod "65a28645-ef66-46b3-8a68-a999694dadc3" (UID: "65a28645-ef66-46b3-8a68-a999694dadc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:11:11.215197 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.215030    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-credential-socket" pod "65a28645-ef66-46b3-8a68-a999694dadc3" (UID: "65a28645-ef66-46b3-8a68-a999694dadc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:11:11.215197 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.215047    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a28645-ef66-46b3-8a68-a999694dadc3-istiod-ca-cert" pod "65a28645-ef66-46b3-8a68-a999694dadc3" (UID: "65a28645-ef66-46b3-8a68-a999694dadc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:11:11.215381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.215207    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-socket" pod "65a28645-ef66-46b3-8a68-a999694dadc3" (UID: "65a28645-ef66-46b3-8a68-a999694dadc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:11:11.215540 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.215523    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:11.215619 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.215545    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65a28645-ef66-46b3-8a68-a999694dadc3-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:11.215619 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.215558    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:11.215619 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.215571    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:11.215619 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.215588    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:11.216981 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.216955    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-istio-token" pod "65a28645-ef66-46b3-8a68-a999694dadc3" (UID: "65a28645-ef66-46b3-8a68-a999694dadc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:11:11.217084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.216990    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-kube-api-access-bmtvb" pod "65a28645-ef66-46b3-8a68-a999694dadc3" (UID: "65a28645-ef66-46b3-8a68-a999694dadc3"). InnerVolumeSpecName "kube-api-access-bmtvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:11:11.217221 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.217201    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-envoy" pod "65a28645-ef66-46b3-8a68-a999694dadc3" (UID: "65a28645-ef66-46b3-8a68-a999694dadc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:11:11.217291 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.217195    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65a28645-ef66-46b3-8a68-a999694dadc3-istio-podinfo" pod "65a28645-ef66-46b3-8a68-a999694dadc3" (UID: "65a28645-ef66-46b3-8a68-a999694dadc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:11:11.316410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.316378    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:11.316410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.316405    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65a28645-ef66-46b3-8a68-a999694dadc3-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:11.316410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.316415    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65a28645-ef66-46b3-8a68-a999694dadc3-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:11.316684 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.316425    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmtvb\" (UniqueName: \"kubernetes.io/projected/65a28645-ef66-46b3-8a68-a999694dadc3-kube-api-access-bmtvb\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:11.838004 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.837967    5508 generic.go:356] "Generic (PLEG): container finished" podID="65a28645-ef66-46b3-8a68-a999694dadc3" containerID="0df9011d5f6dbc1cd4f44b25e5ede5c8df040268d3f28fe8ffc0b62b4fa1b06d" exitCode=0
Nov 03 02:11:11.838208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.838016    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"
Nov 03 02:11:11.838208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.838055    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5" event={"ID":"65a28645-ef66-46b3-8a68-a999694dadc3","Type":"ContainerDied","Data":"0df9011d5f6dbc1cd4f44b25e5ede5c8df040268d3f28fe8ffc0b62b4fa1b06d"}
Nov 03 02:11:11.838208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.838117    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5" event={"ID":"65a28645-ef66-46b3-8a68-a999694dadc3","Type":"ContainerDied","Data":"2620b136af3580ac99014422caefccab057baf392ee022d23076f21ccf8208b2"}
Nov 03 02:11:11.838208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.838141    5508 scope.go:122] "RemoveContainer" containerID="0df9011d5f6dbc1cd4f44b25e5ede5c8df040268d3f28fe8ffc0b62b4fa1b06d"
Nov 03 02:11:11.848947 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.848929    5508 scope.go:122] "RemoveContainer" containerID="917f79b61998840d91a024d2078e9bd3bdcf7022fb67ef672712812e61a81939"
Nov 03 02:11:11.858876 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.858858    5508 scope.go:122] "RemoveContainer" containerID="0df9011d5f6dbc1cd4f44b25e5ede5c8df040268d3f28fe8ffc0b62b4fa1b06d"
Nov 03 02:11:11.859166 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:11:11.859136    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df9011d5f6dbc1cd4f44b25e5ede5c8df040268d3f28fe8ffc0b62b4fa1b06d\": container with ID starting with 0df9011d5f6dbc1cd4f44b25e5ede5c8df040268d3f28fe8ffc0b62b4fa1b06d not found: ID does not exist" containerID="0df9011d5f6dbc1cd4f44b25e5ede5c8df040268d3f28fe8ffc0b62b4fa1b06d"
Nov 03 02:11:11.859268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.859176    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df9011d5f6dbc1cd4f44b25e5ede5c8df040268d3f28fe8ffc0b62b4fa1b06d"} err="failed to get container status \"0df9011d5f6dbc1cd4f44b25e5ede5c8df040268d3f28fe8ffc0b62b4fa1b06d\": rpc error: code = NotFound desc = could not find container \"0df9011d5f6dbc1cd4f44b25e5ede5c8df040268d3f28fe8ffc0b62b4fa1b06d\": container with ID starting with 0df9011d5f6dbc1cd4f44b25e5ede5c8df040268d3f28fe8ffc0b62b4fa1b06d not found: ID does not exist"
Nov 03 02:11:11.859268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.859204    5508 scope.go:122] "RemoveContainer" containerID="917f79b61998840d91a024d2078e9bd3bdcf7022fb67ef672712812e61a81939"
Nov 03 02:11:11.859506 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:11:11.859480    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"917f79b61998840d91a024d2078e9bd3bdcf7022fb67ef672712812e61a81939\": container with ID starting with 917f79b61998840d91a024d2078e9bd3bdcf7022fb67ef672712812e61a81939 not found: ID does not exist" containerID="917f79b61998840d91a024d2078e9bd3bdcf7022fb67ef672712812e61a81939"
Nov 03 02:11:11.859564 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:11.859514    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917f79b61998840d91a024d2078e9bd3bdcf7022fb67ef672712812e61a81939"} err="failed to get container status \"917f79b61998840d91a024d2078e9bd3bdcf7022fb67ef672712812e61a81939\": rpc error: code = NotFound desc = could not find container \"917f79b61998840d91a024d2078e9bd3bdcf7022fb67ef672712812e61a81939\": container with ID starting with 917f79b61998840d91a024d2078e9bd3bdcf7022fb67ef672712812e61a81939 not found: ID does not exist"
Nov 03 02:11:12.040509 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:12.040473    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"]
Nov 03 02:11:12.044199 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:12.044173    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b22fcc-openshift-default-5dc598f79c-446t5"]
Nov 03 02:11:12.697505 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:12.697461    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a28645-ef66-46b3-8a68-a999694dadc3" path="/var/lib/kubelet/pods/65a28645-ef66-46b3-8a68-a999694dadc3/volumes"
Nov 03 02:11:13.841037 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:11:13.841001    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:11:17.282930 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:17.282838    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:11:43.653103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:43.653058    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"]
Nov 03 02:11:43.653750 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:43.653323    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd" podUID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerName="istio-proxy" containerID="cri-o://728496c3ccf36770c1e878cc6012c57a91144cd483548a5b899fd85ed311c33b" gracePeriod=30
Nov 03 02:11:43.866043 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:11:43.865958    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:11:48.926737 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:48.926668    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:11:49.014203 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.014171    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-data\") pod \"b81bb277-1988-46e5-833e-c48ebe58a4a1\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") "
Nov 03 02:11:49.014413 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.014212    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b81bb277-1988-46e5-833e-c48ebe58a4a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81bb277-1988-46e5-833e-c48ebe58a4a1-istiod-ca-cert\") pod \"b81bb277-1988-46e5-833e-c48ebe58a4a1\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") "
Nov 03 02:11:49.014413 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.014233    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-podinfo\") pod \"b81bb277-1988-46e5-833e-c48ebe58a4a1\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") "
Nov 03 02:11:49.014413 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.014255    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-token\" (UniqueName: \"kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-token\") pod \"b81bb277-1988-46e5-833e-c48ebe58a4a1\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") "
Nov 03 02:11:49.014590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.014433    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-envoy\") pod \"b81bb277-1988-46e5-833e-c48ebe58a4a1\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") "
Nov 03 02:11:49.014590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.014467    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-socket\") pod \"b81bb277-1988-46e5-833e-c48ebe58a4a1\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") "
Nov 03 02:11:49.014590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.014495    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-kube-api-access-pjjnq\" (UniqueName: \"kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-kube-api-access-pjjnq\") pod \"b81bb277-1988-46e5-833e-c48ebe58a4a1\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") "
Nov 03 02:11:49.014590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.014525    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-certs\") pod \"b81bb277-1988-46e5-833e-c48ebe58a4a1\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") "
Nov 03 02:11:49.014590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.014532    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-data" pod "b81bb277-1988-46e5-833e-c48ebe58a4a1" (UID: "b81bb277-1988-46e5-833e-c48ebe58a4a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:11:49.014883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.014619    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-credential-socket\") pod \"b81bb277-1988-46e5-833e-c48ebe58a4a1\" (UID: \"b81bb277-1988-46e5-833e-c48ebe58a4a1\") "
Nov 03 02:11:49.014883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.014656    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81bb277-1988-46e5-833e-c48ebe58a4a1-istiod-ca-cert" pod "b81bb277-1988-46e5-833e-c48ebe58a4a1" (UID: "b81bb277-1988-46e5-833e-c48ebe58a4a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:11:49.014883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.014727    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-socket" pod "b81bb277-1988-46e5-833e-c48ebe58a4a1" (UID: "b81bb277-1988-46e5-833e-c48ebe58a4a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:11:49.015050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.014939    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-certs" pod "b81bb277-1988-46e5-833e-c48ebe58a4a1" (UID: "b81bb277-1988-46e5-833e-c48ebe58a4a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:11:49.015050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.015018    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-credential-socket" pod "b81bb277-1988-46e5-833e-c48ebe58a4a1" (UID: "b81bb277-1988-46e5-833e-c48ebe58a4a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:11:49.015233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.015213    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:49.015302 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.015238    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:49.015302 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.015253    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:49.015302 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.015267    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81bb277-1988-46e5-833e-c48ebe58a4a1-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:49.015302 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.015284    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:49.016622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.016593    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-podinfo" pod "b81bb277-1988-46e5-833e-c48ebe58a4a1" (UID: "b81bb277-1988-46e5-833e-c48ebe58a4a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:11:49.016751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.016690    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-envoy" pod "b81bb277-1988-46e5-833e-c48ebe58a4a1" (UID: "b81bb277-1988-46e5-833e-c48ebe58a4a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:11:49.016813 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.016792    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-kube-api-access-pjjnq" pod "b81bb277-1988-46e5-833e-c48ebe58a4a1" (UID: "b81bb277-1988-46e5-833e-c48ebe58a4a1"). InnerVolumeSpecName "kube-api-access-pjjnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:11:49.016853 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.016826    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-token" pod "b81bb277-1988-46e5-833e-c48ebe58a4a1" (UID: "b81bb277-1988-46e5-833e-c48ebe58a4a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:11:49.110141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.110097    5508 generic.go:356] "Generic (PLEG): container finished" podID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerID="728496c3ccf36770c1e878cc6012c57a91144cd483548a5b899fd85ed311c33b" exitCode=0
Nov 03 02:11:49.110141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.110170    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"
Nov 03 02:11:49.110141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.110190    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd" event={"ID":"b81bb277-1988-46e5-833e-c48ebe58a4a1","Type":"ContainerDied","Data":"728496c3ccf36770c1e878cc6012c57a91144cd483548a5b899fd85ed311c33b"}
Nov 03 02:11:49.110141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.110290    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd" event={"ID":"b81bb277-1988-46e5-833e-c48ebe58a4a1","Type":"ContainerDied","Data":"a287cad25b2cc6b98aac63831717d217131e755626e48605c8f8e750837f30af"}
Nov 03 02:11:49.110141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.110341    5508 scope.go:122] "RemoveContainer" containerID="728496c3ccf36770c1e878cc6012c57a91144cd483548a5b899fd85ed311c33b"
Nov 03 02:11:49.117130 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.117103    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:49.117130 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.117128    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:49.117371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.117137    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81bb277-1988-46e5-833e-c48ebe58a4a1-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:49.117371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.117147    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjjnq\" (UniqueName: \"kubernetes.io/projected/b81bb277-1988-46e5-833e-c48ebe58a4a1-kube-api-access-pjjnq\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:11:49.122015 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.121992    5508 scope.go:122] "RemoveContainer" containerID="c99d6df9768c419c51252268157643548f3f8a800c88e03534a95279b2dfbfc6"
Nov 03 02:11:49.131728 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.131711    5508 scope.go:122] "RemoveContainer" containerID="728496c3ccf36770c1e878cc6012c57a91144cd483548a5b899fd85ed311c33b"
Nov 03 02:11:49.132077 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:11:49.132055    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728496c3ccf36770c1e878cc6012c57a91144cd483548a5b899fd85ed311c33b\": container with ID starting with 728496c3ccf36770c1e878cc6012c57a91144cd483548a5b899fd85ed311c33b not found: ID does not exist" containerID="728496c3ccf36770c1e878cc6012c57a91144cd483548a5b899fd85ed311c33b"
Nov 03 02:11:49.132138 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.132091    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728496c3ccf36770c1e878cc6012c57a91144cd483548a5b899fd85ed311c33b"} err="failed to get container status \"728496c3ccf36770c1e878cc6012c57a91144cd483548a5b899fd85ed311c33b\": rpc error: code = NotFound desc = could not find container \"728496c3ccf36770c1e878cc6012c57a91144cd483548a5b899fd85ed311c33b\": container with ID starting with 728496c3ccf36770c1e878cc6012c57a91144cd483548a5b899fd85ed311c33b not found: ID does not exist"
Nov 03 02:11:49.132138 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.132115    5508 scope.go:122] "RemoveContainer" containerID="c99d6df9768c419c51252268157643548f3f8a800c88e03534a95279b2dfbfc6"
Nov 03 02:11:49.132469 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:11:49.132440    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99d6df9768c419c51252268157643548f3f8a800c88e03534a95279b2dfbfc6\": container with ID starting with c99d6df9768c419c51252268157643548f3f8a800c88e03534a95279b2dfbfc6 not found: ID does not exist" containerID="c99d6df9768c419c51252268157643548f3f8a800c88e03534a95279b2dfbfc6"
Nov 03 02:11:49.132548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.132478    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99d6df9768c419c51252268157643548f3f8a800c88e03534a95279b2dfbfc6"} err="failed to get container status \"c99d6df9768c419c51252268157643548f3f8a800c88e03534a95279b2dfbfc6\": rpc error: code = NotFound desc = could not find container \"c99d6df9768c419c51252268157643548f3f8a800c88e03534a95279b2dfbfc6\": container with ID starting with c99d6df9768c419c51252268157643548f3f8a800c88e03534a95279b2dfbfc6 not found: ID does not exist"
Nov 03 02:11:49.266144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.266032    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"]
Nov 03 02:11:49.269319 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:49.269290    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f7f61d-openshift-default-5d6dffcb5c-zlgtd"]
Nov 03 02:11:50.693673 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:50.693635    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b81bb277-1988-46e5-833e-c48ebe58a4a1" path="/var/lib/kubelet/pods/b81bb277-1988-46e5-833e-c48ebe58a4a1/volumes"
Nov 03 02:11:57.125083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125051    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"]
Nov 03 02:11:57.125616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125602    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerName="istio-proxy"
Nov 03 02:11:57.125671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125617    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerName="istio-proxy"
Nov 03 02:11:57.125671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125629    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerName="istio-proxy"
Nov 03 02:11:57.125671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125635    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerName="istio-proxy"
Nov 03 02:11:57.125671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125647    5508 cpu_manager.go:414] "containerMap: removing container" podUID="65a28645-ef66-46b3-8a68-a999694dadc3" containerName="istio-proxy"
Nov 03 02:11:57.125671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125652    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a28645-ef66-46b3-8a68-a999694dadc3" containerName="istio-proxy"
Nov 03 02:11:57.125671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125672    5508 cpu_manager.go:414] "containerMap: removing container" podUID="65a28645-ef66-46b3-8a68-a999694dadc3" containerName="istio-proxy"
Nov 03 02:11:57.125927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125677    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a28645-ef66-46b3-8a68-a999694dadc3" containerName="istio-proxy"
Nov 03 02:11:57.125927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125687    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b0bf6da9-6b35-4113-a0b8-bf0dd61f498e" containerName="maas-api"
Nov 03 02:11:57.125927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125691    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bf6da9-6b35-4113-a0b8-bf0dd61f498e" containerName="maas-api"
Nov 03 02:11:57.125927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125704    5508 cpu_manager.go:414] "containerMap: removing container" podUID="41ea6070-8986-41ae-98be-6e20baf08648" containerName="istio-proxy"
Nov 03 02:11:57.125927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125708    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ea6070-8986-41ae-98be-6e20baf08648" containerName="istio-proxy"
Nov 03 02:11:57.125927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125720    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" containerName="istio-proxy"
Nov 03 02:11:57.125927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125725    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" containerName="istio-proxy"
Nov 03 02:11:57.125927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125796    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef31a9a-62f8-49a2-9a0b-d20cabfa823e" containerName="istio-proxy"
Nov 03 02:11:57.125927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125807    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerName="istio-proxy"
Nov 03 02:11:57.125927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125815    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b81bb277-1988-46e5-833e-c48ebe58a4a1" containerName="istio-proxy"
Nov 03 02:11:57.125927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125823    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a28645-ef66-46b3-8a68-a999694dadc3" containerName="istio-proxy"
Nov 03 02:11:57.125927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125832    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bf6da9-6b35-4113-a0b8-bf0dd61f498e" containerName="maas-api"
Nov 03 02:11:57.125927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.125843    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ea6070-8986-41ae-98be-6e20baf08648" containerName="istio-proxy"
Nov 03 02:11:57.130957 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.130935    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.179972 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.179928    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6a5d73\"/\"openshift-service-ca.crt\""
Nov 03 02:11:57.214981 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.214933    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-dshm\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.215186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.214990    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78e0b60-4511-45ab-914c-0d4330324f9a-tls-certs\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.215186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.215080    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-model-cache\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.215186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.215132    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2tm\" (UniqueName: \"kubernetes.io/projected/f78e0b60-4511-45ab-914c-0d4330324f9a-kube-api-access-9c2tm\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.215186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.215160    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-home\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.215490 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.215267    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-tmp-dir\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.226085 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.226041    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6a5d73\"/\"default-dockercfg-hgwfm\""
Nov 03 02:11:57.226085 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.226041    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-6a5d73\"/\"kube-root-ca.crt\""
Nov 03 02:11:57.226294 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.226132    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-6a5d73\"/\"auth-test-model-6a5d73-kserve-self-signed-certs\""
Nov 03 02:11:57.231233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.231203    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"]
Nov 03 02:11:57.317046 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.317008    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-dshm\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.317046 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.317057    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78e0b60-4511-45ab-914c-0d4330324f9a-tls-certs\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.317330 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.317174    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-model-cache\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.317330 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.317255    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2tm\" (UniqueName: \"kubernetes.io/projected/f78e0b60-4511-45ab-914c-0d4330324f9a-kube-api-access-9c2tm\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.317330 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.317300    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-home\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.317482 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.317403    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-tmp-dir\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.317622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.317602    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-model-cache\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.317812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.317774    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-tmp-dir\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.318036 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.318013    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-home\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.319536 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.319506    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-dshm\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.319712 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.319694    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78e0b60-4511-45ab-914c-0d4330324f9a-tls-certs\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.375553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.375472    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2tm\" (UniqueName: \"kubernetes.io/projected/f78e0b60-4511-45ab-914c-0d4330324f9a-kube-api-access-9c2tm\") pod \"auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") " pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.442890 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.442854    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:11:57.603310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.603260    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:11:57.831527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:57.831490    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"]
Nov 03 02:11:57.838363 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:11:57.838322    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf78e0b60_4511_45ab_914c_0d4330324f9a.slice/crio-dcf37d045809ec00d5c340e1a17586ec0344d8b9917b15df1659f47ed167ddcb WatchSource:0}: Error finding container dcf37d045809ec00d5c340e1a17586ec0344d8b9917b15df1659f47ed167ddcb: Status 404 returned error can't find the container with id dcf37d045809ec00d5c340e1a17586ec0344d8b9917b15df1659f47ed167ddcb
Nov 03 02:11:58.175030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:58.174995    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w" event={"ID":"f78e0b60-4511-45ab-914c-0d4330324f9a","Type":"ContainerStarted","Data":"a196e9f1daf171066ae850b3141a9f0e13b514dc4f901b06c9a6259aaf0dbf1b"}
Nov 03 02:11:58.175462 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:58.175039    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w" event={"ID":"f78e0b60-4511-45ab-914c-0d4330324f9a","Type":"ContainerStarted","Data":"dcf37d045809ec00d5c340e1a17586ec0344d8b9917b15df1659f47ed167ddcb"}
Nov 03 02:11:58.285965 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:58.285908    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w" podStartSLOduration=1.2858951969999999 podStartE2EDuration="1.285895197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:11:58.284159661 +0000 UTC m=+3197.709766125" watchObservedRunningTime="2026-11-03 02:11:58.285895197 +0000 UTC m=+3197.711501709"
Nov 03 02:11:58.444027 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:11:58.443940    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:12:04.102100 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.102051    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"]
Nov 03 02:12:04.102996 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.102970    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a28645-ef66-46b3-8a68-a999694dadc3" containerName="istio-proxy"
Nov 03 02:12:04.106955 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.106932    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.163985 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.163951    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-b16721-openshift-default-dockercfg-w8nqd\""
Nov 03 02:12:04.232393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.217744    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-data\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.232393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.217842    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcbd9\" (UniqueName: \"kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-kube-api-access-gcbd9\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.232393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.217880    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-certs\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.232393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.217978    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-podinfo\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.232393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.218018    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-envoy\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.232393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.218138    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-token\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.232393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.218213    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e105e299-ee4c-4664-ab9c-39e42f762b8d-istiod-ca-cert\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.232393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.218255    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-credential-socket\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.232393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.218282    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-socket\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.299718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.299674    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"]
Nov 03 02:12:04.319782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.319752    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-podinfo\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.319965 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.319792    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-envoy\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.319965 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.319866    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-token\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.319965 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.319914    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e105e299-ee4c-4664-ab9c-39e42f762b8d-istiod-ca-cert\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.320112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.319974    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-credential-socket\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.320112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.320001    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-socket\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.320112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.320032    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-data\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.320326 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.320233    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcbd9\" (UniqueName: \"kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-kube-api-access-gcbd9\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.320326 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.320283    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-certs\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.320528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.320417    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-credential-socket\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.320649 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.320625    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-socket\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.320808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.320720    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-data\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.320808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.320788    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e105e299-ee4c-4664-ab9c-39e42f762b8d-istiod-ca-cert\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.321815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.321301    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-certs\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.323104 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.323016    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-envoy\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.323454 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.323423    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-podinfo\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.377635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.377498    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-token\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.377812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.377786    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcbd9\" (UniqueName: \"kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-kube-api-access-gcbd9\") pod \"e2e-explicit-b16721-openshift-default-74984b654f-xqftp\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") " pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.421414 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.421387    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:04.914296 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.914259    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"]
Nov 03 02:12:04.920663 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:12:04.920634    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode105e299_ee4c_4664_ab9c_39e42f762b8d.slice/crio-dc4730e65ae4bb0eabb78b46377464b8a9d0d1188c3e9f0ce269776678708be7 WatchSource:0}: Error finding container dc4730e65ae4bb0eabb78b46377464b8a9d0d1188c3e9f0ce269776678708be7: Status 404 returned error can't find the container with id dc4730e65ae4bb0eabb78b46377464b8a9d0d1188c3e9f0ce269776678708be7
Nov 03 02:12:04.923462 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.923430    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:12:04.923545 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.923497    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:12:04.923545 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:04.923526    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:12:05.271425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:05.271323    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" event={"ID":"e105e299-ee4c-4664-ab9c-39e42f762b8d","Type":"ContainerStarted","Data":"0fe2152ddfaebebb287fabc443623ab2f23ed7ec3b04d0feb7b94c22dbbb3c04"}
Nov 03 02:12:05.271912 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:05.271434    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" event={"ID":"e105e299-ee4c-4664-ab9c-39e42f762b8d","Type":"ContainerStarted","Data":"dc4730e65ae4bb0eabb78b46377464b8a9d0d1188c3e9f0ce269776678708be7"}
Nov 03 02:12:05.388544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:05.388470    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" podStartSLOduration=1.388451594 podStartE2EDuration="1.388451594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:12:05.384195825 +0000 UTC m=+3204.809802290" watchObservedRunningTime="2026-11-03 02:12:05.388451594 +0000 UTC m=+3204.814058060"
Nov 03 02:12:05.421832 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:05.421788    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:05.423606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:05.423573    5508 patch_prober.go:29] interesting pod/e2e-explicit-b16721-openshift-default-74984b654f-xqftp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Nov 03 02:12:05.423749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:05.423632    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Nov 03 02:12:06.427839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:06.427801    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:07.285845 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:07.285815    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:07.286928 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:07.286906    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:12:07.445574 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:07.445536    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:12:07.464131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:07.464104    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:12:08.303045 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:08.303004    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:12:12.792742 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:12.792703    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"]
Nov 03 02:12:12.797118 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:12.797094    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"
Nov 03 02:12:12.846917 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:12.846577    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b16721\""
Nov 03 02:12:12.903373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:12.897570    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b16721-dockercfg-fzqxb\""
Nov 03 02:12:12.946382 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:12.943710    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"]
Nov 03 02:12:12.957006 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:12.956720    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79\" (UID: \"7513eb12-2a38-43d0-b4cc-06a78e5ca89f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"
Nov 03 02:12:12.958320 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:12.958291    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j4hd\" (UniqueName: \"kubernetes.io/projected/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-kube-api-access-6j4hd\") pod \"payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79\" (UID: \"7513eb12-2a38-43d0-b4cc-06a78e5ca89f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"
Nov 03 02:12:12.976460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:12.976430    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"]
Nov 03 02:12:13.007607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.007583    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"
Nov 03 02:12:13.029611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.029582    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"]
Nov 03 02:12:13.060130 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.060043    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j4hd\" (UniqueName: \"kubernetes.io/projected/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-kube-api-access-6j4hd\") pod \"payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79\" (UID: \"7513eb12-2a38-43d0-b4cc-06a78e5ca89f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"
Nov 03 02:12:13.060452 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.060416    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq\" (UID: \"d9e4a089-412b-4ed3-8dbf-abc2ab20d453\") " pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"
Nov 03 02:12:13.060561 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.060475    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79\" (UID: \"7513eb12-2a38-43d0-b4cc-06a78e5ca89f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"
Nov 03 02:12:13.061237 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.061213    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79\" (UID: \"7513eb12-2a38-43d0-b4cc-06a78e5ca89f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"
Nov 03 02:12:13.063367 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.060622    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92tcs\" (UniqueName: \"kubernetes.io/projected/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-kube-api-access-92tcs\") pod \"payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq\" (UID: \"d9e4a089-412b-4ed3-8dbf-abc2ab20d453\") " pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"
Nov 03 02:12:13.117586 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.117554    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j4hd\" (UniqueName: \"kubernetes.io/projected/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-kube-api-access-6j4hd\") pod \"payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79\" (UID: \"7513eb12-2a38-43d0-b4cc-06a78e5ca89f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"
Nov 03 02:12:13.164386 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.164327    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92tcs\" (UniqueName: \"kubernetes.io/projected/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-kube-api-access-92tcs\") pod \"payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq\" (UID: \"d9e4a089-412b-4ed3-8dbf-abc2ab20d453\") " pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"
Nov 03 02:12:13.164535 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.164493    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq\" (UID: \"d9e4a089-412b-4ed3-8dbf-abc2ab20d453\") " pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"
Nov 03 02:12:13.165198 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.165178    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq\" (UID: \"d9e4a089-412b-4ed3-8dbf-abc2ab20d453\") " pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"
Nov 03 02:12:13.222722 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.222688    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92tcs\" (UniqueName: \"kubernetes.io/projected/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-kube-api-access-92tcs\") pod \"payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq\" (UID: \"d9e4a089-412b-4ed3-8dbf-abc2ab20d453\") " pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"
Nov 03 02:12:13.322430 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.322322    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"
Nov 03 02:12:13.412317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.412105    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"
Nov 03 02:12:13.698011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.697969    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"]
Nov 03 02:12:13.704983 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:12:13.704951    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e4a089_412b_4ed3_8dbf_abc2ab20d453.slice/crio-3fe19b68cae4a2c5ee547f551ba7ac44305154d8eee5d023123f8403671bd5be WatchSource:0}: Error finding container 3fe19b68cae4a2c5ee547f551ba7ac44305154d8eee5d023123f8403671bd5be: Status 404 returned error can't find the container with id 3fe19b68cae4a2c5ee547f551ba7ac44305154d8eee5d023123f8403671bd5be
Nov 03 02:12:13.803388 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:13.803341    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"]
Nov 03 02:12:13.809916 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:12:13.809889    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7513eb12_2a38_43d0_b4cc_06a78e5ca89f.slice/crio-448f865a335f35667e3bdd7728cc17ff8d48575bbfbed8658bcdb757d363042b WatchSource:0}: Error finding container 448f865a335f35667e3bdd7728cc17ff8d48575bbfbed8658bcdb757d363042b: Status 404 returned error can't find the container with id 448f865a335f35667e3bdd7728cc17ff8d48575bbfbed8658bcdb757d363042b
Nov 03 02:12:13.816775 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:12:13.816738    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:12:14.334063 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:14.334023    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq" event={"ID":"d9e4a089-412b-4ed3-8dbf-abc2ab20d453","Type":"ContainerStarted","Data":"318811fb345cd4827446b1c72e3c23885a953444360c4e45c7718fc10ecb0518"}
Nov 03 02:12:14.334285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:14.334123    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq" event={"ID":"d9e4a089-412b-4ed3-8dbf-abc2ab20d453","Type":"ContainerStarted","Data":"3fe19b68cae4a2c5ee547f551ba7ac44305154d8eee5d023123f8403671bd5be"}
Nov 03 02:12:14.334285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:14.334235    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"
Nov 03 02:12:14.335850 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:14.335829    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79" event={"ID":"7513eb12-2a38-43d0-b4cc-06a78e5ca89f","Type":"ContainerStarted","Data":"56635ffd271e541296f01367d0ddcd443300639b3334c463343be5ae17f60b16"}
Nov 03 02:12:14.335987 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:14.335866    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79" event={"ID":"7513eb12-2a38-43d0-b4cc-06a78e5ca89f","Type":"ContainerStarted","Data":"448f865a335f35667e3bdd7728cc17ff8d48575bbfbed8658bcdb757d363042b"}
Nov 03 02:12:14.335987 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:14.335922    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"
Nov 03 02:12:14.445989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:14.445871    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq" podStartSLOduration=2.44585286 podStartE2EDuration="2.44585286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:12:14.44151556 +0000 UTC m=+3213.867122046" watchObservedRunningTime="2026-11-03 02:12:14.44585286 +0000 UTC m=+3213.871459326"
Nov 03 02:12:25.345505 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:25.345474    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"
Nov 03 02:12:25.346021 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:25.345560    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"
Nov 03 02:12:25.457058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:25.456998    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79" podStartSLOduration=13.456982233 podStartE2EDuration="13.456982233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:12:14.557017704 +0000 UTC m=+3213.982624181" watchObservedRunningTime="2026-11-03 02:12:25.456982233 +0000 UTC m=+3224.882588740"
Nov 03 02:12:43.813170 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:12:43.813130    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:12:51.936139 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:12:51.936102    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:13:13.813684 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:13:13.813645    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:13:16.179957 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.179920    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"]
Nov 03 02:13:16.184157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.184137    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:16.232955 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.232920    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-b16721-serving-cert\""
Nov 03 02:13:16.282550 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.282514    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"]
Nov 03 02:13:16.292344 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.292289    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db97d4e-dbd2-4135-ad39-f4f992b87f07-openshift-service-ca\") pod \"maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:16.292648 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.292626    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlprw\" (UniqueName: \"kubernetes.io/projected/5db97d4e-dbd2-4135-ad39-f4f992b87f07-kube-api-access-nlprw\") pod \"maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:16.292905 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.292879    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-metrics-certs\") pod \"maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:16.293009 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.293001    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-maas-api-tls\") pod \"maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:16.395160 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.395120    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-metrics-certs\") pod \"maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:16.395621 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.395595    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-maas-api-tls\") pod \"maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:16.395737 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.395678    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db97d4e-dbd2-4135-ad39-f4f992b87f07-openshift-service-ca\") pod \"maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:16.395737 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.395724    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlprw\" (UniqueName: \"kubernetes.io/projected/5db97d4e-dbd2-4135-ad39-f4f992b87f07-kube-api-access-nlprw\") pod \"maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:16.395849 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:13:16.395782    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-serving-cert: secret "maas-api-e2e-explicit-b16721-serving-cert" not found
Nov 03 02:13:16.395909 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:13:16.395853    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-maas-api-tls podName:5db97d4e-dbd2-4135-ad39-f4f992b87f07 nodeName:}" failed. No retries permitted until 2026-11-03 02:13:16.895832381 +0000 UTC m=+3276.321438824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-maas-api-tls") pod "maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5" (UID: "5db97d4e-dbd2-4135-ad39-f4f992b87f07") : secret "maas-api-e2e-explicit-b16721-serving-cert" not found
Nov 03 02:13:16.396503 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.396479    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db97d4e-dbd2-4135-ad39-f4f992b87f07-openshift-service-ca\") pod \"maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:16.398166 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.398145    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-metrics-certs\") pod \"maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:16.459826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.459748    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlprw\" (UniqueName: \"kubernetes.io/projected/5db97d4e-dbd2-4135-ad39-f4f992b87f07-kube-api-access-nlprw\") pod \"maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:16.905496 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.905394    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-maas-api-tls\") pod \"maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:16.907770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:16.907749    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-maas-api-tls\") pod \"maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:17.097928 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:17.097895    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:17.478028 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:17.477933    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"]
Nov 03 02:13:17.485984 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:13:17.485949    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db97d4e_dbd2_4135_ad39_f4f992b87f07.slice/crio-696ccc16a3e04f1ac5a46b657904dc1b056a4012a7e090a8511a165248578931 WatchSource:0}: Error finding container 696ccc16a3e04f1ac5a46b657904dc1b056a4012a7e090a8511a165248578931: Status 404 returned error can't find the container with id 696ccc16a3e04f1ac5a46b657904dc1b056a4012a7e090a8511a165248578931
Nov 03 02:13:17.487797 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:17.487774    5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:13:17.845690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:17.845598    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5" event={"ID":"5db97d4e-dbd2-4135-ad39-f4f992b87f07","Type":"ContainerStarted","Data":"696ccc16a3e04f1ac5a46b657904dc1b056a4012a7e090a8511a165248578931"}
Nov 03 02:13:18.854661 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:18.854560    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5" event={"ID":"5db97d4e-dbd2-4135-ad39-f4f992b87f07","Type":"ContainerStarted","Data":"34355e37bcb1098854c98654647cd3978545065f4c10142c61d11d96bb8794c0"}
Nov 03 02:13:18.854661 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:18.854651    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:18.966517 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:18.966463    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5" podStartSLOduration=1.9408093050000002 podStartE2EDuration="2.966448522s" totalImagesPullingTime="1.025639217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:13:18.965639459 +0000 UTC m=+3278.391245924" watchObservedRunningTime="2026-11-03 02:13:18.966448522 +0000 UTC m=+3278.392055025"
Nov 03 02:13:24.871515 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:24.871483    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:13:41.341769 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:41.341741    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:13:41.410276 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:41.410251    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:13:43.813807 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:13:43.813770    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:13:48.960555 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:13:48.960515    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:14:01.230518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:01.230484    5508 generic.go:356] "Generic (PLEG): container finished" podID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerID="0fe2152ddfaebebb287fabc443623ab2f23ed7ec3b04d0feb7b94c22dbbb3c04" exitCode=0
Nov 03 02:14:01.230978 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:01.230561    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" event={"ID":"e105e299-ee4c-4664-ab9c-39e42f762b8d","Type":"ContainerDied","Data":"0fe2152ddfaebebb287fabc443623ab2f23ed7ec3b04d0feb7b94c22dbbb3c04"}
Nov 03 02:14:01.231026 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:01.230988    5508 scope.go:122] "RemoveContainer" containerID="0fe2152ddfaebebb287fabc443623ab2f23ed7ec3b04d0feb7b94c22dbbb3c04"
Nov 03 02:14:01.232544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:01.232514    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:14:01.232611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:01.232589    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:14:01.232650 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:01.232620    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:14:02.239127 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:02.239093    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" event={"ID":"e105e299-ee4c-4664-ab9c-39e42f762b8d","Type":"ContainerStarted","Data":"9d95e9e693b489343c4191ff85897638bd2a0ce1a461adebb4ffd991672d895d"}
Nov 03 02:14:02.422114 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:02.422078    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:14:02.422312 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:02.422244    5508 patch_prober.go:29] interesting pod/e2e-explicit-b16721-openshift-default-74984b654f-xqftp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Nov 03 02:14:02.422312 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:02.422295    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Nov 03 02:14:03.421994 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:03.421959    5508 patch_prober.go:29] interesting pod/e2e-explicit-b16721-openshift-default-74984b654f-xqftp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Nov 03 02:14:03.422420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:03.422009    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Nov 03 02:14:04.422091 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:04.422056    5508 patch_prober.go:29] interesting pod/e2e-explicit-b16721-openshift-default-74984b654f-xqftp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Nov 03 02:14:04.422521 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:04.422111    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Nov 03 02:14:05.045465 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:05.045424    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"]
Nov 03 02:14:05.092106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:05.092069    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4c77c7-dnw9b"]
Nov 03 02:14:05.422729 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:05.422694    5508 patch_prober.go:29] interesting pod/e2e-explicit-b16721-openshift-default-74984b654f-xqftp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Nov 03 02:14:05.423156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:05.422750    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Nov 03 02:14:06.422518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:06.422475    5508 patch_prober.go:29] interesting pod/e2e-explicit-b16721-openshift-default-74984b654f-xqftp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Nov 03 02:14:06.422715 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:06.422556    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Nov 03 02:14:06.693295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:06.693204    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9ea631-7c13-46c8-831a-89e4ac4c6776" path="/var/lib/kubelet/pods/2a9ea631-7c13-46c8-831a-89e4ac4c6776/volumes"
Nov 03 02:14:07.286729 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:07.286687    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:14:07.489701 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:07.489668    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:14:07.491964 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:07.491931    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:14:13.815738 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:14:13.815701    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:14:36.786174 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:36.786136    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"]
Nov 03 02:14:36.787059 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:36.786428    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b" podUID="fad7557e-8b17-47c6-8ae5-86aee55e2c79" containerName="istio-proxy" containerID="cri-o://c066a37ec8945da17654abcc599bf8792d1813113df4627e4fa970aa798f3dba" gracePeriod=30
Nov 03 02:14:39.335476 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:39.335441    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:14:42.063528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.063463    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:14:42.203970 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.203931    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-data\") pod \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") "
Nov 03 02:14:42.204185 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204014    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-credential-socket\") pod \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") "
Nov 03 02:14:42.204185 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204055    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-socket\") pod \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") "
Nov 03 02:14:42.204185 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204080    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-podinfo\") pod \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") "
Nov 03 02:14:42.204185 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204137    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-envoy\") pod \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") "
Nov 03 02:14:42.204460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204191    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-kube-api-access-q64gn\" (UniqueName: \"kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-kube-api-access-q64gn\") pod \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") "
Nov 03 02:14:42.204460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204224    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istiod-ca-cert\") pod \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") "
Nov 03 02:14:42.204460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204306    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-token\" (UniqueName: \"kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-token\") pod \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") "
Nov 03 02:14:42.204460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204307    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-credential-socket" pod "fad7557e-8b17-47c6-8ae5-86aee55e2c79" (UID: "fad7557e-8b17-47c6-8ae5-86aee55e2c79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:14:42.204460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204321    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-socket" pod "fad7557e-8b17-47c6-8ae5-86aee55e2c79" (UID: "fad7557e-8b17-47c6-8ae5-86aee55e2c79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:14:42.204460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204334    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-data" pod "fad7557e-8b17-47c6-8ae5-86aee55e2c79" (UID: "fad7557e-8b17-47c6-8ae5-86aee55e2c79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:14:42.204460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204404    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-certs\") pod \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\" (UID: \"fad7557e-8b17-47c6-8ae5-86aee55e2c79\") "
Nov 03 02:14:42.204847 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204650    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istiod-ca-cert" pod "fad7557e-8b17-47c6-8ae5-86aee55e2c79" (UID: "fad7557e-8b17-47c6-8ae5-86aee55e2c79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:14:42.204847 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204670    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-certs" pod "fad7557e-8b17-47c6-8ae5-86aee55e2c79" (UID: "fad7557e-8b17-47c6-8ae5-86aee55e2c79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:14:42.205022 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.204997    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:14:42.205022 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.205021    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:14:42.205259 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.205031    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:14:42.205259 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.205046    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:14:42.205259 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.205062    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:14:42.206432 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.206407    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-podinfo" pod "fad7557e-8b17-47c6-8ae5-86aee55e2c79" (UID: "fad7557e-8b17-47c6-8ae5-86aee55e2c79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:14:42.206583 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.206563    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-envoy" pod "fad7557e-8b17-47c6-8ae5-86aee55e2c79" (UID: "fad7557e-8b17-47c6-8ae5-86aee55e2c79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:14:42.206835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.206811    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-token" pod "fad7557e-8b17-47c6-8ae5-86aee55e2c79" (UID: "fad7557e-8b17-47c6-8ae5-86aee55e2c79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:14:42.206929 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.206864    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-kube-api-access-q64gn" pod "fad7557e-8b17-47c6-8ae5-86aee55e2c79" (UID: "fad7557e-8b17-47c6-8ae5-86aee55e2c79"). InnerVolumeSpecName "kube-api-access-q64gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:14:42.306518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.306484    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:14:42.306518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.306513    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:14:42.306518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.306523    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q64gn\" (UniqueName: \"kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-kube-api-access-q64gn\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:14:42.306874 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.306534    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fad7557e-8b17-47c6-8ae5-86aee55e2c79-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:14:42.558929 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.558891    5508 generic.go:356] "Generic (PLEG): container finished" podID="fad7557e-8b17-47c6-8ae5-86aee55e2c79" containerID="c066a37ec8945da17654abcc599bf8792d1813113df4627e4fa970aa798f3dba" exitCode=0
Nov 03 02:14:42.559153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.558951    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"
Nov 03 02:14:42.559153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.558984    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b" event={"ID":"fad7557e-8b17-47c6-8ae5-86aee55e2c79","Type":"ContainerDied","Data":"c066a37ec8945da17654abcc599bf8792d1813113df4627e4fa970aa798f3dba"}
Nov 03 02:14:42.559153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.559055    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b" event={"ID":"fad7557e-8b17-47c6-8ae5-86aee55e2c79","Type":"ContainerDied","Data":"bdcecc3c9cac5197f4283bcc740efa8c385e1ea16102e6e49b76dde143b25e50"}
Nov 03 02:14:42.559153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.559085    5508 scope.go:122] "RemoveContainer" containerID="c066a37ec8945da17654abcc599bf8792d1813113df4627e4fa970aa798f3dba"
Nov 03 02:14:42.570489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.570464    5508 scope.go:122] "RemoveContainer" containerID="c066a37ec8945da17654abcc599bf8792d1813113df4627e4fa970aa798f3dba"
Nov 03 02:14:42.570788 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:14:42.570765    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c066a37ec8945da17654abcc599bf8792d1813113df4627e4fa970aa798f3dba\": container with ID starting with c066a37ec8945da17654abcc599bf8792d1813113df4627e4fa970aa798f3dba not found: ID does not exist" containerID="c066a37ec8945da17654abcc599bf8792d1813113df4627e4fa970aa798f3dba"
Nov 03 02:14:42.570838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.570795    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c066a37ec8945da17654abcc599bf8792d1813113df4627e4fa970aa798f3dba"} err="failed to get container status \"c066a37ec8945da17654abcc599bf8792d1813113df4627e4fa970aa798f3dba\": rpc error: code = NotFound desc = could not find container \"c066a37ec8945da17654abcc599bf8792d1813113df4627e4fa970aa798f3dba\": container with ID starting with c066a37ec8945da17654abcc599bf8792d1813113df4627e4fa970aa798f3dba not found: ID does not exist"
Nov 03 02:14:42.757808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.757774    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"]
Nov 03 02:14:42.807401 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:42.806257    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0bf34a-openshift-default-7dcc4cb77-rb55b"]
Nov 03 02:14:43.363328 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.363294    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"]
Nov 03 02:14:43.364250 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.364111    5508 cpu_manager.go:414] "containerMap: removing container" podUID="fad7557e-8b17-47c6-8ae5-86aee55e2c79" containerName="istio-proxy"
Nov 03 02:14:43.364250 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.364132    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad7557e-8b17-47c6-8ae5-86aee55e2c79" containerName="istio-proxy"
Nov 03 02:14:43.364506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.364287    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad7557e-8b17-47c6-8ae5-86aee55e2c79" containerName="istio-proxy"
Nov 03 02:14:43.369735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.369712    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.419066 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.419013    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c70365-openshift-default-dockercfg-kfq8b\""
Nov 03 02:14:43.429118 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.429084    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"]
Nov 03 02:14:43.436797 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.436769    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-certs\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.436963 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.436804    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-socket\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.436963 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.436831    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-istio-token\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.437056 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.436996    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320b0833-f978-4d66-9d40-8693e461f63b-istiod-ca-cert\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.437056 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.437022    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-credential-socket\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.437150 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.437126    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-envoy\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.437199 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.437164    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4pms\" (UniqueName: \"kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-kube-api-access-k4pms\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.437328 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.437308    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-data\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.437426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.437342    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320b0833-f978-4d66-9d40-8693e461f63b-istio-podinfo\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.548377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.542128    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-envoy\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.548377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.542221    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4pms\" (UniqueName: \"kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-kube-api-access-k4pms\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.548377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.542326    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-data\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.548377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.542367    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320b0833-f978-4d66-9d40-8693e461f63b-istio-podinfo\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.548377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.542435    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-certs\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.548377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.542464    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-socket\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.548377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.542495    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-istio-token\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.548377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.542590    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320b0833-f978-4d66-9d40-8693e461f63b-istiod-ca-cert\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.548377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.542633    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-credential-socket\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.548377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.543165    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-credential-socket\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.548377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.543764    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-certs\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.548377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.544289    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-data\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.553688 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.550073    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-socket\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.553688 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.550752    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320b0833-f978-4d66-9d40-8693e461f63b-istiod-ca-cert\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.553688 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.552279    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320b0833-f978-4d66-9d40-8693e461f63b-istio-podinfo\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.554127 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.554103    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-envoy\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.600865 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.600835    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4pms\" (UniqueName: \"kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-kube-api-access-k4pms\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.606159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.606133    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-istio-token\") pod \"e2e-extmod-a-c70365-openshift-default-5674757df-ng472\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") " pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.687507 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:43.687472    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:43.825044 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:14:43.825005    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:14:44.158430 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:44.158399    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"]
Nov 03 02:14:44.164600 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:14:44.164574    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320b0833_f978_4d66_9d40_8693e461f63b.slice/crio-7760f17bfaf73ff5fb57091a77225b3f4f1353f238f752738006fe886fca8f5a WatchSource:0}: Error finding container 7760f17bfaf73ff5fb57091a77225b3f4f1353f238f752738006fe886fca8f5a: Status 404 returned error can't find the container with id 7760f17bfaf73ff5fb57091a77225b3f4f1353f238f752738006fe886fca8f5a
Nov 03 02:14:44.167104 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:44.167072    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:14:44.167185 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:44.167141    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:14:44.167185 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:44.167173    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:14:44.582006 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:44.581921    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" event={"ID":"320b0833-f978-4d66-9d40-8693e461f63b","Type":"ContainerStarted","Data":"f3193bca4642683df7d464ac01275425c055ecece98424cb47e1c1818df965b1"}
Nov 03 02:14:44.582006 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:44.581981    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" event={"ID":"320b0833-f978-4d66-9d40-8693e461f63b","Type":"ContainerStarted","Data":"7760f17bfaf73ff5fb57091a77225b3f4f1353f238f752738006fe886fca8f5a"}
Nov 03 02:14:44.695060 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:44.694995    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad7557e-8b17-47c6-8ae5-86aee55e2c79" path="/var/lib/kubelet/pods/fad7557e-8b17-47c6-8ae5-86aee55e2c79/volumes"
Nov 03 02:14:44.695590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:44.695569    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:44.696987 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:44.696941    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" podStartSLOduration=1.696923774 podStartE2EDuration="1.696923774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:44.694094127 +0000 UTC m=+3364.119700594" watchObservedRunningTime="2026-11-03 02:14:44.696923774 +0000 UTC m=+3364.122530241"
Nov 03 02:14:45.694721 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:45.694690    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:46.599961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:46.599929    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:46.601065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:46.601044    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:14:51.323371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:51.322144    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"]
Nov 03 02:14:51.334495 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:51.329422    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"
Nov 03 02:14:51.366775 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:51.366727    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"]
Nov 03 02:14:51.407131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:51.407095    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c70365\""
Nov 03 02:14:51.407131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:51.407095    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c70365-dockercfg-vbpvs\""
Nov 03 02:14:51.425111 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:51.425082    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs457\" (UniqueName: \"kubernetes.io/projected/3a7116ab-aa11-4085-93c1-3d5086a796de-kube-api-access-cs457\") pod \"payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w\" (UID: \"3a7116ab-aa11-4085-93c1-3d5086a796de\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"
Nov 03 02:14:51.425261 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:51.425125    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7116ab-aa11-4085-93c1-3d5086a796de-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w\" (UID: \"3a7116ab-aa11-4085-93c1-3d5086a796de\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"
Nov 03 02:14:51.526619 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:51.526582    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs457\" (UniqueName: \"kubernetes.io/projected/3a7116ab-aa11-4085-93c1-3d5086a796de-kube-api-access-cs457\") pod \"payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w\" (UID: \"3a7116ab-aa11-4085-93c1-3d5086a796de\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"
Nov 03 02:14:51.526813 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:51.526634    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7116ab-aa11-4085-93c1-3d5086a796de-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w\" (UID: \"3a7116ab-aa11-4085-93c1-3d5086a796de\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"
Nov 03 02:14:51.527435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:51.527413    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7116ab-aa11-4085-93c1-3d5086a796de-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w\" (UID: \"3a7116ab-aa11-4085-93c1-3d5086a796de\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"
Nov 03 02:14:51.584263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:51.584174    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs457\" (UniqueName: \"kubernetes.io/projected/3a7116ab-aa11-4085-93c1-3d5086a796de-kube-api-access-cs457\") pod \"payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w\" (UID: \"3a7116ab-aa11-4085-93c1-3d5086a796de\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"
Nov 03 02:14:51.652629 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:51.652589    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"
Nov 03 02:14:52.031567 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:52.031527    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"]
Nov 03 02:14:52.038153 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:14:52.038126    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7116ab_aa11_4085_93c1_3d5086a796de.slice/crio-2b6da2a4a95b30a1f9c0d30b94552b6964225540cf6ad9f974c59b45c181e20d WatchSource:0}: Error finding container 2b6da2a4a95b30a1f9c0d30b94552b6964225540cf6ad9f974c59b45c181e20d: Status 404 returned error can't find the container with id 2b6da2a4a95b30a1f9c0d30b94552b6964225540cf6ad9f974c59b45c181e20d
Nov 03 02:14:52.650313 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:52.650262    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w" event={"ID":"3a7116ab-aa11-4085-93c1-3d5086a796de","Type":"ContainerStarted","Data":"1c79c08354a37e7cbe3f584ec29a968f1c507f4924f80d88ede6a7fd746f412f"}
Nov 03 02:14:52.650767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:52.650393    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"
Nov 03 02:14:52.650767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:52.650414    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w" event={"ID":"3a7116ab-aa11-4085-93c1-3d5086a796de","Type":"ContainerStarted","Data":"2b6da2a4a95b30a1f9c0d30b94552b6964225540cf6ad9f974c59b45c181e20d"}
Nov 03 02:14:52.765656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:14:52.765604    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w" podStartSLOduration=1.7655863379999999 podStartE2EDuration="1.765586338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:14:52.764319905 +0000 UTC m=+3372.189926370" watchObservedRunningTime="2026-11-03 02:14:52.765586338 +0000 UTC m=+3372.191192802"
Nov 03 02:15:00.160543 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.160505    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"]
Nov 03 02:15:00.165976 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.165950    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"]
Nov 03 02:15:00.166128 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.166061    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"
Nov 03 02:15:00.170276 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.170256    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"
Nov 03 02:15:00.185564 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.185538    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"]
Nov 03 02:15:00.189833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.189810    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"
Nov 03 02:15:00.207109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.207077    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"]
Nov 03 02:15:00.212105 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.212071    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"]
Nov 03 02:15:00.212249 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.212199    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"
Nov 03 02:15:00.216977 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.216956    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"]
Nov 03 02:15:00.217117 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.217093    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"
Nov 03 02:15:00.221532 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.221505    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"
Nov 03 02:15:00.227176 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.227150    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:15:00.238449 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.238422    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e5b069-944c-4a7a-b547-2dd00b060a2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8\" (UID: \"19e5b069-944c-4a7a-b547-2dd00b060a2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"
Nov 03 02:15:00.238590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.238497    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvdvz\" (UniqueName: \"kubernetes.io/projected/a0337f2e-ae3a-4815-9c06-2abf4fb84490-kube-api-access-kvdvz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p\" (UID: \"a0337f2e-ae3a-4815-9c06-2abf4fb84490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"
Nov 03 02:15:00.238754 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.238728    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9sl2\" (UniqueName: \"kubernetes.io/projected/e8090231-a51a-4f90-bd68-97ae767ea797-kube-api-access-r9sl2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm\" (UID: \"e8090231-a51a-4f90-bd68-97ae767ea797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"
Nov 03 02:15:00.238824 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.238809    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0337f2e-ae3a-4815-9c06-2abf4fb84490-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p\" (UID: \"a0337f2e-ae3a-4815-9c06-2abf4fb84490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"
Nov 03 02:15:00.238962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.238940    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8090231-a51a-4f90-bd68-97ae767ea797-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm\" (UID: \"e8090231-a51a-4f90-bd68-97ae767ea797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"
Nov 03 02:15:00.239083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.238984    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x7w7\" (UniqueName: \"kubernetes.io/projected/19e5b069-944c-4a7a-b547-2dd00b060a2e-kube-api-access-2x7w7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8\" (UID: \"19e5b069-944c-4a7a-b547-2dd00b060a2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"
Nov 03 02:15:00.260712 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.260677    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"]
Nov 03 02:15:00.311533 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.311421    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"]
Nov 03 02:15:00.316443 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.315320    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"]
Nov 03 02:15:00.322190 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.322152    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"]
Nov 03 02:15:00.340375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.340321    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nq5v\" (UniqueName: \"kubernetes.io/projected/1ec2d52e-2bde-429f-8adc-31be523c9220-kube-api-access-5nq5v\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f\" (UID: \"1ec2d52e-2bde-429f-8adc-31be523c9220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"
Nov 03 02:15:00.340513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.340442    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec2d52e-2bde-429f-8adc-31be523c9220-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f\" (UID: \"1ec2d52e-2bde-429f-8adc-31be523c9220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"
Nov 03 02:15:00.340513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.340480    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe6ee18-4681-4e63-9675-a098c98a7aea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g\" (UID: \"8fe6ee18-4681-4e63-9675-a098c98a7aea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"
Nov 03 02:15:00.340721 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.340541    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9sl2\" (UniqueName: \"kubernetes.io/projected/e8090231-a51a-4f90-bd68-97ae767ea797-kube-api-access-r9sl2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm\" (UID: \"e8090231-a51a-4f90-bd68-97ae767ea797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"
Nov 03 02:15:00.340721 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.340598    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0337f2e-ae3a-4815-9c06-2abf4fb84490-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p\" (UID: \"a0337f2e-ae3a-4815-9c06-2abf4fb84490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"
Nov 03 02:15:00.340721 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.340639    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj99g\" (UniqueName: \"kubernetes.io/projected/a2656448-c23b-42f9-b85d-145af6aa7426-kube-api-access-jj99g\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw\" (UID: \"a2656448-c23b-42f9-b85d-145af6aa7426\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"
Nov 03 02:15:00.340721 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.340665    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b722z\" (UniqueName: \"kubernetes.io/projected/8fe6ee18-4681-4e63-9675-a098c98a7aea-kube-api-access-b722z\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g\" (UID: \"8fe6ee18-4681-4e63-9675-a098c98a7aea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"
Nov 03 02:15:00.340938 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.340714    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2656448-c23b-42f9-b85d-145af6aa7426-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw\" (UID: \"a2656448-c23b-42f9-b85d-145af6aa7426\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"
Nov 03 02:15:00.340938 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.340850    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8090231-a51a-4f90-bd68-97ae767ea797-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm\" (UID: \"e8090231-a51a-4f90-bd68-97ae767ea797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"
Nov 03 02:15:00.340938 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.340894    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x7w7\" (UniqueName: \"kubernetes.io/projected/19e5b069-944c-4a7a-b547-2dd00b060a2e-kube-api-access-2x7w7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8\" (UID: \"19e5b069-944c-4a7a-b547-2dd00b060a2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"
Nov 03 02:15:00.340938 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.340934    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e5b069-944c-4a7a-b547-2dd00b060a2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8\" (UID: \"19e5b069-944c-4a7a-b547-2dd00b060a2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"
Nov 03 02:15:00.341157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.340975    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvdvz\" (UniqueName: \"kubernetes.io/projected/a0337f2e-ae3a-4815-9c06-2abf4fb84490-kube-api-access-kvdvz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p\" (UID: \"a0337f2e-ae3a-4815-9c06-2abf4fb84490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"
Nov 03 02:15:00.341216 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.341194    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0337f2e-ae3a-4815-9c06-2abf4fb84490-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p\" (UID: \"a0337f2e-ae3a-4815-9c06-2abf4fb84490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"
Nov 03 02:15:00.341572 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.341551    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8090231-a51a-4f90-bd68-97ae767ea797-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm\" (UID: \"e8090231-a51a-4f90-bd68-97ae767ea797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"
Nov 03 02:15:00.342030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.342011    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e5b069-944c-4a7a-b547-2dd00b060a2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8\" (UID: \"19e5b069-944c-4a7a-b547-2dd00b060a2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"
Nov 03 02:15:00.365871 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.365840    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"]
Nov 03 02:15:00.371593 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.371558    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"]
Nov 03 02:15:00.398957 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.398929    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9sl2\" (UniqueName: \"kubernetes.io/projected/e8090231-a51a-4f90-bd68-97ae767ea797-kube-api-access-r9sl2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm\" (UID: \"e8090231-a51a-4f90-bd68-97ae767ea797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"
Nov 03 02:15:00.398957 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.398951    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvdvz\" (UniqueName: \"kubernetes.io/projected/a0337f2e-ae3a-4815-9c06-2abf4fb84490-kube-api-access-kvdvz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p\" (UID: \"a0337f2e-ae3a-4815-9c06-2abf4fb84490\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"
Nov 03 02:15:00.399557 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.399529    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x7w7\" (UniqueName: \"kubernetes.io/projected/19e5b069-944c-4a7a-b547-2dd00b060a2e-kube-api-access-2x7w7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8\" (UID: \"19e5b069-944c-4a7a-b547-2dd00b060a2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"
Nov 03 02:15:00.442175 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.442085    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj99g\" (UniqueName: \"kubernetes.io/projected/a2656448-c23b-42f9-b85d-145af6aa7426-kube-api-access-jj99g\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw\" (UID: \"a2656448-c23b-42f9-b85d-145af6aa7426\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"
Nov 03 02:15:00.442175 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.442134    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b722z\" (UniqueName: \"kubernetes.io/projected/8fe6ee18-4681-4e63-9675-a098c98a7aea-kube-api-access-b722z\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g\" (UID: \"8fe6ee18-4681-4e63-9675-a098c98a7aea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"
Nov 03 02:15:00.442417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.442188    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2656448-c23b-42f9-b85d-145af6aa7426-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw\" (UID: \"a2656448-c23b-42f9-b85d-145af6aa7426\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"
Nov 03 02:15:00.442417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.442395    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nq5v\" (UniqueName: \"kubernetes.io/projected/1ec2d52e-2bde-429f-8adc-31be523c9220-kube-api-access-5nq5v\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f\" (UID: \"1ec2d52e-2bde-429f-8adc-31be523c9220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"
Nov 03 02:15:00.442551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.442450    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec2d52e-2bde-429f-8adc-31be523c9220-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f\" (UID: \"1ec2d52e-2bde-429f-8adc-31be523c9220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"
Nov 03 02:15:00.442551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.442479    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe6ee18-4681-4e63-9675-a098c98a7aea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g\" (UID: \"8fe6ee18-4681-4e63-9675-a098c98a7aea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"
Nov 03 02:15:00.443083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.443061    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2656448-c23b-42f9-b85d-145af6aa7426-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw\" (UID: \"a2656448-c23b-42f9-b85d-145af6aa7426\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"
Nov 03 02:15:00.443193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.443069    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec2d52e-2bde-429f-8adc-31be523c9220-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f\" (UID: \"1ec2d52e-2bde-429f-8adc-31be523c9220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"
Nov 03 02:15:00.443193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.443109    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe6ee18-4681-4e63-9675-a098c98a7aea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g\" (UID: \"8fe6ee18-4681-4e63-9675-a098c98a7aea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"
Nov 03 02:15:00.481135 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.481108    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"
Nov 03 02:15:00.491673 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.491649    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"
Nov 03 02:15:00.500211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.500183    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b722z\" (UniqueName: \"kubernetes.io/projected/8fe6ee18-4681-4e63-9675-a098c98a7aea-kube-api-access-b722z\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g\" (UID: \"8fe6ee18-4681-4e63-9675-a098c98a7aea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"
Nov 03 02:15:00.500337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.500231    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj99g\" (UniqueName: \"kubernetes.io/projected/a2656448-c23b-42f9-b85d-145af6aa7426-kube-api-access-jj99g\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw\" (UID: \"a2656448-c23b-42f9-b85d-145af6aa7426\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"
Nov 03 02:15:00.500337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.500231    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nq5v\" (UniqueName: \"kubernetes.io/projected/1ec2d52e-2bde-429f-8adc-31be523c9220-kube-api-access-5nq5v\") pod \"maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f\" (UID: \"1ec2d52e-2bde-429f-8adc-31be523c9220\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"
Nov 03 02:15:00.504293 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.504033    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"
Nov 03 02:15:00.529421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.529141    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"
Nov 03 02:15:00.537906 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.537751    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"
Nov 03 02:15:00.551414 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.550722    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"
Nov 03 02:15:00.868992 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.868962    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"]
Nov 03 02:15:00.876082 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:15:00.876054    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0337f2e_ae3a_4815_9c06_2abf4fb84490.slice/crio-02fee778bde3f559d96e321412001765e81a70036d6c9b75e05060c7ad6f8732 WatchSource:0}: Error finding container 02fee778bde3f559d96e321412001765e81a70036d6c9b75e05060c7ad6f8732: Status 404 returned error can't find the container with id 02fee778bde3f559d96e321412001765e81a70036d6c9b75e05060c7ad6f8732
Nov 03 02:15:00.920701 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.920677    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"]
Nov 03 02:15:00.927504 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:15:00.927471    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8090231_a51a_4f90_bd68_97ae767ea797.slice/crio-213df75ab775cec1f666348fcacc2add62fdf62abffb8b47627315d3995a580a WatchSource:0}: Error finding container 213df75ab775cec1f666348fcacc2add62fdf62abffb8b47627315d3995a580a: Status 404 returned error can't find the container with id 213df75ab775cec1f666348fcacc2add62fdf62abffb8b47627315d3995a580a
Nov 03 02:15:00.974960 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.974933    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"]
Nov 03 02:15:00.978629 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.978481    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"]
Nov 03 02:15:00.981953 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:15:00.981927    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e5b069_944c_4a7a_b547_2dd00b060a2e.slice/crio-b9dfad36cc1696ff4bf2f1774a4aa19d94d3b2bec84a891ae63968bbc05bf078 WatchSource:0}: Error finding container b9dfad36cc1696ff4bf2f1774a4aa19d94d3b2bec84a891ae63968bbc05bf078: Status 404 returned error can't find the container with id b9dfad36cc1696ff4bf2f1774a4aa19d94d3b2bec84a891ae63968bbc05bf078
Nov 03 02:15:00.984374 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:15:00.984280    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe6ee18_4681_4e63_9675_a098c98a7aea.slice/crio-f3e4947c22e20140a6c59cde65695de4fbc4f5e94cdf617681afc0731be9f694 WatchSource:0}: Error finding container f3e4947c22e20140a6c59cde65695de4fbc4f5e94cdf617681afc0731be9f694: Status 404 returned error can't find the container with id f3e4947c22e20140a6c59cde65695de4fbc4f5e94cdf617681afc0731be9f694
Nov 03 02:15:00.986128 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:00.986102    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"]
Nov 03 02:15:00.995330 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:15:00.995305    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2656448_c23b_42f9_b85d_145af6aa7426.slice/crio-5834f016733514bde96ee657ea43d72caa7456bfa20efe07db80a9071982d0d7 WatchSource:0}: Error finding container 5834f016733514bde96ee657ea43d72caa7456bfa20efe07db80a9071982d0d7: Status 404 returned error can't find the container with id 5834f016733514bde96ee657ea43d72caa7456bfa20efe07db80a9071982d0d7
Nov 03 02:15:01.006411 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.006382    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"]
Nov 03 02:15:01.021830 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:15:01.021804    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec2d52e_2bde_429f_8adc_31be523c9220.slice/crio-a5ddec2d8566ef953c322b89e990b447ae7ee9c39f13edd2eae8cbf020c5abb0 WatchSource:0}: Error finding container a5ddec2d8566ef953c322b89e990b447ae7ee9c39f13edd2eae8cbf020c5abb0: Status 404 returned error can't find the container with id a5ddec2d8566ef953c322b89e990b447ae7ee9c39f13edd2eae8cbf020c5abb0
Nov 03 02:15:01.734371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.734313    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm" event={"ID":"e8090231-a51a-4f90-bd68-97ae767ea797","Type":"ContainerStarted","Data":"b31cb135996c3cd4cf2203add1c5c0b9a58a056ee66e39db180d6394702d8e12"}
Nov 03 02:15:01.734975 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.734394    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm" event={"ID":"e8090231-a51a-4f90-bd68-97ae767ea797","Type":"ContainerStarted","Data":"213df75ab775cec1f666348fcacc2add62fdf62abffb8b47627315d3995a580a"}
Nov 03 02:15:01.736022 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.735987    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g" event={"ID":"8fe6ee18-4681-4e63-9675-a098c98a7aea","Type":"ContainerStarted","Data":"49c1ccc73b7b41ac0bf2d5d0261590e4f75cd608a5f15ca516481de8baf8f873"}
Nov 03 02:15:01.736185 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.736039    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g" event={"ID":"8fe6ee18-4681-4e63-9675-a098c98a7aea","Type":"ContainerStarted","Data":"f3e4947c22e20140a6c59cde65695de4fbc4f5e94cdf617681afc0731be9f694"}
Nov 03 02:15:01.737731 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.737710    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p" event={"ID":"a0337f2e-ae3a-4815-9c06-2abf4fb84490","Type":"ContainerStarted","Data":"1feb3043557ba60c2361df15e2ffb2d89ab331719de462464b5feb079d4d959b"}
Nov 03 02:15:01.737868 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.737755    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p" event={"ID":"a0337f2e-ae3a-4815-9c06-2abf4fb84490","Type":"ContainerStarted","Data":"02fee778bde3f559d96e321412001765e81a70036d6c9b75e05060c7ad6f8732"}
Nov 03 02:15:01.739407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.739382    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw" event={"ID":"a2656448-c23b-42f9-b85d-145af6aa7426","Type":"ContainerStarted","Data":"ad518c4ade8856350db710bae33636f5620c6346185b5c40d25f0326be23b833"}
Nov 03 02:15:01.739525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.739430    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw" event={"ID":"a2656448-c23b-42f9-b85d-145af6aa7426","Type":"ContainerStarted","Data":"5834f016733514bde96ee657ea43d72caa7456bfa20efe07db80a9071982d0d7"}
Nov 03 02:15:01.741158 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.741131    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8" event={"ID":"19e5b069-944c-4a7a-b547-2dd00b060a2e","Type":"ContainerStarted","Data":"1de1b5059846442588303f3c6e69652ade313faf33a4d7b7397314fff2460290"}
Nov 03 02:15:01.741272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.741179    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8" event={"ID":"19e5b069-944c-4a7a-b547-2dd00b060a2e","Type":"ContainerStarted","Data":"b9dfad36cc1696ff4bf2f1774a4aa19d94d3b2bec84a891ae63968bbc05bf078"}
Nov 03 02:15:01.742991 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.742968    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f" event={"ID":"1ec2d52e-2bde-429f-8adc-31be523c9220","Type":"ContainerStarted","Data":"00d00611d1d35a9a3cb45e716b70a7ba57a0faa08abdccc99299ca95db2f00ec"}
Nov 03 02:15:01.743102 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.743013    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f" event={"ID":"1ec2d52e-2bde-429f-8adc-31be523c9220","Type":"ContainerStarted","Data":"a5ddec2d8566ef953c322b89e990b447ae7ee9c39f13edd2eae8cbf020c5abb0"}
Nov 03 02:15:01.842172 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.842113    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm" podStartSLOduration=1.8420987690000001 podStartE2EDuration="1.842098769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:01.840987567 +0000 UTC m=+3381.266594033" watchObservedRunningTime="2026-11-03 02:15:01.842098769 +0000 UTC m=+3381.267705232"
Nov 03 02:15:01.951722 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:01.951669    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p" podStartSLOduration=1.9516532340000001 podStartE2EDuration="1.951653234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:01.947860061 +0000 UTC m=+3381.373466527" watchObservedRunningTime="2026-11-03 02:15:01.951653234 +0000 UTC m=+3381.377259699"
Nov 03 02:15:02.060371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.059811    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g" podStartSLOduration=2.059792135 podStartE2EDuration="2.059792135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:02.05524727 +0000 UTC m=+3381.480853763" watchObservedRunningTime="2026-11-03 02:15:02.059792135 +0000 UTC m=+3381.485398602"
Nov 03 02:15:02.163787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.163725    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f" podStartSLOduration=2.163708734 podStartE2EDuration="2.163708734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:02.162854683 +0000 UTC m=+3381.588461166" watchObservedRunningTime="2026-11-03 02:15:02.163708734 +0000 UTC m=+3381.589315204"
Nov 03 02:15:02.269982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.269918    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw" podStartSLOduration=2.269907974 podStartE2EDuration="2.269907974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:02.269438252 +0000 UTC m=+3381.695044738" watchObservedRunningTime="2026-11-03 02:15:02.269907974 +0000 UTC m=+3381.695514438"
Nov 03 02:15:02.382142 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.382032    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8" podStartSLOduration=2.382016678 podStartE2EDuration="2.382016678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:15:02.379196098 +0000 UTC m=+3381.804802574" watchObservedRunningTime="2026-11-03 02:15:02.382016678 +0000 UTC m=+3381.807623142"
Nov 03 02:15:02.751911 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.751876    5508 generic.go:356] "Generic (PLEG): container finished" podID="a2656448-c23b-42f9-b85d-145af6aa7426" containerID="ad518c4ade8856350db710bae33636f5620c6346185b5c40d25f0326be23b833" exitCode=60
Nov 03 02:15:02.752425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.751956    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw" event={"ID":"a2656448-c23b-42f9-b85d-145af6aa7426","Type":"ContainerDied","Data":"ad518c4ade8856350db710bae33636f5620c6346185b5c40d25f0326be23b833"}
Nov 03 02:15:02.752425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.752237    5508 scope.go:122] "RemoveContainer" containerID="ad518c4ade8856350db710bae33636f5620c6346185b5c40d25f0326be23b833"
Nov 03 02:15:02.753811 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.753789    5508 generic.go:356] "Generic (PLEG): container finished" podID="19e5b069-944c-4a7a-b547-2dd00b060a2e" containerID="1de1b5059846442588303f3c6e69652ade313faf33a4d7b7397314fff2460290" exitCode=60
Nov 03 02:15:02.753939 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.753875    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8" event={"ID":"19e5b069-944c-4a7a-b547-2dd00b060a2e","Type":"ContainerDied","Data":"1de1b5059846442588303f3c6e69652ade313faf33a4d7b7397314fff2460290"}
Nov 03 02:15:02.754120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.754104    5508 scope.go:122] "RemoveContainer" containerID="1de1b5059846442588303f3c6e69652ade313faf33a4d7b7397314fff2460290"
Nov 03 02:15:02.755815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.755793    5508 generic.go:356] "Generic (PLEG): container finished" podID="1ec2d52e-2bde-429f-8adc-31be523c9220" containerID="00d00611d1d35a9a3cb45e716b70a7ba57a0faa08abdccc99299ca95db2f00ec" exitCode=60
Nov 03 02:15:02.756041 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.755881    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f" event={"ID":"1ec2d52e-2bde-429f-8adc-31be523c9220","Type":"ContainerDied","Data":"00d00611d1d35a9a3cb45e716b70a7ba57a0faa08abdccc99299ca95db2f00ec"}
Nov 03 02:15:02.756157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.756100    5508 scope.go:122] "RemoveContainer" containerID="00d00611d1d35a9a3cb45e716b70a7ba57a0faa08abdccc99299ca95db2f00ec"
Nov 03 02:15:02.757759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.757728    5508 generic.go:356] "Generic (PLEG): container finished" podID="e8090231-a51a-4f90-bd68-97ae767ea797" containerID="b31cb135996c3cd4cf2203add1c5c0b9a58a056ee66e39db180d6394702d8e12" exitCode=60
Nov 03 02:15:02.757843 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.757813    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm" event={"ID":"e8090231-a51a-4f90-bd68-97ae767ea797","Type":"ContainerDied","Data":"b31cb135996c3cd4cf2203add1c5c0b9a58a056ee66e39db180d6394702d8e12"}
Nov 03 02:15:02.758010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.757994    5508 scope.go:122] "RemoveContainer" containerID="b31cb135996c3cd4cf2203add1c5c0b9a58a056ee66e39db180d6394702d8e12"
Nov 03 02:15:02.759502 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.759479    5508 generic.go:356] "Generic (PLEG): container finished" podID="8fe6ee18-4681-4e63-9675-a098c98a7aea" containerID="49c1ccc73b7b41ac0bf2d5d0261590e4f75cd608a5f15ca516481de8baf8f873" exitCode=60
Nov 03 02:15:02.759601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.759551    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g" event={"ID":"8fe6ee18-4681-4e63-9675-a098c98a7aea","Type":"ContainerDied","Data":"49c1ccc73b7b41ac0bf2d5d0261590e4f75cd608a5f15ca516481de8baf8f873"}
Nov 03 02:15:02.759743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.759729    5508 scope.go:122] "RemoveContainer" containerID="49c1ccc73b7b41ac0bf2d5d0261590e4f75cd608a5f15ca516481de8baf8f873"
Nov 03 02:15:02.761120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.761098    5508 generic.go:356] "Generic (PLEG): container finished" podID="a0337f2e-ae3a-4815-9c06-2abf4fb84490" containerID="1feb3043557ba60c2361df15e2ffb2d89ab331719de462464b5feb079d4d959b" exitCode=60
Nov 03 02:15:02.761213 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.761146    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p" event={"ID":"a0337f2e-ae3a-4815-9c06-2abf4fb84490","Type":"ContainerDied","Data":"1feb3043557ba60c2361df15e2ffb2d89ab331719de462464b5feb079d4d959b"}
Nov 03 02:15:02.761427 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:02.761411    5508 scope.go:122] "RemoveContainer" containerID="1feb3043557ba60c2361df15e2ffb2d89ab331719de462464b5feb079d4d959b"
Nov 03 02:15:03.660505 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:03.660472    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"
Nov 03 02:15:03.770724 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:03.770682    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f" event={"ID":"1ec2d52e-2bde-429f-8adc-31be523c9220","Type":"ContainerStarted","Data":"a6a615fcb86033ca9bbfc7357eebb971240af9eb754fb3b8641e91c751b60867"}
Nov 03 02:15:03.773898 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:03.773865    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm" event={"ID":"e8090231-a51a-4f90-bd68-97ae767ea797","Type":"ContainerStarted","Data":"3cdda3bbcf437b25c60ee5db5def7721a802de6c8fd88ba08fd9320e6e4e5197"}
Nov 03 02:15:03.775784 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:03.775757    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g" event={"ID":"8fe6ee18-4681-4e63-9675-a098c98a7aea","Type":"ContainerStarted","Data":"13d9e543ce89a63429bb0fdd7cf9102e262625c80db2b249aa85d07137095993"}
Nov 03 02:15:03.778282 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:03.778257    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p" event={"ID":"a0337f2e-ae3a-4815-9c06-2abf4fb84490","Type":"ContainerStarted","Data":"9afe6d22c623cae7ef984bb75f420fc84c2b474887adbddd440e48997b0bc76c"}
Nov 03 02:15:03.780265 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:03.780240    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw" event={"ID":"a2656448-c23b-42f9-b85d-145af6aa7426","Type":"ContainerStarted","Data":"54c65803a12cc3f2ec613bc02c3e5173bf65683a026424f406c728b4dd7ba8d4"}
Nov 03 02:15:03.782770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:03.782742    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8" event={"ID":"19e5b069-944c-4a7a-b547-2dd00b060a2e","Type":"ContainerStarted","Data":"969f1342a71ce3c896c750b9d68799ac77b90c2443970878ea36f2c5c2e41117"}
Nov 03 02:15:04.795895 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.795857    5508 generic.go:356] "Generic (PLEG): container finished" podID="1ec2d52e-2bde-429f-8adc-31be523c9220" containerID="a6a615fcb86033ca9bbfc7357eebb971240af9eb754fb3b8641e91c751b60867" exitCode=60
Nov 03 02:15:04.795895 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.795890    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f" event={"ID":"1ec2d52e-2bde-429f-8adc-31be523c9220","Type":"ContainerDied","Data":"a6a615fcb86033ca9bbfc7357eebb971240af9eb754fb3b8641e91c751b60867"}
Nov 03 02:15:04.796512 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.795953    5508 scope.go:122] "RemoveContainer" containerID="00d00611d1d35a9a3cb45e716b70a7ba57a0faa08abdccc99299ca95db2f00ec"
Nov 03 02:15:04.796512 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.796292    5508 scope.go:122] "RemoveContainer" containerID="a6a615fcb86033ca9bbfc7357eebb971240af9eb754fb3b8641e91c751b60867"
Nov 03 02:15:04.796713 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:04.796681    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f_odh-ai-gateway-infra(1ec2d52e-2bde-429f-8adc-31be523c9220)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f" podUID="1ec2d52e-2bde-429f-8adc-31be523c9220"
Nov 03 02:15:04.798465 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.798437    5508 generic.go:356] "Generic (PLEG): container finished" podID="e8090231-a51a-4f90-bd68-97ae767ea797" containerID="3cdda3bbcf437b25c60ee5db5def7721a802de6c8fd88ba08fd9320e6e4e5197" exitCode=60
Nov 03 02:15:04.798575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.798519    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm" event={"ID":"e8090231-a51a-4f90-bd68-97ae767ea797","Type":"ContainerDied","Data":"3cdda3bbcf437b25c60ee5db5def7721a802de6c8fd88ba08fd9320e6e4e5197"}
Nov 03 02:15:04.798808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.798788    5508 scope.go:122] "RemoveContainer" containerID="3cdda3bbcf437b25c60ee5db5def7721a802de6c8fd88ba08fd9320e6e4e5197"
Nov 03 02:15:04.799129 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:04.799109    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm_odh-ai-gateway-infra(e8090231-a51a-4f90-bd68-97ae767ea797)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm" podUID="e8090231-a51a-4f90-bd68-97ae767ea797"
Nov 03 02:15:04.800775 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.800745    5508 generic.go:356] "Generic (PLEG): container finished" podID="8fe6ee18-4681-4e63-9675-a098c98a7aea" containerID="13d9e543ce89a63429bb0fdd7cf9102e262625c80db2b249aa85d07137095993" exitCode=60
Nov 03 02:15:04.800883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.800837    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g" event={"ID":"8fe6ee18-4681-4e63-9675-a098c98a7aea","Type":"ContainerDied","Data":"13d9e543ce89a63429bb0fdd7cf9102e262625c80db2b249aa85d07137095993"}
Nov 03 02:15:04.801092 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.801076    5508 scope.go:122] "RemoveContainer" containerID="13d9e543ce89a63429bb0fdd7cf9102e262625c80db2b249aa85d07137095993"
Nov 03 02:15:04.801422 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:04.801398    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g_odh-ai-gateway-infra(8fe6ee18-4681-4e63-9675-a098c98a7aea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g" podUID="8fe6ee18-4681-4e63-9675-a098c98a7aea"
Nov 03 02:15:04.803192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.803169    5508 generic.go:356] "Generic (PLEG): container finished" podID="a0337f2e-ae3a-4815-9c06-2abf4fb84490" containerID="9afe6d22c623cae7ef984bb75f420fc84c2b474887adbddd440e48997b0bc76c" exitCode=60
Nov 03 02:15:04.803312 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.803196    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p" event={"ID":"a0337f2e-ae3a-4815-9c06-2abf4fb84490","Type":"ContainerDied","Data":"9afe6d22c623cae7ef984bb75f420fc84c2b474887adbddd440e48997b0bc76c"}
Nov 03 02:15:04.803543 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.803519    5508 scope.go:122] "RemoveContainer" containerID="9afe6d22c623cae7ef984bb75f420fc84c2b474887adbddd440e48997b0bc76c"
Nov 03 02:15:04.803854 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:04.803827    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p_odh-ai-gateway-infra(a0337f2e-ae3a-4815-9c06-2abf4fb84490)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p" podUID="a0337f2e-ae3a-4815-9c06-2abf4fb84490"
Nov 03 02:15:04.805591 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.805569    5508 generic.go:356] "Generic (PLEG): container finished" podID="a2656448-c23b-42f9-b85d-145af6aa7426" containerID="54c65803a12cc3f2ec613bc02c3e5173bf65683a026424f406c728b4dd7ba8d4" exitCode=60
Nov 03 02:15:04.805697 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.805647    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw" event={"ID":"a2656448-c23b-42f9-b85d-145af6aa7426","Type":"ContainerDied","Data":"54c65803a12cc3f2ec613bc02c3e5173bf65683a026424f406c728b4dd7ba8d4"}
Nov 03 02:15:04.805883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.805863    5508 scope.go:122] "RemoveContainer" containerID="54c65803a12cc3f2ec613bc02c3e5173bf65683a026424f406c728b4dd7ba8d4"
Nov 03 02:15:04.806167 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:04.806139    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw_odh-ai-gateway-infra(a2656448-c23b-42f9-b85d-145af6aa7426)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw" podUID="a2656448-c23b-42f9-b85d-145af6aa7426"
Nov 03 02:15:04.807810 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.807790    5508 generic.go:356] "Generic (PLEG): container finished" podID="19e5b069-944c-4a7a-b547-2dd00b060a2e" containerID="969f1342a71ce3c896c750b9d68799ac77b90c2443970878ea36f2c5c2e41117" exitCode=60
Nov 03 02:15:04.807922 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.807868    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8" event={"ID":"19e5b069-944c-4a7a-b547-2dd00b060a2e","Type":"ContainerDied","Data":"969f1342a71ce3c896c750b9d68799ac77b90c2443970878ea36f2c5c2e41117"}
Nov 03 02:15:04.808044 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.808030    5508 scope.go:122] "RemoveContainer" containerID="969f1342a71ce3c896c750b9d68799ac77b90c2443970878ea36f2c5c2e41117"
Nov 03 02:15:04.808319 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:04.808285    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8_odh-ai-gateway-infra(19e5b069-944c-4a7a-b547-2dd00b060a2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8" podUID="19e5b069-944c-4a7a-b547-2dd00b060a2e"
Nov 03 02:15:04.810702 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.810682    5508 scope.go:122] "RemoveContainer" containerID="b31cb135996c3cd4cf2203add1c5c0b9a58a056ee66e39db180d6394702d8e12"
Nov 03 02:15:04.824822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.824801    5508 scope.go:122] "RemoveContainer" containerID="49c1ccc73b7b41ac0bf2d5d0261590e4f75cd608a5f15ca516481de8baf8f873"
Nov 03 02:15:04.838083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.838064    5508 scope.go:122] "RemoveContainer" containerID="1feb3043557ba60c2361df15e2ffb2d89ab331719de462464b5feb079d4d959b"
Nov 03 02:15:04.852059 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.852037    5508 scope.go:122] "RemoveContainer" containerID="ad518c4ade8856350db710bae33636f5620c6346185b5c40d25f0326be23b833"
Nov 03 02:15:04.866296 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:04.866277    5508 scope.go:122] "RemoveContainer" containerID="1de1b5059846442588303f3c6e69652ade313faf33a4d7b7397314fff2460290"
Nov 03 02:15:05.817904 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:05.817866    5508 scope.go:122] "RemoveContainer" containerID="3cdda3bbcf437b25c60ee5db5def7721a802de6c8fd88ba08fd9320e6e4e5197"
Nov 03 02:15:05.818404 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:05.818212    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm_odh-ai-gateway-infra(e8090231-a51a-4f90-bd68-97ae767ea797)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm" podUID="e8090231-a51a-4f90-bd68-97ae767ea797"
Nov 03 02:15:05.819705 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:05.819680    5508 scope.go:122] "RemoveContainer" containerID="13d9e543ce89a63429bb0fdd7cf9102e262625c80db2b249aa85d07137095993"
Nov 03 02:15:05.819949 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:05.819929    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g_odh-ai-gateway-infra(8fe6ee18-4681-4e63-9675-a098c98a7aea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g" podUID="8fe6ee18-4681-4e63-9675-a098c98a7aea"
Nov 03 02:15:05.821431 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:05.821407    5508 scope.go:122] "RemoveContainer" containerID="9afe6d22c623cae7ef984bb75f420fc84c2b474887adbddd440e48997b0bc76c"
Nov 03 02:15:05.821665 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:05.821645    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p_odh-ai-gateway-infra(a0337f2e-ae3a-4815-9c06-2abf4fb84490)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p" podUID="a0337f2e-ae3a-4815-9c06-2abf4fb84490"
Nov 03 02:15:05.823044 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:05.823021    5508 scope.go:122] "RemoveContainer" containerID="54c65803a12cc3f2ec613bc02c3e5173bf65683a026424f406c728b4dd7ba8d4"
Nov 03 02:15:05.823277 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:05.823258    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw_odh-ai-gateway-infra(a2656448-c23b-42f9-b85d-145af6aa7426)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw" podUID="a2656448-c23b-42f9-b85d-145af6aa7426"
Nov 03 02:15:05.824587 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:05.824566    5508 scope.go:122] "RemoveContainer" containerID="969f1342a71ce3c896c750b9d68799ac77b90c2443970878ea36f2c5c2e41117"
Nov 03 02:15:05.824791 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:05.824768    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8_odh-ai-gateway-infra(19e5b069-944c-4a7a-b547-2dd00b060a2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8" podUID="19e5b069-944c-4a7a-b547-2dd00b060a2e"
Nov 03 02:15:05.826192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:05.826177    5508 scope.go:122] "RemoveContainer" containerID="a6a615fcb86033ca9bbfc7357eebb971240af9eb754fb3b8641e91c751b60867"
Nov 03 02:15:05.826387 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:05.826366    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f_odh-ai-gateway-infra(1ec2d52e-2bde-429f-8adc-31be523c9220)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f" podUID="1ec2d52e-2bde-429f-8adc-31be523c9220"
Nov 03 02:15:13.814333 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:13.814285    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:15:14.696995 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:14.696964    5508 scope.go:122] "RemoveContainer" containerID="a6a615fcb86033ca9bbfc7357eebb971240af9eb754fb3b8641e91c751b60867"
Nov 03 02:15:15.684943 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:15.684911    5508 scope.go:122] "RemoveContainer" containerID="969f1342a71ce3c896c750b9d68799ac77b90c2443970878ea36f2c5c2e41117"
Nov 03 02:15:15.910643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:15.910610    5508 generic.go:356] "Generic (PLEG): container finished" podID="1ec2d52e-2bde-429f-8adc-31be523c9220" containerID="48bcf748079d5f2d727bb9a9ef4b4ab5ff02f9cd1b9d80cb51bbb92fadb5577c" exitCode=60
Nov 03 02:15:15.910820 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:15.910691    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f" event={"ID":"1ec2d52e-2bde-429f-8adc-31be523c9220","Type":"ContainerDied","Data":"48bcf748079d5f2d727bb9a9ef4b4ab5ff02f9cd1b9d80cb51bbb92fadb5577c"}
Nov 03 02:15:15.910875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:15.910862    5508 scope.go:122] "RemoveContainer" containerID="a6a615fcb86033ca9bbfc7357eebb971240af9eb754fb3b8641e91c751b60867"
Nov 03 02:15:15.911285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:15.911270    5508 scope.go:122] "RemoveContainer" containerID="48bcf748079d5f2d727bb9a9ef4b4ab5ff02f9cd1b9d80cb51bbb92fadb5577c"
Nov 03 02:15:15.911627 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:15.911603    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f_odh-ai-gateway-infra(1ec2d52e-2bde-429f-8adc-31be523c9220)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f" podUID="1ec2d52e-2bde-429f-8adc-31be523c9220"
Nov 03 02:15:16.685364 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:16.685325    5508 scope.go:122] "RemoveContainer" containerID="9afe6d22c623cae7ef984bb75f420fc84c2b474887adbddd440e48997b0bc76c"
Nov 03 02:15:16.921271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:16.921234    5508 generic.go:356] "Generic (PLEG): container finished" podID="19e5b069-944c-4a7a-b547-2dd00b060a2e" containerID="e265003c3a6118055dfcaa3274b7a5138c237e928dde7e82845e998a8f1e6293" exitCode=60
Nov 03 02:15:16.921445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:16.921302    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8" event={"ID":"19e5b069-944c-4a7a-b547-2dd00b060a2e","Type":"ContainerDied","Data":"e265003c3a6118055dfcaa3274b7a5138c237e928dde7e82845e998a8f1e6293"}
Nov 03 02:15:16.921445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:16.921385    5508 scope.go:122] "RemoveContainer" containerID="969f1342a71ce3c896c750b9d68799ac77b90c2443970878ea36f2c5c2e41117"
Nov 03 02:15:16.921799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:16.921784    5508 scope.go:122] "RemoveContainer" containerID="e265003c3a6118055dfcaa3274b7a5138c237e928dde7e82845e998a8f1e6293"
Nov 03 02:15:16.922142 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:16.922123    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8_odh-ai-gateway-infra(19e5b069-944c-4a7a-b547-2dd00b060a2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8" podUID="19e5b069-944c-4a7a-b547-2dd00b060a2e"
Nov 03 02:15:17.030964 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.030932    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"]
Nov 03 02:15:17.180302 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.180272    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"
Nov 03 02:15:17.273840 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.273806    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec2d52e-2bde-429f-8adc-31be523c9220-kube-api-access-5nq5v\" (UniqueName: \"kubernetes.io/projected/1ec2d52e-2bde-429f-8adc-31be523c9220-kube-api-access-5nq5v\") pod \"1ec2d52e-2bde-429f-8adc-31be523c9220\" (UID: \"1ec2d52e-2bde-429f-8adc-31be523c9220\") "
Nov 03 02:15:17.274050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.273984    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec2d52e-2bde-429f-8adc-31be523c9220-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec2d52e-2bde-429f-8adc-31be523c9220-maas-api-service-ca\") pod \"1ec2d52e-2bde-429f-8adc-31be523c9220\" (UID: \"1ec2d52e-2bde-429f-8adc-31be523c9220\") "
Nov 03 02:15:17.274384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.274327    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec2d52e-2bde-429f-8adc-31be523c9220-maas-api-service-ca" pod "1ec2d52e-2bde-429f-8adc-31be523c9220" (UID: "1ec2d52e-2bde-429f-8adc-31be523c9220"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:15:17.274583 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.274565    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec2d52e-2bde-429f-8adc-31be523c9220-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:17.276007 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.275983    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec2d52e-2bde-429f-8adc-31be523c9220-kube-api-access-5nq5v" pod "1ec2d52e-2bde-429f-8adc-31be523c9220" (UID: "1ec2d52e-2bde-429f-8adc-31be523c9220"). InnerVolumeSpecName "kube-api-access-5nq5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:15:17.375502 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.375467    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nq5v\" (UniqueName: \"kubernetes.io/projected/1ec2d52e-2bde-429f-8adc-31be523c9220-kube-api-access-5nq5v\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:17.685526 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.685495    5508 scope.go:122] "RemoveContainer" containerID="13d9e543ce89a63429bb0fdd7cf9102e262625c80db2b249aa85d07137095993"
Nov 03 02:15:17.686031 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.685548    5508 scope.go:122] "RemoveContainer" containerID="54c65803a12cc3f2ec613bc02c3e5173bf65683a026424f406c728b4dd7ba8d4"
Nov 03 02:15:17.930753 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.930722    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"
Nov 03 02:15:17.930935 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.930768    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f" event={"ID":"1ec2d52e-2bde-429f-8adc-31be523c9220","Type":"ContainerDied","Data":"a5ddec2d8566ef953c322b89e990b447ae7ee9c39f13edd2eae8cbf020c5abb0"}
Nov 03 02:15:17.930935 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.930827    5508 scope.go:122] "RemoveContainer" containerID="48bcf748079d5f2d727bb9a9ef4b4ab5ff02f9cd1b9d80cb51bbb92fadb5577c"
Nov 03 02:15:17.933109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.933084    5508 generic.go:356] "Generic (PLEG): container finished" podID="a0337f2e-ae3a-4815-9c06-2abf4fb84490" containerID="a1db26517be9d1a4206682f0423dfd64ddf24b08ec3ea19b0dfd518d0e8c772b" exitCode=60
Nov 03 02:15:17.933252 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.933153    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p" event={"ID":"a0337f2e-ae3a-4815-9c06-2abf4fb84490","Type":"ContainerDied","Data":"a1db26517be9d1a4206682f0423dfd64ddf24b08ec3ea19b0dfd518d0e8c772b"}
Nov 03 02:15:17.933679 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.933659    5508 scope.go:122] "RemoveContainer" containerID="a1db26517be9d1a4206682f0423dfd64ddf24b08ec3ea19b0dfd518d0e8c772b"
Nov 03 02:15:17.935470 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:17.934005    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p_odh-ai-gateway-infra(a0337f2e-ae3a-4815-9c06-2abf4fb84490)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p" podUID="a0337f2e-ae3a-4815-9c06-2abf4fb84490"
Nov 03 02:15:17.943730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:17.943709    5508 scope.go:122] "RemoveContainer" containerID="9afe6d22c623cae7ef984bb75f420fc84c2b474887adbddd440e48997b0bc76c"
Nov 03 02:15:18.042814 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.042782    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"]
Nov 03 02:15:18.088386 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.088332    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"]
Nov 03 02:15:18.096075 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.095973    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-41aa5f-29894535-vpb8f"]
Nov 03 02:15:18.272249 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.272227    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"
Nov 03 02:15:18.284579 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.284555    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e5b069-944c-4a7a-b547-2dd00b060a2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e5b069-944c-4a7a-b547-2dd00b060a2e-maas-api-service-ca\") pod \"19e5b069-944c-4a7a-b547-2dd00b060a2e\" (UID: \"19e5b069-944c-4a7a-b547-2dd00b060a2e\") "
Nov 03 02:15:18.284771 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.284748    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e5b069-944c-4a7a-b547-2dd00b060a2e-kube-api-access-2x7w7\" (UniqueName: \"kubernetes.io/projected/19e5b069-944c-4a7a-b547-2dd00b060a2e-kube-api-access-2x7w7\") pod \"19e5b069-944c-4a7a-b547-2dd00b060a2e\" (UID: \"19e5b069-944c-4a7a-b547-2dd00b060a2e\") "
Nov 03 02:15:18.284896 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.284863    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e5b069-944c-4a7a-b547-2dd00b060a2e-maas-api-service-ca" pod "19e5b069-944c-4a7a-b547-2dd00b060a2e" (UID: "19e5b069-944c-4a7a-b547-2dd00b060a2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:15:18.285159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.285142    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e5b069-944c-4a7a-b547-2dd00b060a2e-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:18.286736 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.286711    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e5b069-944c-4a7a-b547-2dd00b060a2e-kube-api-access-2x7w7" pod "19e5b069-944c-4a7a-b547-2dd00b060a2e" (UID: "19e5b069-944c-4a7a-b547-2dd00b060a2e"). InnerVolumeSpecName "kube-api-access-2x7w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:15:18.387578 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.387546    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x7w7\" (UniqueName: \"kubernetes.io/projected/19e5b069-944c-4a7a-b547-2dd00b060a2e-kube-api-access-2x7w7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:18.685057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.685025    5508 scope.go:122] "RemoveContainer" containerID="3cdda3bbcf437b25c60ee5db5def7721a802de6c8fd88ba08fd9320e6e4e5197"
Nov 03 02:15:18.694900 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.694877    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec2d52e-2bde-429f-8adc-31be523c9220" path="/var/lib/kubelet/pods/1ec2d52e-2bde-429f-8adc-31be523c9220/volumes"
Nov 03 02:15:18.946888 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.946853    5508 generic.go:356] "Generic (PLEG): container finished" podID="8fe6ee18-4681-4e63-9675-a098c98a7aea" containerID="826317d8b5c0a562c92b4d7e0c35e9910d47dc294ca3e4edc72844395c475118" exitCode=60
Nov 03 02:15:18.947077 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.946945    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g" event={"ID":"8fe6ee18-4681-4e63-9675-a098c98a7aea","Type":"ContainerDied","Data":"826317d8b5c0a562c92b4d7e0c35e9910d47dc294ca3e4edc72844395c475118"}
Nov 03 02:15:18.947077 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.947002    5508 scope.go:122] "RemoveContainer" containerID="13d9e543ce89a63429bb0fdd7cf9102e262625c80db2b249aa85d07137095993"
Nov 03 02:15:18.947428 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.947410    5508 scope.go:122] "RemoveContainer" containerID="826317d8b5c0a562c92b4d7e0c35e9910d47dc294ca3e4edc72844395c475118"
Nov 03 02:15:18.947740 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:18.947720    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g_odh-ai-gateway-infra(8fe6ee18-4681-4e63-9675-a098c98a7aea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g" podUID="8fe6ee18-4681-4e63-9675-a098c98a7aea"
Nov 03 02:15:18.959731 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.959696    5508 generic.go:356] "Generic (PLEG): container finished" podID="a2656448-c23b-42f9-b85d-145af6aa7426" containerID="180d90c047e6447d8d074d2b2719898abbd493f6fc2233f9f4d2c128f63c86e5" exitCode=60
Nov 03 02:15:18.959827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.959787    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw" event={"ID":"a2656448-c23b-42f9-b85d-145af6aa7426","Type":"ContainerDied","Data":"180d90c047e6447d8d074d2b2719898abbd493f6fc2233f9f4d2c128f63c86e5"}
Nov 03 02:15:18.960276 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.960255    5508 scope.go:122] "RemoveContainer" containerID="180d90c047e6447d8d074d2b2719898abbd493f6fc2233f9f4d2c128f63c86e5"
Nov 03 02:15:18.960701 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:18.960664    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw_odh-ai-gateway-infra(a2656448-c23b-42f9-b85d-145af6aa7426)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw" podUID="a2656448-c23b-42f9-b85d-145af6aa7426"
Nov 03 02:15:18.962049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.962029    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"
Nov 03 02:15:18.962181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.962152    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8" event={"ID":"19e5b069-944c-4a7a-b547-2dd00b060a2e","Type":"ContainerDied","Data":"b9dfad36cc1696ff4bf2f1774a4aa19d94d3b2bec84a891ae63968bbc05bf078"}
Nov 03 02:15:18.964816 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.964720    5508 scope.go:122] "RemoveContainer" containerID="54c65803a12cc3f2ec613bc02c3e5173bf65683a026424f406c728b4dd7ba8d4"
Nov 03 02:15:18.980655 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:18.980633    5508 scope.go:122] "RemoveContainer" containerID="e265003c3a6118055dfcaa3274b7a5138c237e928dde7e82845e998a8f1e6293"
Nov 03 02:15:19.211323 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.211286    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"]
Nov 03 02:15:19.277076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.273606    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"]
Nov 03 02:15:19.279194 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.279157    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894535-v8tc8"]
Nov 03 02:15:19.353503 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.353479    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"
Nov 03 02:15:19.400317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.400287    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0337f2e-ae3a-4815-9c06-2abf4fb84490-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0337f2e-ae3a-4815-9c06-2abf4fb84490-maas-api-service-ca\") pod \"a0337f2e-ae3a-4815-9c06-2abf4fb84490\" (UID: \"a0337f2e-ae3a-4815-9c06-2abf4fb84490\") "
Nov 03 02:15:19.400500 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.400427    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0337f2e-ae3a-4815-9c06-2abf4fb84490-kube-api-access-kvdvz\" (UniqueName: \"kubernetes.io/projected/a0337f2e-ae3a-4815-9c06-2abf4fb84490-kube-api-access-kvdvz\") pod \"a0337f2e-ae3a-4815-9c06-2abf4fb84490\" (UID: \"a0337f2e-ae3a-4815-9c06-2abf4fb84490\") "
Nov 03 02:15:19.400684 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.400663    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0337f2e-ae3a-4815-9c06-2abf4fb84490-maas-api-service-ca" pod "a0337f2e-ae3a-4815-9c06-2abf4fb84490" (UID: "a0337f2e-ae3a-4815-9c06-2abf4fb84490"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:15:19.400910 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.400895    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0337f2e-ae3a-4815-9c06-2abf4fb84490-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:19.402599 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.402574    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0337f2e-ae3a-4815-9c06-2abf4fb84490-kube-api-access-kvdvz" pod "a0337f2e-ae3a-4815-9c06-2abf4fb84490" (UID: "a0337f2e-ae3a-4815-9c06-2abf4fb84490"). InnerVolumeSpecName "kube-api-access-kvdvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:15:19.502131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.502041    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvdvz\" (UniqueName: \"kubernetes.io/projected/a0337f2e-ae3a-4815-9c06-2abf4fb84490-kube-api-access-kvdvz\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:19.974481 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.974449    5508 generic.go:356] "Generic (PLEG): container finished" podID="e8090231-a51a-4f90-bd68-97ae767ea797" containerID="e4e7b0a43b19aae0a2eae916df32df405930ee3c9617363e13eccfc0db74dffb" exitCode=60
Nov 03 02:15:19.974951 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.974525    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm" event={"ID":"e8090231-a51a-4f90-bd68-97ae767ea797","Type":"ContainerDied","Data":"e4e7b0a43b19aae0a2eae916df32df405930ee3c9617363e13eccfc0db74dffb"}
Nov 03 02:15:19.974951 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.974590    5508 scope.go:122] "RemoveContainer" containerID="3cdda3bbcf437b25c60ee5db5def7721a802de6c8fd88ba08fd9320e6e4e5197"
Nov 03 02:15:19.975108 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.974957    5508 scope.go:122] "RemoveContainer" containerID="e4e7b0a43b19aae0a2eae916df32df405930ee3c9617363e13eccfc0db74dffb"
Nov 03 02:15:19.975381 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:19.975356    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm_odh-ai-gateway-infra(e8090231-a51a-4f90-bd68-97ae767ea797)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm" podUID="e8090231-a51a-4f90-bd68-97ae767ea797"
Nov 03 02:15:19.978432 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.978407    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p" event={"ID":"a0337f2e-ae3a-4815-9c06-2abf4fb84490","Type":"ContainerDied","Data":"02fee778bde3f559d96e321412001765e81a70036d6c9b75e05060c7ad6f8732"}
Nov 03 02:15:19.978522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.978488    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"
Nov 03 02:15:19.989186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:19.989166    5508 scope.go:122] "RemoveContainer" containerID="a1db26517be9d1a4206682f0423dfd64ddf24b08ec3ea19b0dfd518d0e8c772b"
Nov 03 02:15:20.083309 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.083257    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"]
Nov 03 02:15:20.179038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.179000    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"]
Nov 03 02:15:20.192545 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.192473    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"]
Nov 03 02:15:20.217484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.217323    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-434b6a-29894535-kbl2p"]
Nov 03 02:15:20.311467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.307131    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"
Nov 03 02:15:20.349961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.349939    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"
Nov 03 02:15:20.420149 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.420116    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2656448-c23b-42f9-b85d-145af6aa7426-kube-api-access-jj99g\" (UniqueName: \"kubernetes.io/projected/a2656448-c23b-42f9-b85d-145af6aa7426-kube-api-access-jj99g\") pod \"a2656448-c23b-42f9-b85d-145af6aa7426\" (UID: \"a2656448-c23b-42f9-b85d-145af6aa7426\") "
Nov 03 02:15:20.420378 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.420200    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fe6ee18-4681-4e63-9675-a098c98a7aea-kube-api-access-b722z\" (UniqueName: \"kubernetes.io/projected/8fe6ee18-4681-4e63-9675-a098c98a7aea-kube-api-access-b722z\") pod \"8fe6ee18-4681-4e63-9675-a098c98a7aea\" (UID: \"8fe6ee18-4681-4e63-9675-a098c98a7aea\") "
Nov 03 02:15:20.420378 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.420362    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fe6ee18-4681-4e63-9675-a098c98a7aea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe6ee18-4681-4e63-9675-a098c98a7aea-maas-api-service-ca\") pod \"8fe6ee18-4681-4e63-9675-a098c98a7aea\" (UID: \"8fe6ee18-4681-4e63-9675-a098c98a7aea\") "
Nov 03 02:15:20.420527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.420412    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2656448-c23b-42f9-b85d-145af6aa7426-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2656448-c23b-42f9-b85d-145af6aa7426-maas-api-service-ca\") pod \"a2656448-c23b-42f9-b85d-145af6aa7426\" (UID: \"a2656448-c23b-42f9-b85d-145af6aa7426\") "
Nov 03 02:15:20.420808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.420784    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe6ee18-4681-4e63-9675-a098c98a7aea-maas-api-service-ca" pod "8fe6ee18-4681-4e63-9675-a098c98a7aea" (UID: "8fe6ee18-4681-4e63-9675-a098c98a7aea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:15:20.420921 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.420829    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2656448-c23b-42f9-b85d-145af6aa7426-maas-api-service-ca" pod "a2656448-c23b-42f9-b85d-145af6aa7426" (UID: "a2656448-c23b-42f9-b85d-145af6aa7426"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:15:20.420998 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.420979    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fe6ee18-4681-4e63-9675-a098c98a7aea-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:20.421053 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.421003    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2656448-c23b-42f9-b85d-145af6aa7426-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:20.422557 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.422534    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe6ee18-4681-4e63-9675-a098c98a7aea-kube-api-access-b722z" pod "8fe6ee18-4681-4e63-9675-a098c98a7aea" (UID: "8fe6ee18-4681-4e63-9675-a098c98a7aea"). InnerVolumeSpecName "kube-api-access-b722z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:15:20.422662 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.422563    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2656448-c23b-42f9-b85d-145af6aa7426-kube-api-access-jj99g" pod "a2656448-c23b-42f9-b85d-145af6aa7426" (UID: "a2656448-c23b-42f9-b85d-145af6aa7426"). InnerVolumeSpecName "kube-api-access-jj99g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:15:20.522125 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.522034    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj99g\" (UniqueName: \"kubernetes.io/projected/a2656448-c23b-42f9-b85d-145af6aa7426-kube-api-access-jj99g\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:20.522125 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.522063    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b722z\" (UniqueName: \"kubernetes.io/projected/8fe6ee18-4681-4e63-9675-a098c98a7aea-kube-api-access-b722z\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:20.697164 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.697123    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e5b069-944c-4a7a-b547-2dd00b060a2e" path="/var/lib/kubelet/pods/19e5b069-944c-4a7a-b547-2dd00b060a2e/volumes"
Nov 03 02:15:20.697799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:20.697781    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0337f2e-ae3a-4815-9c06-2abf4fb84490" path="/var/lib/kubelet/pods/a0337f2e-ae3a-4815-9c06-2abf4fb84490/volumes"
Nov 03 02:15:21.001004 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.000965    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g" event={"ID":"8fe6ee18-4681-4e63-9675-a098c98a7aea","Type":"ContainerDied","Data":"f3e4947c22e20140a6c59cde65695de4fbc4f5e94cdf617681afc0731be9f694"}
Nov 03 02:15:21.001448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.001036    5508 scope.go:122] "RemoveContainer" containerID="826317d8b5c0a562c92b4d7e0c35e9910d47dc294ca3e4edc72844395c475118"
Nov 03 02:15:21.001448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.000978    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"
Nov 03 02:15:21.004184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.003904    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw" event={"ID":"a2656448-c23b-42f9-b85d-145af6aa7426","Type":"ContainerDied","Data":"5834f016733514bde96ee657ea43d72caa7456bfa20efe07db80a9071982d0d7"}
Nov 03 02:15:21.004184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.003999    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"
Nov 03 02:15:21.012128 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.012109    5508 scope.go:122] "RemoveContainer" containerID="180d90c047e6447d8d074d2b2719898abbd493f6fc2233f9f4d2c128f63c86e5"
Nov 03 02:15:21.097737 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.097702    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"]
Nov 03 02:15:21.158782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.158748    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"]
Nov 03 02:15:21.162999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.162744    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894535-g9g2g"]
Nov 03 02:15:21.265007 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.264977    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"]
Nov 03 02:15:21.269506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.269476    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894535-ktxvw"]
Nov 03 02:15:21.274809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.274786    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"
Nov 03 02:15:21.331242 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.331208    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8090231-a51a-4f90-bd68-97ae767ea797-kube-api-access-r9sl2\" (UniqueName: \"kubernetes.io/projected/e8090231-a51a-4f90-bd68-97ae767ea797-kube-api-access-r9sl2\") pod \"e8090231-a51a-4f90-bd68-97ae767ea797\" (UID: \"e8090231-a51a-4f90-bd68-97ae767ea797\") "
Nov 03 02:15:21.331449 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.331380    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8090231-a51a-4f90-bd68-97ae767ea797-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8090231-a51a-4f90-bd68-97ae767ea797-maas-api-service-ca\") pod \"e8090231-a51a-4f90-bd68-97ae767ea797\" (UID: \"e8090231-a51a-4f90-bd68-97ae767ea797\") "
Nov 03 02:15:21.331782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.331757    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8090231-a51a-4f90-bd68-97ae767ea797-maas-api-service-ca" pod "e8090231-a51a-4f90-bd68-97ae767ea797" (UID: "e8090231-a51a-4f90-bd68-97ae767ea797"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:15:21.331982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.331963    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8090231-a51a-4f90-bd68-97ae767ea797-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:21.333569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.333542    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8090231-a51a-4f90-bd68-97ae767ea797-kube-api-access-r9sl2" pod "e8090231-a51a-4f90-bd68-97ae767ea797" (UID: "e8090231-a51a-4f90-bd68-97ae767ea797"). InnerVolumeSpecName "kube-api-access-r9sl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:15:21.433004 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:21.432970    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9sl2\" (UniqueName: \"kubernetes.io/projected/e8090231-a51a-4f90-bd68-97ae767ea797-kube-api-access-r9sl2\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:22.012030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:22.011998    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"
Nov 03 02:15:22.012537 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:22.012029    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm" event={"ID":"e8090231-a51a-4f90-bd68-97ae767ea797","Type":"ContainerDied","Data":"213df75ab775cec1f666348fcacc2add62fdf62abffb8b47627315d3995a580a"}
Nov 03 02:15:22.012537 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:22.012095    5508 scope.go:122] "RemoveContainer" containerID="e4e7b0a43b19aae0a2eae916df32df405930ee3c9617363e13eccfc0db74dffb"
Nov 03 02:15:22.166288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:22.166253    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"]
Nov 03 02:15:22.170751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:22.170728    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1c9b4e-29894535-jhcqm"]
Nov 03 02:15:22.693917 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:22.693881    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fe6ee18-4681-4e63-9675-a098c98a7aea" path="/var/lib/kubelet/pods/8fe6ee18-4681-4e63-9675-a098c98a7aea/volumes"
Nov 03 02:15:22.694335 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:22.694321    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2656448-c23b-42f9-b85d-145af6aa7426" path="/var/lib/kubelet/pods/a2656448-c23b-42f9-b85d-145af6aa7426/volumes"
Nov 03 02:15:22.694754 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:22.694740    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8090231-a51a-4f90-bd68-97ae767ea797" path="/var/lib/kubelet/pods/e8090231-a51a-4f90-bd68-97ae767ea797/volumes"
Nov 03 02:15:23.372670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.372637    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"]
Nov 03 02:15:23.373211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373196    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a0337f2e-ae3a-4815-9c06-2abf4fb84490" containerName="cleanup"
Nov 03 02:15:23.373316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373212    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0337f2e-ae3a-4815-9c06-2abf4fb84490" containerName="cleanup"
Nov 03 02:15:23.373316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373227    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8fe6ee18-4681-4e63-9675-a098c98a7aea" containerName="cleanup"
Nov 03 02:15:23.373316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373232    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe6ee18-4681-4e63-9675-a098c98a7aea" containerName="cleanup"
Nov 03 02:15:23.373316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373242    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e8090231-a51a-4f90-bd68-97ae767ea797" containerName="cleanup"
Nov 03 02:15:23.373316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373255    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8090231-a51a-4f90-bd68-97ae767ea797" containerName="cleanup"
Nov 03 02:15:23.373316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373269    5508 cpu_manager.go:414] "containerMap: removing container" podUID="1ec2d52e-2bde-429f-8adc-31be523c9220" containerName="cleanup"
Nov 03 02:15:23.373316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373274    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec2d52e-2bde-429f-8adc-31be523c9220" containerName="cleanup"
Nov 03 02:15:23.373316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373283    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a0337f2e-ae3a-4815-9c06-2abf4fb84490" containerName="cleanup"
Nov 03 02:15:23.373316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373288    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0337f2e-ae3a-4815-9c06-2abf4fb84490" containerName="cleanup"
Nov 03 02:15:23.373316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373300    5508 cpu_manager.go:414] "containerMap: removing container" podUID="19e5b069-944c-4a7a-b547-2dd00b060a2e" containerName="cleanup"
Nov 03 02:15:23.373316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373309    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e5b069-944c-4a7a-b547-2dd00b060a2e" containerName="cleanup"
Nov 03 02:15:23.373316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373317    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e8090231-a51a-4f90-bd68-97ae767ea797" containerName="cleanup"
Nov 03 02:15:23.373316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373322    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8090231-a51a-4f90-bd68-97ae767ea797" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373333    5508 cpu_manager.go:414] "containerMap: removing container" podUID="1ec2d52e-2bde-429f-8adc-31be523c9220" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373338    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec2d52e-2bde-429f-8adc-31be523c9220" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373344    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e8090231-a51a-4f90-bd68-97ae767ea797" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373364    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8090231-a51a-4f90-bd68-97ae767ea797" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373389    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a2656448-c23b-42f9-b85d-145af6aa7426" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373394    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2656448-c23b-42f9-b85d-145af6aa7426" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373402    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a2656448-c23b-42f9-b85d-145af6aa7426" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373406    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2656448-c23b-42f9-b85d-145af6aa7426" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373414    5508 cpu_manager.go:414] "containerMap: removing container" podUID="1ec2d52e-2bde-429f-8adc-31be523c9220" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373419    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec2d52e-2bde-429f-8adc-31be523c9220" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373425    5508 cpu_manager.go:414] "containerMap: removing container" podUID="19e5b069-944c-4a7a-b547-2dd00b060a2e" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373429    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e5b069-944c-4a7a-b547-2dd00b060a2e" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373437    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8fe6ee18-4681-4e63-9675-a098c98a7aea" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373444    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe6ee18-4681-4e63-9675-a098c98a7aea" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373459    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a2656448-c23b-42f9-b85d-145af6aa7426" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373466    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2656448-c23b-42f9-b85d-145af6aa7426" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373601    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2656448-c23b-42f9-b85d-145af6aa7426" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373614    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2656448-c23b-42f9-b85d-145af6aa7426" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373621    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e5b069-944c-4a7a-b547-2dd00b060a2e" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373631    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec2d52e-2bde-429f-8adc-31be523c9220" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373639    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e5b069-944c-4a7a-b547-2dd00b060a2e" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373647    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fe6ee18-4681-4e63-9675-a098c98a7aea" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373658    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2656448-c23b-42f9-b85d-145af6aa7426" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373670    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0337f2e-ae3a-4815-9c06-2abf4fb84490" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373678    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fe6ee18-4681-4e63-9675-a098c98a7aea" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373691    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fe6ee18-4681-4e63-9675-a098c98a7aea" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373701    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8090231-a51a-4f90-bd68-97ae767ea797" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373709    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0337f2e-ae3a-4815-9c06-2abf4fb84490" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373716    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8090231-a51a-4f90-bd68-97ae767ea797" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373722    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec2d52e-2bde-429f-8adc-31be523c9220" containerName="cleanup"
Nov 03 02:15:23.374050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.373727    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8090231-a51a-4f90-bd68-97ae767ea797" containerName="cleanup"
Nov 03 02:15:23.378771 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.378747    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"
Nov 03 02:15:23.427579 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.427552    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:15:23.455801 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.455769    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e47db7-jd6vd\" (UID: \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"
Nov 03 02:15:23.455970 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.455830    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e47db7-jd6vd\" (UID: \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"
Nov 03 02:15:23.456117 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.456092    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9df9\" (UniqueName: \"kubernetes.io/projected/8b00055e-a35c-4c0e-8ce7-1758f5dea910-kube-api-access-m9df9\") pod \"maas-api-revoke-sub-10e47db7-jd6vd\" (UID: \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"
Nov 03 02:15:23.473751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.473722    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:15:23.477272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.477242    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"]
Nov 03 02:15:23.557541 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.557496    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e47db7-jd6vd\" (UID: \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"
Nov 03 02:15:23.557731 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.557564    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e47db7-jd6vd\" (UID: \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"
Nov 03 02:15:23.557817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.557765    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9df9\" (UniqueName: \"kubernetes.io/projected/8b00055e-a35c-4c0e-8ce7-1758f5dea910-kube-api-access-m9df9\") pod \"maas-api-revoke-sub-10e47db7-jd6vd\" (UID: \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"
Nov 03 02:15:23.558288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.558264    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e47db7-jd6vd\" (UID: \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"
Nov 03 02:15:23.560086 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.560063    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e47db7-jd6vd\" (UID: \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"
Nov 03 02:15:23.615656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.615621    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9df9\" (UniqueName: \"kubernetes.io/projected/8b00055e-a35c-4c0e-8ce7-1758f5dea910-kube-api-access-m9df9\") pod \"maas-api-revoke-sub-10e47db7-jd6vd\" (UID: \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"
Nov 03 02:15:23.695536 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:23.695495    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"
Nov 03 02:15:24.079137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:24.079046    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"]
Nov 03 02:15:24.084125 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:15:24.084101    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b00055e_a35c_4c0e_8ce7_1758f5dea910.slice/crio-ebba6ce4a6437634a27f72e41612acb4c43b1fbd28682fa5adf4a9916f190815 WatchSource:0}: Error finding container ebba6ce4a6437634a27f72e41612acb4c43b1fbd28682fa5adf4a9916f190815: Status 404 returned error can't find the container with id ebba6ce4a6437634a27f72e41612acb4c43b1fbd28682fa5adf4a9916f190815
Nov 03 02:15:25.038557 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:25.038522    5508 generic.go:356] "Generic (PLEG): container finished" podID="8b00055e-a35c-4c0e-8ce7-1758f5dea910" containerID="db391f1d3f1d5e5015a5a3d055b3a2bbc870aadfec7be38b4443d42bafb839eb" exitCode=0
Nov 03 02:15:25.039003 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:25.038606    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd" event={"ID":"8b00055e-a35c-4c0e-8ce7-1758f5dea910","Type":"ContainerDied","Data":"db391f1d3f1d5e5015a5a3d055b3a2bbc870aadfec7be38b4443d42bafb839eb"}
Nov 03 02:15:25.039003 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:25.038662    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd" event={"ID":"8b00055e-a35c-4c0e-8ce7-1758f5dea910","Type":"ContainerStarted","Data":"ebba6ce4a6437634a27f72e41612acb4c43b1fbd28682fa5adf4a9916f190815"}
Nov 03 02:15:26.225030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:26.225001    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"
Nov 03 02:15:26.300381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:26.300280    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b00055e-a35c-4c0e-8ce7-1758f5dea910-kube-api-access-m9df9\" (UniqueName: \"kubernetes.io/projected/8b00055e-a35c-4c0e-8ce7-1758f5dea910-kube-api-access-m9df9\") pod \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\" (UID: \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\") "
Nov 03 02:15:26.300572 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:26.300386    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-xks-ca\") pod \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\" (UID: \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\") "
Nov 03 02:15:26.300572 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:26.300458    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-openshift-ca\") pod \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\" (UID: \"8b00055e-a35c-4c0e-8ce7-1758f5dea910\") "
Nov 03 02:15:26.300871 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:26.300843    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-openshift-ca" pod "8b00055e-a35c-4c0e-8ce7-1758f5dea910" (UID: "8b00055e-a35c-4c0e-8ce7-1758f5dea910"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:15:26.301139 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:26.301120    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:26.303042 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:26.303013    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-xks-ca" pod "8b00055e-a35c-4c0e-8ce7-1758f5dea910" (UID: "8b00055e-a35c-4c0e-8ce7-1758f5dea910"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:15:26.303153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:26.303048    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b00055e-a35c-4c0e-8ce7-1758f5dea910-kube-api-access-m9df9" pod "8b00055e-a35c-4c0e-8ce7-1758f5dea910" (UID: "8b00055e-a35c-4c0e-8ce7-1758f5dea910"). InnerVolumeSpecName "kube-api-access-m9df9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:15:26.402070 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:26.402033    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9df9\" (UniqueName: \"kubernetes.io/projected/8b00055e-a35c-4c0e-8ce7-1758f5dea910-kube-api-access-m9df9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:26.402070 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:26.402064    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b00055e-a35c-4c0e-8ce7-1758f5dea910-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:27.053644 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:27.053551    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"
Nov 03 02:15:27.053644 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:27.053580    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd" event={"ID":"8b00055e-a35c-4c0e-8ce7-1758f5dea910","Type":"ContainerDied","Data":"ebba6ce4a6437634a27f72e41612acb4c43b1fbd28682fa5adf4a9916f190815"}
Nov 03 02:15:27.053644 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:27.053639    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebba6ce4a6437634a27f72e41612acb4c43b1fbd28682fa5adf4a9916f190815"
Nov 03 02:15:34.097516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.097460    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"]
Nov 03 02:15:34.098448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.098424    5508 cpu_manager.go:414] "containerMap: removing container" podUID="19e5b069-944c-4a7a-b547-2dd00b060a2e" containerName="cleanup"
Nov 03 02:15:34.098519 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.098451    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e5b069-944c-4a7a-b547-2dd00b060a2e" containerName="cleanup"
Nov 03 02:15:34.098519 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.098469    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8fe6ee18-4681-4e63-9675-a098c98a7aea" containerName="cleanup"
Nov 03 02:15:34.098519 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.098479    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fe6ee18-4681-4e63-9675-a098c98a7aea" containerName="cleanup"
Nov 03 02:15:34.098625 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.098524    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8b00055e-a35c-4c0e-8ce7-1758f5dea910" containerName="revoke-keys"
Nov 03 02:15:34.098625 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.098532    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b00055e-a35c-4c0e-8ce7-1758f5dea910" containerName="revoke-keys"
Nov 03 02:15:34.098625 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.098553    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a0337f2e-ae3a-4815-9c06-2abf4fb84490" containerName="cleanup"
Nov 03 02:15:34.098625 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.098559    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0337f2e-ae3a-4815-9c06-2abf4fb84490" containerName="cleanup"
Nov 03 02:15:34.098752 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.098717    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e5b069-944c-4a7a-b547-2dd00b060a2e" containerName="cleanup"
Nov 03 02:15:34.098752 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.098733    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0337f2e-ae3a-4815-9c06-2abf4fb84490" containerName="cleanup"
Nov 03 02:15:34.098752 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.098748    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b00055e-a35c-4c0e-8ce7-1758f5dea910" containerName="revoke-keys"
Nov 03 02:15:34.098909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.098764    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec2d52e-2bde-429f-8adc-31be523c9220" containerName="cleanup"
Nov 03 02:15:34.103666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.103644    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"
Nov 03 02:15:34.153221 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.153182    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:15:34.199563 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.199525    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:15:34.204358 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.204316    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"]
Nov 03 02:15:34.204506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.204470    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28d39226-d46jf\" (UID: \"e08feef3-9db7-4c18-8c3b-075d6ee44667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"
Nov 03 02:15:34.204659 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.204644    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q56s\" (UniqueName: \"kubernetes.io/projected/e08feef3-9db7-4c18-8c3b-075d6ee44667-kube-api-access-9q56s\") pod \"maas-api-revoke-sub-28d39226-d46jf\" (UID: \"e08feef3-9db7-4c18-8c3b-075d6ee44667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"
Nov 03 02:15:34.204700 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.204687    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28d39226-d46jf\" (UID: \"e08feef3-9db7-4c18-8c3b-075d6ee44667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"
Nov 03 02:15:34.305992 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.305951    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q56s\" (UniqueName: \"kubernetes.io/projected/e08feef3-9db7-4c18-8c3b-075d6ee44667-kube-api-access-9q56s\") pod \"maas-api-revoke-sub-28d39226-d46jf\" (UID: \"e08feef3-9db7-4c18-8c3b-075d6ee44667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"
Nov 03 02:15:34.306211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.306035    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28d39226-d46jf\" (UID: \"e08feef3-9db7-4c18-8c3b-075d6ee44667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"
Nov 03 02:15:34.306211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.306172    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28d39226-d46jf\" (UID: \"e08feef3-9db7-4c18-8c3b-075d6ee44667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"
Nov 03 02:15:34.306953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.306920    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28d39226-d46jf\" (UID: \"e08feef3-9db7-4c18-8c3b-075d6ee44667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"
Nov 03 02:15:34.309105 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.309079    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28d39226-d46jf\" (UID: \"e08feef3-9db7-4c18-8c3b-075d6ee44667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"
Nov 03 02:15:34.364434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.364339    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q56s\" (UniqueName: \"kubernetes.io/projected/e08feef3-9db7-4c18-8c3b-075d6ee44667-kube-api-access-9q56s\") pod \"maas-api-revoke-sub-28d39226-d46jf\" (UID: \"e08feef3-9db7-4c18-8c3b-075d6ee44667\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"
Nov 03 02:15:34.418098 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.418059    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"
Nov 03 02:15:34.471368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.471325    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:15:34.820883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:34.820841    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"]
Nov 03 02:15:34.826459 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:15:34.826429    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08feef3_9db7_4c18_8c3b_075d6ee44667.slice/crio-c1a888a5bfaad0a8c094fffe86af45decf447ab6c95f2f50c7865a76f086a753 WatchSource:0}: Error finding container c1a888a5bfaad0a8c094fffe86af45decf447ab6c95f2f50c7865a76f086a753: Status 404 returned error can't find the container with id c1a888a5bfaad0a8c094fffe86af45decf447ab6c95f2f50c7865a76f086a753
Nov 03 02:15:35.119163 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:35.119064    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf" event={"ID":"e08feef3-9db7-4c18-8c3b-075d6ee44667","Type":"ContainerStarted","Data":"c1a888a5bfaad0a8c094fffe86af45decf447ab6c95f2f50c7865a76f086a753"}
Nov 03 02:15:36.128204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:36.128113    5508 generic.go:356] "Generic (PLEG): container finished" podID="e08feef3-9db7-4c18-8c3b-075d6ee44667" containerID="4727eff282d10d10c5b62c03454c5f95c71f66a3fe7b5a1c5ab60bc776c905a2" exitCode=0
Nov 03 02:15:36.128616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:36.128199    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf" event={"ID":"e08feef3-9db7-4c18-8c3b-075d6ee44667","Type":"ContainerDied","Data":"4727eff282d10d10c5b62c03454c5f95c71f66a3fe7b5a1c5ab60bc776c905a2"}
Nov 03 02:15:37.286527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:37.286497    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"
Nov 03 02:15:37.453277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:37.453235    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08feef3-9db7-4c18-8c3b-075d6ee44667-kube-api-access-9q56s\" (UniqueName: \"kubernetes.io/projected/e08feef3-9db7-4c18-8c3b-075d6ee44667-kube-api-access-9q56s\") pod \"e08feef3-9db7-4c18-8c3b-075d6ee44667\" (UID: \"e08feef3-9db7-4c18-8c3b-075d6ee44667\") "
Nov 03 02:15:37.453525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:37.453375    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-openshift-ca\") pod \"e08feef3-9db7-4c18-8c3b-075d6ee44667\" (UID: \"e08feef3-9db7-4c18-8c3b-075d6ee44667\") "
Nov 03 02:15:37.453525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:37.453474    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-xks-ca\") pod \"e08feef3-9db7-4c18-8c3b-075d6ee44667\" (UID: \"e08feef3-9db7-4c18-8c3b-075d6ee44667\") "
Nov 03 02:15:37.453719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:37.453696    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-openshift-ca" pod "e08feef3-9db7-4c18-8c3b-075d6ee44667" (UID: "e08feef3-9db7-4c18-8c3b-075d6ee44667"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:15:37.453974 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:37.453952    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:37.455606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:37.455580    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08feef3-9db7-4c18-8c3b-075d6ee44667-kube-api-access-9q56s" pod "e08feef3-9db7-4c18-8c3b-075d6ee44667" (UID: "e08feef3-9db7-4c18-8c3b-075d6ee44667"). InnerVolumeSpecName "kube-api-access-9q56s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:15:37.455700 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:37.455602    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-xks-ca" pod "e08feef3-9db7-4c18-8c3b-075d6ee44667" (UID: "e08feef3-9db7-4c18-8c3b-075d6ee44667"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:15:37.554971 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:37.554931    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q56s\" (UniqueName: \"kubernetes.io/projected/e08feef3-9db7-4c18-8c3b-075d6ee44667-kube-api-access-9q56s\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:37.554971 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:37.554967    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e08feef3-9db7-4c18-8c3b-075d6ee44667-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:15:38.144137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:38.144105    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"
Nov 03 02:15:38.144326 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:38.144109    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf" event={"ID":"e08feef3-9db7-4c18-8c3b-075d6ee44667","Type":"ContainerDied","Data":"c1a888a5bfaad0a8c094fffe86af45decf447ab6c95f2f50c7865a76f086a753"}
Nov 03 02:15:38.144326 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:38.144231    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1a888a5bfaad0a8c094fffe86af45decf447ab6c95f2f50c7865a76f086a753"
Nov 03 02:15:43.812776 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:15:43.812740    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:15:45.671862 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:45.671829    5508 scope.go:122] "RemoveContainer" containerID="d465401c71cfa9e50adad42195c7d8891a066c0a186c4d9f0f887446ed40f1d3"
Nov 03 02:15:57.878500 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:15:57.878464    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"]
Nov 03 02:16:12.899455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:12.899406    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp" podUID="e054f5c9-0628-45b3-8c8a-956d5b305518" containerName="main" containerID="cri-o://e0673802b1af0001bbe669eb4b1b9e94d3f9439fbe261c1d29afae930fd3c072" gracePeriod=45
Nov 03 02:16:13.186556 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.186525    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:16:13.260545 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.260508    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-dshm\") pod \"e054f5c9-0628-45b3-8c8a-956d5b305518\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") "
Nov 03 02:16:13.260755 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.260575    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e054f5c9-0628-45b3-8c8a-956d5b305518-kube-api-access-972xs\" (UniqueName: \"kubernetes.io/projected/e054f5c9-0628-45b3-8c8a-956d5b305518-kube-api-access-972xs\") pod \"e054f5c9-0628-45b3-8c8a-956d5b305518\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") "
Nov 03 02:16:13.260755 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.260614    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-tmp-dir\") pod \"e054f5c9-0628-45b3-8c8a-956d5b305518\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") "
Nov 03 02:16:13.260755 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.260722    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e054f5c9-0628-45b3-8c8a-956d5b305518-tls-certs\" (UniqueName: \"kubernetes.io/secret/e054f5c9-0628-45b3-8c8a-956d5b305518-tls-certs\") pod \"e054f5c9-0628-45b3-8c8a-956d5b305518\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") "
Nov 03 02:16:13.260942 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.260833    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-model-cache\") pod \"e054f5c9-0628-45b3-8c8a-956d5b305518\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") "
Nov 03 02:16:13.260942 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.260898    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-home\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-home\") pod \"e054f5c9-0628-45b3-8c8a-956d5b305518\" (UID: \"e054f5c9-0628-45b3-8c8a-956d5b305518\") "
Nov 03 02:16:13.261149 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.261109    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-model-cache" pod "e054f5c9-0628-45b3-8c8a-956d5b305518" (UID: "e054f5c9-0628-45b3-8c8a-956d5b305518"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:13.261310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.261179    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-tmp-dir" pod "e054f5c9-0628-45b3-8c8a-956d5b305518" (UID: "e054f5c9-0628-45b3-8c8a-956d5b305518"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:13.261404 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.261368    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-home" pod "e054f5c9-0628-45b3-8c8a-956d5b305518" (UID: "e054f5c9-0628-45b3-8c8a-956d5b305518"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:13.261522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.261494    5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-tmp-dir\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:13.261522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.261518    5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-model-cache\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:13.261638 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.261532    5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-home\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:13.263104 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.263077    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e054f5c9-0628-45b3-8c8a-956d5b305518-kube-api-access-972xs" pod "e054f5c9-0628-45b3-8c8a-956d5b305518" (UID: "e054f5c9-0628-45b3-8c8a-956d5b305518"). InnerVolumeSpecName "kube-api-access-972xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:13.263239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.263219    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-dshm" pod "e054f5c9-0628-45b3-8c8a-956d5b305518" (UID: "e054f5c9-0628-45b3-8c8a-956d5b305518"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:13.263340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.263323    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e054f5c9-0628-45b3-8c8a-956d5b305518-tls-certs" pod "e054f5c9-0628-45b3-8c8a-956d5b305518" (UID: "e054f5c9-0628-45b3-8c8a-956d5b305518"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:16:13.362960 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.362918    5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e054f5c9-0628-45b3-8c8a-956d5b305518-tls-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:13.362960 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.362958    5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e054f5c9-0628-45b3-8c8a-956d5b305518-dshm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:13.363187 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.362973    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-972xs\" (UniqueName: \"kubernetes.io/projected/e054f5c9-0628-45b3-8c8a-956d5b305518-kube-api-access-972xs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:13.416396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.416292    5508 generic.go:356] "Generic (PLEG): container finished" podID="e054f5c9-0628-45b3-8c8a-956d5b305518" containerID="e0673802b1af0001bbe669eb4b1b9e94d3f9439fbe261c1d29afae930fd3c072" exitCode=0
Nov 03 02:16:13.416396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.416339    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"
Nov 03 02:16:13.416396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.416378    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp" event={"ID":"e054f5c9-0628-45b3-8c8a-956d5b305518","Type":"ContainerDied","Data":"e0673802b1af0001bbe669eb4b1b9e94d3f9439fbe261c1d29afae930fd3c072"}
Nov 03 02:16:13.416698 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.416435    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp" event={"ID":"e054f5c9-0628-45b3-8c8a-956d5b305518","Type":"ContainerDied","Data":"b0586454866357865aabd5c9d01abc97cc7027ed78c53fd7b491ebbc5185c97a"}
Nov 03 02:16:13.416698 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.416457    5508 scope.go:122] "RemoveContainer" containerID="e0673802b1af0001bbe669eb4b1b9e94d3f9439fbe261c1d29afae930fd3c072"
Nov 03 02:16:13.427204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.427187    5508 scope.go:122] "RemoveContainer" containerID="e0673802b1af0001bbe669eb4b1b9e94d3f9439fbe261c1d29afae930fd3c072"
Nov 03 02:16:13.427464 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:16:13.427443    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0673802b1af0001bbe669eb4b1b9e94d3f9439fbe261c1d29afae930fd3c072\": container with ID starting with e0673802b1af0001bbe669eb4b1b9e94d3f9439fbe261c1d29afae930fd3c072 not found: ID does not exist" containerID="e0673802b1af0001bbe669eb4b1b9e94d3f9439fbe261c1d29afae930fd3c072"
Nov 03 02:16:13.427562 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.427473    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0673802b1af0001bbe669eb4b1b9e94d3f9439fbe261c1d29afae930fd3c072"} err="failed to get container status \"e0673802b1af0001bbe669eb4b1b9e94d3f9439fbe261c1d29afae930fd3c072\": rpc error: code = NotFound desc = could not find container \"e0673802b1af0001bbe669eb4b1b9e94d3f9439fbe261c1d29afae930fd3c072\": container with ID starting with e0673802b1af0001bbe669eb4b1b9e94d3f9439fbe261c1d29afae930fd3c072 not found: ID does not exist"
Nov 03 02:16:13.572604 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.572571    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"]
Nov 03 02:16:13.577383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:13.577328    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-98a394/auth-test-model-98a394-kserve-75b977bcc-njnxp"]
Nov 03 02:16:13.823105 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:16:13.823014    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:16:14.695225 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:14.695186    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e054f5c9-0628-45b3-8c8a-956d5b305518" path="/var/lib/kubelet/pods/e054f5c9-0628-45b3-8c8a-956d5b305518/volumes"
Nov 03 02:16:18.127044 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:18.127012    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"]
Nov 03 02:16:27.355020 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.354969    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"]
Nov 03 02:16:27.355821 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.355800    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e054f5c9-0628-45b3-8c8a-956d5b305518" containerName="main"
Nov 03 02:16:27.355821 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.355822    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e054f5c9-0628-45b3-8c8a-956d5b305518" containerName="main"
Nov 03 02:16:27.355961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.355849    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e08feef3-9db7-4c18-8c3b-075d6ee44667" containerName="revoke-keys"
Nov 03 02:16:27.355961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.355856    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08feef3-9db7-4c18-8c3b-075d6ee44667" containerName="revoke-keys"
Nov 03 02:16:27.355961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.355935    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e054f5c9-0628-45b3-8c8a-956d5b305518" containerName="main"
Nov 03 02:16:27.355961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.355947    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08feef3-9db7-4c18-8c3b-075d6ee44667" containerName="revoke-keys"
Nov 03 02:16:27.361205 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.361183    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"
Nov 03 02:16:27.416489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.416397    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"]
Nov 03 02:16:27.489656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.489624    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-50ca6d\""
Nov 03 02:16:27.490009 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.489988    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-50ca6d-dockercfg-h7fds\""
Nov 03 02:16:27.524376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.524324    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt8tc\" (UniqueName: \"kubernetes.io/projected/641e8ee2-7527-4832-9e44-72dbb37b208a-kube-api-access-wt8tc\") pod \"payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9\" (UID: \"641e8ee2-7527-4832-9e44-72dbb37b208a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"
Nov 03 02:16:27.525483 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.525451    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641e8ee2-7527-4832-9e44-72dbb37b208a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9\" (UID: \"641e8ee2-7527-4832-9e44-72dbb37b208a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"
Nov 03 02:16:27.626810 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.626721    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641e8ee2-7527-4832-9e44-72dbb37b208a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9\" (UID: \"641e8ee2-7527-4832-9e44-72dbb37b208a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"
Nov 03 02:16:27.626996 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.626932    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt8tc\" (UniqueName: \"kubernetes.io/projected/641e8ee2-7527-4832-9e44-72dbb37b208a-kube-api-access-wt8tc\") pod \"payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9\" (UID: \"641e8ee2-7527-4832-9e44-72dbb37b208a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"
Nov 03 02:16:27.627457 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.627438    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641e8ee2-7527-4832-9e44-72dbb37b208a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9\" (UID: \"641e8ee2-7527-4832-9e44-72dbb37b208a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"
Nov 03 02:16:27.685054 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.685021    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt8tc\" (UniqueName: \"kubernetes.io/projected/641e8ee2-7527-4832-9e44-72dbb37b208a-kube-api-access-wt8tc\") pod \"payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9\" (UID: \"641e8ee2-7527-4832-9e44-72dbb37b208a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"
Nov 03 02:16:27.975943 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:27.975914    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"
Nov 03 02:16:28.364199 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:28.364066    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"]
Nov 03 02:16:28.371224 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:16:28.371190    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641e8ee2_7527_4832_9e44_72dbb37b208a.slice/crio-ecb3a559b083eafab32c14859a91e1a20f8c5ac319491cbc055ac3b6b8146a6e WatchSource:0}: Error finding container ecb3a559b083eafab32c14859a91e1a20f8c5ac319491cbc055ac3b6b8146a6e: Status 404 returned error can't find the container with id ecb3a559b083eafab32c14859a91e1a20f8c5ac319491cbc055ac3b6b8146a6e
Nov 03 02:16:28.580734 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:28.580696    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9" event={"ID":"641e8ee2-7527-4832-9e44-72dbb37b208a","Type":"ContainerStarted","Data":"2abf57990cc49d8e7c6c6ac82e0ea7195b59e84ef114a011c950068720574e13"}
Nov 03 02:16:28.580929 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:28.580756    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9" event={"ID":"641e8ee2-7527-4832-9e44-72dbb37b208a","Type":"ContainerStarted","Data":"ecb3a559b083eafab32c14859a91e1a20f8c5ac319491cbc055ac3b6b8146a6e"}
Nov 03 02:16:28.580929 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:28.580839    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"
Nov 03 02:16:28.690319 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:28.690247    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9" podStartSLOduration=1.6902283059999998 podStartE2EDuration="1.690228306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:28.689262194 +0000 UTC m=+3468.114868675" watchObservedRunningTime="2026-11-03 02:16:28.690228306 +0000 UTC m=+3468.115834772"
Nov 03 02:16:33.173373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.168049    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w" podUID="f78e0b60-4511-45ab-914c-0d4330324f9a" containerName="main" containerID="cri-o://a196e9f1daf171066ae850b3141a9f0e13b514dc4f901b06c9a6259aaf0dbf1b" gracePeriod=45
Nov 03 02:16:33.293067 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.293023    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w" podUID="f78e0b60-4511-45ab-914c-0d4330324f9a" containerName="main" probeResult="failure" output="Get \"https://10.133.1.27:8000/ready\": dial tcp 10.133.1.27:8000: connect: connection refused"
Nov 03 02:16:33.453011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.452983    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:16:33.506417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.506384    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f78e0b60-4511-45ab-914c-0d4330324f9a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f78e0b60-4511-45ab-914c-0d4330324f9a-tls-certs\") pod \"f78e0b60-4511-45ab-914c-0d4330324f9a\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") "
Nov 03 02:16:33.506594 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.506443    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78e0b60-4511-45ab-914c-0d4330324f9a-kube-api-access-9c2tm\" (UniqueName: \"kubernetes.io/projected/f78e0b60-4511-45ab-914c-0d4330324f9a-kube-api-access-9c2tm\") pod \"f78e0b60-4511-45ab-914c-0d4330324f9a\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") "
Nov 03 02:16:33.506648 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.506619    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-home\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-home\") pod \"f78e0b60-4511-45ab-914c-0d4330324f9a\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") "
Nov 03 02:16:33.506721 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.506701    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-dshm\") pod \"f78e0b60-4511-45ab-914c-0d4330324f9a\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") "
Nov 03 02:16:33.506887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.506769    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-tmp-dir\") pod \"f78e0b60-4511-45ab-914c-0d4330324f9a\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") "
Nov 03 02:16:33.506887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.506840    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-model-cache\") pod \"f78e0b60-4511-45ab-914c-0d4330324f9a\" (UID: \"f78e0b60-4511-45ab-914c-0d4330324f9a\") "
Nov 03 02:16:33.506887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.506837    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-home" pod "f78e0b60-4511-45ab-914c-0d4330324f9a" (UID: "f78e0b60-4511-45ab-914c-0d4330324f9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:33.507079 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.507037    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-tmp-dir" pod "f78e0b60-4511-45ab-914c-0d4330324f9a" (UID: "f78e0b60-4511-45ab-914c-0d4330324f9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:33.507141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.507121    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-model-cache" pod "f78e0b60-4511-45ab-914c-0d4330324f9a" (UID: "f78e0b60-4511-45ab-914c-0d4330324f9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:33.507341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.507324    5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-model-cache\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:33.507425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.507341    5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-home\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:33.507425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.507369    5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-tmp-dir\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:33.508747 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.508729    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78e0b60-4511-45ab-914c-0d4330324f9a-kube-api-access-9c2tm" pod "f78e0b60-4511-45ab-914c-0d4330324f9a" (UID: "f78e0b60-4511-45ab-914c-0d4330324f9a"). InnerVolumeSpecName "kube-api-access-9c2tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:33.508845 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.508749    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-dshm" pod "f78e0b60-4511-45ab-914c-0d4330324f9a" (UID: "f78e0b60-4511-45ab-914c-0d4330324f9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:16:33.508845 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.508813    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f78e0b60-4511-45ab-914c-0d4330324f9a-tls-certs" pod "f78e0b60-4511-45ab-914c-0d4330324f9a" (UID: "f78e0b60-4511-45ab-914c-0d4330324f9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:16:33.609263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.609217    5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f78e0b60-4511-45ab-914c-0d4330324f9a-tls-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:33.609263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.609254    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c2tm\" (UniqueName: \"kubernetes.io/projected/f78e0b60-4511-45ab-914c-0d4330324f9a-kube-api-access-9c2tm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:33.609263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.609269    5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f78e0b60-4511-45ab-914c-0d4330324f9a-dshm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:33.670889 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.670858    5508 generic.go:356] "Generic (PLEG): container finished" podID="f78e0b60-4511-45ab-914c-0d4330324f9a" containerID="a196e9f1daf171066ae850b3141a9f0e13b514dc4f901b06c9a6259aaf0dbf1b" exitCode=0
Nov 03 02:16:33.671069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.670924    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"
Nov 03 02:16:33.671069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.670946    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w" event={"ID":"f78e0b60-4511-45ab-914c-0d4330324f9a","Type":"ContainerDied","Data":"a196e9f1daf171066ae850b3141a9f0e13b514dc4f901b06c9a6259aaf0dbf1b"}
Nov 03 02:16:33.671069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.670997    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w" event={"ID":"f78e0b60-4511-45ab-914c-0d4330324f9a","Type":"ContainerDied","Data":"dcf37d045809ec00d5c340e1a17586ec0344d8b9917b15df1659f47ed167ddcb"}
Nov 03 02:16:33.671069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.671020    5508 scope.go:122] "RemoveContainer" containerID="a196e9f1daf171066ae850b3141a9f0e13b514dc4f901b06c9a6259aaf0dbf1b"
Nov 03 02:16:33.682188 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.682072    5508 scope.go:122] "RemoveContainer" containerID="a196e9f1daf171066ae850b3141a9f0e13b514dc4f901b06c9a6259aaf0dbf1b"
Nov 03 02:16:33.682600 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:16:33.682579    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a196e9f1daf171066ae850b3141a9f0e13b514dc4f901b06c9a6259aaf0dbf1b\": container with ID starting with a196e9f1daf171066ae850b3141a9f0e13b514dc4f901b06c9a6259aaf0dbf1b not found: ID does not exist" containerID="a196e9f1daf171066ae850b3141a9f0e13b514dc4f901b06c9a6259aaf0dbf1b"
Nov 03 02:16:33.682693 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.682607    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a196e9f1daf171066ae850b3141a9f0e13b514dc4f901b06c9a6259aaf0dbf1b"} err="failed to get container status \"a196e9f1daf171066ae850b3141a9f0e13b514dc4f901b06c9a6259aaf0dbf1b\": rpc error: code = NotFound desc = could not find container \"a196e9f1daf171066ae850b3141a9f0e13b514dc4f901b06c9a6259aaf0dbf1b\": container with ID starting with a196e9f1daf171066ae850b3141a9f0e13b514dc4f901b06c9a6259aaf0dbf1b not found: ID does not exist"
Nov 03 02:16:33.825741 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.825708    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"]
Nov 03 02:16:33.834785 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:33.834758    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-6a5d73/auth-test-model-6a5d73-kserve-85f76d74fc-ndj4w"]
Nov 03 02:16:34.694730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:34.694698    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f78e0b60-4511-45ab-914c-0d4330324f9a" path="/var/lib/kubelet/pods/f78e0b60-4511-45ab-914c-0d4330324f9a/volumes"
Nov 03 02:16:39.418767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.418719    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"]
Nov 03 02:16:39.419863 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.419388    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f78e0b60-4511-45ab-914c-0d4330324f9a" containerName="main"
Nov 03 02:16:39.419863 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.419409    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78e0b60-4511-45ab-914c-0d4330324f9a" containerName="main"
Nov 03 02:16:39.419863 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.419526    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78e0b60-4511-45ab-914c-0d4330324f9a" containerName="main"
Nov 03 02:16:39.422965 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.422947    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"
Nov 03 02:16:39.480410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.471883    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d8cc55\""
Nov 03 02:16:39.497157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.497114    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx\" (UID: \"c5c694d7-f5c9-4288-93ef-ea5a4011e87c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"
Nov 03 02:16:39.497596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.497571    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcp6c\" (UniqueName: \"kubernetes.io/projected/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-kube-api-access-bcp6c\") pod \"payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx\" (UID: \"c5c694d7-f5c9-4288-93ef-ea5a4011e87c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"
Nov 03 02:16:39.525056 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.525012    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d8cc55-dockercfg-t9bzq\""
Nov 03 02:16:39.535378 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.532877    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"]
Nov 03 02:16:39.583171 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.583121    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"]
Nov 03 02:16:39.589012 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.588988    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"
Nov 03 02:16:39.590439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.590416    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"
Nov 03 02:16:39.598926 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.598895    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx\" (UID: \"c5c694d7-f5c9-4288-93ef-ea5a4011e87c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"
Nov 03 02:16:39.599158 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.599140    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcp6c\" (UniqueName: \"kubernetes.io/projected/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-kube-api-access-bcp6c\") pod \"payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx\" (UID: \"c5c694d7-f5c9-4288-93ef-ea5a4011e87c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"
Nov 03 02:16:39.600508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.599928    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx\" (UID: \"c5c694d7-f5c9-4288-93ef-ea5a4011e87c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"
Nov 03 02:16:39.627396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.627365    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"]
Nov 03 02:16:39.657883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.657809    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcp6c\" (UniqueName: \"kubernetes.io/projected/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-kube-api-access-bcp6c\") pod \"payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx\" (UID: \"c5c694d7-f5c9-4288-93ef-ea5a4011e87c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"
Nov 03 02:16:39.707340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.701602    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dxzp\" (UniqueName: \"kubernetes.io/projected/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-kube-api-access-9dxzp\") pod \"payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd\" (UID: \"15c2316d-4bf4-47f1-8804-cb1d17ef28ba\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"
Nov 03 02:16:39.707340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.701900    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd\" (UID: \"15c2316d-4bf4-47f1-8804-cb1d17ef28ba\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"
Nov 03 02:16:39.736778 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.736748    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"
Nov 03 02:16:39.807299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.803494    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd\" (UID: \"15c2316d-4bf4-47f1-8804-cb1d17ef28ba\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"
Nov 03 02:16:39.807299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.803597    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dxzp\" (UniqueName: \"kubernetes.io/projected/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-kube-api-access-9dxzp\") pod \"payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd\" (UID: \"15c2316d-4bf4-47f1-8804-cb1d17ef28ba\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"
Nov 03 02:16:39.807299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.805527    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd\" (UID: \"15c2316d-4bf4-47f1-8804-cb1d17ef28ba\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"
Nov 03 02:16:39.872378 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.871766    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dxzp\" (UniqueName: \"kubernetes.io/projected/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-kube-api-access-9dxzp\") pod \"payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd\" (UID: \"15c2316d-4bf4-47f1-8804-cb1d17ef28ba\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"
Nov 03 02:16:39.907664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:39.907634    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"
Nov 03 02:16:40.123687 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:40.123640    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"]
Nov 03 02:16:40.128228 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:16:40.128193    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c694d7_f5c9_4288_93ef_ea5a4011e87c.slice/crio-b690fa6bc5ed45d0a5b631fcf41552ed1fafdb277cfb79298cddb1c7edbb02d4 WatchSource:0}: Error finding container b690fa6bc5ed45d0a5b631fcf41552ed1fafdb277cfb79298cddb1c7edbb02d4: Status 404 returned error can't find the container with id b690fa6bc5ed45d0a5b631fcf41552ed1fafdb277cfb79298cddb1c7edbb02d4
Nov 03 02:16:40.128389 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:40.128368    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"]
Nov 03 02:16:40.221274 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:40.221235    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"]
Nov 03 02:16:40.297968 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:40.297881    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"]
Nov 03 02:16:40.302842 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:16:40.302815    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c2316d_4bf4_47f1_8804_cb1d17ef28ba.slice/crio-089e080aaf9ad11519ea1b6abfbe696959265ecdc50f801c937ea14c8af75277 WatchSource:0}: Error finding container 089e080aaf9ad11519ea1b6abfbe696959265ecdc50f801c937ea14c8af75277: Status 404 returned error can't find the container with id 089e080aaf9ad11519ea1b6abfbe696959265ecdc50f801c937ea14c8af75277
Nov 03 02:16:40.612790 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:40.612662    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:16:40.744228 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:40.743847    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd" event={"ID":"15c2316d-4bf4-47f1-8804-cb1d17ef28ba","Type":"ContainerStarted","Data":"260846b983afef3b8d909b1ccd0baa21f3ce8a87eb2e31f8ff1088e26b02ed58"}
Nov 03 02:16:40.744228 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:40.743910    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd" event={"ID":"15c2316d-4bf4-47f1-8804-cb1d17ef28ba","Type":"ContainerStarted","Data":"089e080aaf9ad11519ea1b6abfbe696959265ecdc50f801c937ea14c8af75277"}
Nov 03 02:16:40.744228 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:40.743989    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"
Nov 03 02:16:40.745910 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:40.745880    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx" event={"ID":"c5c694d7-f5c9-4288-93ef-ea5a4011e87c","Type":"ContainerStarted","Data":"991b91962f8b3de19919820f40e718c464886825c2fee64e66845f36f8e2373d"}
Nov 03 02:16:40.746058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:40.745937    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"
Nov 03 02:16:40.746058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:40.745961    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx" event={"ID":"c5c694d7-f5c9-4288-93ef-ea5a4011e87c","Type":"ContainerStarted","Data":"b690fa6bc5ed45d0a5b631fcf41552ed1fafdb277cfb79298cddb1c7edbb02d4"}
Nov 03 02:16:40.854084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:40.854033    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd" podStartSLOduration=1.854020507 podStartE2EDuration="1.854020507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:40.852498432 +0000 UTC m=+3480.278104916" watchObservedRunningTime="2026-11-03 02:16:40.854020507 +0000 UTC m=+3480.279626988"
Nov 03 02:16:40.963118 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:40.963062    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx" podStartSLOduration=1.96304633 podStartE2EDuration="1.96304633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:16:40.959897955 +0000 UTC m=+3480.385504441" watchObservedRunningTime="2026-11-03 02:16:40.96304633 +0000 UTC m=+3480.388652794"
Nov 03 02:16:43.813213 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:16:43.813179    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:16:45.128931 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.128888    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq" podUID="d9e4a089-412b-4ed3-8dbf-abc2ab20d453" containerName="payload-processing" containerID="cri-o://318811fb345cd4827446b1c72e3c23885a953444360c4e45c7718fc10ecb0518" gracePeriod=25
Nov 03 02:16:45.221820 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.221770    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79" podUID="7513eb12-2a38-43d0-b4cc-06a78e5ca89f" containerName="payload-pre-processing" containerID="cri-o://56635ffd271e541296f01367d0ddcd443300639b3334c463343be5ae17f60b16" gracePeriod=25
Nov 03 02:16:45.343191 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:16:45.343161    5508 logging.go:55] [core] [Channel #8037 SubChannel #8040]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.30:50052", ServerName: "10.133.1.30:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.30:50052: connect: connection refused"
Nov 03 02:16:45.343191 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:16:45.343161    5508 logging.go:55] [core] [Channel #8038 SubChannel #8039]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.29:50052", ServerName: "10.133.1.29:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.29:50052: connect: connection refused"
Nov 03 02:16:45.532756 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.532731    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"
Nov 03 02:16:45.605785 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.605758    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"
Nov 03 02:16:45.629392 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.629342    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-plugins-config-volume\") pod \"7513eb12-2a38-43d0-b4cc-06a78e5ca89f\" (UID: \"7513eb12-2a38-43d0-b4cc-06a78e5ca89f\") "
Nov 03 02:16:45.629546 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.629442    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-plugins-config-volume\") pod \"d9e4a089-412b-4ed3-8dbf-abc2ab20d453\" (UID: \"d9e4a089-412b-4ed3-8dbf-abc2ab20d453\") "
Nov 03 02:16:45.629546 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.629490    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-kube-api-access-6j4hd\" (UniqueName: \"kubernetes.io/projected/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-kube-api-access-6j4hd\") pod \"7513eb12-2a38-43d0-b4cc-06a78e5ca89f\" (UID: \"7513eb12-2a38-43d0-b4cc-06a78e5ca89f\") "
Nov 03 02:16:45.629546 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.629529    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-kube-api-access-92tcs\" (UniqueName: \"kubernetes.io/projected/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-kube-api-access-92tcs\") pod \"d9e4a089-412b-4ed3-8dbf-abc2ab20d453\" (UID: \"d9e4a089-412b-4ed3-8dbf-abc2ab20d453\") "
Nov 03 02:16:45.629847 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.629819    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-plugins-config-volume" pod "7513eb12-2a38-43d0-b4cc-06a78e5ca89f" (UID: "7513eb12-2a38-43d0-b4cc-06a78e5ca89f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:16:45.629889 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.629857    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-plugins-config-volume" pod "d9e4a089-412b-4ed3-8dbf-abc2ab20d453" (UID: "d9e4a089-412b-4ed3-8dbf-abc2ab20d453"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:16:45.630113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.630096    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:45.630179 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.630116    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:45.631769 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.631740    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-kube-api-access-92tcs" pod "d9e4a089-412b-4ed3-8dbf-abc2ab20d453" (UID: "d9e4a089-412b-4ed3-8dbf-abc2ab20d453"). InnerVolumeSpecName "kube-api-access-92tcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:45.631877 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.631766    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-kube-api-access-6j4hd" pod "7513eb12-2a38-43d0-b4cc-06a78e5ca89f" (UID: "7513eb12-2a38-43d0-b4cc-06a78e5ca89f"). InnerVolumeSpecName "kube-api-access-6j4hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:45.731370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.731267    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92tcs\" (UniqueName: \"kubernetes.io/projected/d9e4a089-412b-4ed3-8dbf-abc2ab20d453-kube-api-access-92tcs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:45.731370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.731302    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j4hd\" (UniqueName: \"kubernetes.io/projected/7513eb12-2a38-43d0-b4cc-06a78e5ca89f-kube-api-access-6j4hd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:45.736973 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.736947    5508 scope.go:122] "RemoveContainer" containerID="bbc6fab9aa7864098f1ab51342e57bd561ff47b44500ac1554077b40032078ca"
Nov 03 02:16:45.886514 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.886475    5508 generic.go:356] "Generic (PLEG): container finished" podID="d9e4a089-412b-4ed3-8dbf-abc2ab20d453" containerID="318811fb345cd4827446b1c72e3c23885a953444360c4e45c7718fc10ecb0518" exitCode=0
Nov 03 02:16:45.886715 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.886544    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"
Nov 03 02:16:45.886715 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.886550    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq" event={"ID":"d9e4a089-412b-4ed3-8dbf-abc2ab20d453","Type":"ContainerDied","Data":"318811fb345cd4827446b1c72e3c23885a953444360c4e45c7718fc10ecb0518"}
Nov 03 02:16:45.886715 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.886613    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq" event={"ID":"d9e4a089-412b-4ed3-8dbf-abc2ab20d453","Type":"ContainerDied","Data":"3fe19b68cae4a2c5ee547f551ba7ac44305154d8eee5d023123f8403671bd5be"}
Nov 03 02:16:45.886715 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.886640    5508 scope.go:122] "RemoveContainer" containerID="318811fb345cd4827446b1c72e3c23885a953444360c4e45c7718fc10ecb0518"
Nov 03 02:16:45.888556 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.888528    5508 generic.go:356] "Generic (PLEG): container finished" podID="7513eb12-2a38-43d0-b4cc-06a78e5ca89f" containerID="56635ffd271e541296f01367d0ddcd443300639b3334c463343be5ae17f60b16" exitCode=0
Nov 03 02:16:45.888740 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.888721    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79" event={"ID":"7513eb12-2a38-43d0-b4cc-06a78e5ca89f","Type":"ContainerDied","Data":"56635ffd271e541296f01367d0ddcd443300639b3334c463343be5ae17f60b16"}
Nov 03 02:16:45.888817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.888768    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79" event={"ID":"7513eb12-2a38-43d0-b4cc-06a78e5ca89f","Type":"ContainerDied","Data":"448f865a335f35667e3bdd7728cc17ff8d48575bbfbed8658bcdb757d363042b"}
Nov 03 02:16:45.888885 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.888839    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"
Nov 03 02:16:45.898025 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.897993    5508 scope.go:122] "RemoveContainer" containerID="318811fb345cd4827446b1c72e3c23885a953444360c4e45c7718fc10ecb0518"
Nov 03 02:16:45.898501 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:16:45.898341    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318811fb345cd4827446b1c72e3c23885a953444360c4e45c7718fc10ecb0518\": container with ID starting with 318811fb345cd4827446b1c72e3c23885a953444360c4e45c7718fc10ecb0518 not found: ID does not exist" containerID="318811fb345cd4827446b1c72e3c23885a953444360c4e45c7718fc10ecb0518"
Nov 03 02:16:45.898501 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.898410    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318811fb345cd4827446b1c72e3c23885a953444360c4e45c7718fc10ecb0518"} err="failed to get container status \"318811fb345cd4827446b1c72e3c23885a953444360c4e45c7718fc10ecb0518\": rpc error: code = NotFound desc = could not find container \"318811fb345cd4827446b1c72e3c23885a953444360c4e45c7718fc10ecb0518\": container with ID starting with 318811fb345cd4827446b1c72e3c23885a953444360c4e45c7718fc10ecb0518 not found: ID does not exist"
Nov 03 02:16:45.898501 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.898431    5508 scope.go:122] "RemoveContainer" containerID="56635ffd271e541296f01367d0ddcd443300639b3334c463343be5ae17f60b16"
Nov 03 02:16:45.908122 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.908106    5508 scope.go:122] "RemoveContainer" containerID="56635ffd271e541296f01367d0ddcd443300639b3334c463343be5ae17f60b16"
Nov 03 02:16:45.908464 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:16:45.908442    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56635ffd271e541296f01367d0ddcd443300639b3334c463343be5ae17f60b16\": container with ID starting with 56635ffd271e541296f01367d0ddcd443300639b3334c463343be5ae17f60b16 not found: ID does not exist" containerID="56635ffd271e541296f01367d0ddcd443300639b3334c463343be5ae17f60b16"
Nov 03 02:16:45.908526 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:45.908471    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56635ffd271e541296f01367d0ddcd443300639b3334c463343be5ae17f60b16"} err="failed to get container status \"56635ffd271e541296f01367d0ddcd443300639b3334c463343be5ae17f60b16\": rpc error: code = NotFound desc = could not find container \"56635ffd271e541296f01367d0ddcd443300639b3334c463343be5ae17f60b16\": container with ID starting with 56635ffd271e541296f01367d0ddcd443300639b3334c463343be5ae17f60b16 not found: ID does not exist"
Nov 03 02:16:46.041328 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:46.041235    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"]
Nov 03 02:16:46.045468 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:46.045442    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq"]
Nov 03 02:16:46.147329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:46.147299    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"]
Nov 03 02:16:46.177999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:46.177956    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79"]
Nov 03 02:16:46.343192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:46.343097    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b16721-5dbccbb547-mnr79" podUID="7513eb12-2a38-43d0-b4cc-06a78e5ca89f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.29:50052\" within 1s: context deadline exceeded"
Nov 03 02:16:46.343372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:46.343097    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-explicit-b16721-57998cd4b7-xvtrq" podUID="d9e4a089-412b-4ed3-8dbf-abc2ab20d453" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.30:50052\" within 1s: context deadline exceeded"
Nov 03 02:16:46.692627 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:46.692596    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7513eb12-2a38-43d0-b4cc-06a78e5ca89f" path="/var/lib/kubelet/pods/7513eb12-2a38-43d0-b4cc-06a78e5ca89f/volumes"
Nov 03 02:16:46.693049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:46.693035    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e4a089-412b-4ed3-8dbf-abc2ab20d453" path="/var/lib/kubelet/pods/d9e4a089-412b-4ed3-8dbf-abc2ab20d453/volumes"
Nov 03 02:16:51.829155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:51.829128    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"
Nov 03 02:16:51.829669 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:51.829192    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"
Nov 03 02:16:51.934854 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:51.934816    5508 generic.go:356] "Generic (PLEG): container finished" podID="320b0833-f978-4d66-9d40-8693e461f63b" containerID="f3193bca4642683df7d464ac01275425c055ecece98424cb47e1c1818df965b1" exitCode=0
Nov 03 02:16:51.935047 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:51.934885    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" event={"ID":"320b0833-f978-4d66-9d40-8693e461f63b","Type":"ContainerDied","Data":"f3193bca4642683df7d464ac01275425c055ecece98424cb47e1c1818df965b1"}
Nov 03 02:16:51.935304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:51.935289    5508 scope.go:122] "RemoveContainer" containerID="f3193bca4642683df7d464ac01275425c055ecece98424cb47e1c1818df965b1"
Nov 03 02:16:51.936856 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:51.936828    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:16:51.936967 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:51.936894    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:16:51.936967 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:51.936925    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:16:52.944123 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:52.944080    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" event={"ID":"320b0833-f978-4d66-9d40-8693e461f63b","Type":"ContainerStarted","Data":"6e6fcba7e2380576729c5fad2e80d64337cbd1dc0a0f316e4d157f5cd0fb3328"}
Nov 03 02:16:53.688205 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:53.688166    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:16:53.688419 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:53.688342    5508 patch_prober.go:29] interesting pod/e2e-extmod-a-c70365-openshift-default-5674757df-ng472 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 03 02:16:53.688419 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:53.688406    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" podUID="320b0833-f978-4d66-9d40-8693e461f63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 03 02:16:54.688773 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:54.688739    5508 patch_prober.go:29] interesting pod/e2e-extmod-a-c70365-openshift-default-5674757df-ng472 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 03 02:16:54.689173 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:54.688796    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" podUID="320b0833-f978-4d66-9d40-8693e461f63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 03 02:16:55.688541 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:55.688499    5508 patch_prober.go:29] interesting pod/e2e-extmod-a-c70365-openshift-default-5674757df-ng472 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 03 02:16:55.688763 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:55.688558    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" podUID="320b0833-f978-4d66-9d40-8693e461f63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 03 02:16:56.688737 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:56.688703    5508 patch_prober.go:29] interesting pod/e2e-extmod-a-c70365-openshift-default-5674757df-ng472 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 03 02:16:56.689364 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:56.689321    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" podUID="320b0833-f978-4d66-9d40-8693e461f63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 03 02:16:57.688811 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:57.688778    5508 patch_prober.go:29] interesting pod/e2e-extmod-a-c70365-openshift-default-5674757df-ng472 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body=
Nov 03 02:16:57.689239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:57.688831    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" podUID="320b0833-f978-4d66-9d40-8693e461f63b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused"
Nov 03 02:16:58.419872 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.419836    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"]
Nov 03 02:16:58.420073 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.420053    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5" podUID="5db97d4e-dbd2-4135-ad39-f4f992b87f07" containerName="maas-api" containerID="cri-o://34355e37bcb1098854c98654647cd3978545065f4c10142c61d11d96bb8794c0" gracePeriod=30
Nov 03 02:16:58.700880 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.700853    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:16:58.701268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.701246    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:16:58.702233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.702210    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:16:58.717516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.717490    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:16:58.829173 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.829130    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-metrics-certs\") pod \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") "
Nov 03 02:16:58.829438 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.829274    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db97d4e-dbd2-4135-ad39-f4f992b87f07-kube-api-access-nlprw\" (UniqueName: \"kubernetes.io/projected/5db97d4e-dbd2-4135-ad39-f4f992b87f07-kube-api-access-nlprw\") pod \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") "
Nov 03 02:16:58.829438 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.829363    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db97d4e-dbd2-4135-ad39-f4f992b87f07-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db97d4e-dbd2-4135-ad39-f4f992b87f07-openshift-service-ca\") pod \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") "
Nov 03 02:16:58.829438 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.829416    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-maas-api-tls\") pod \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\" (UID: \"5db97d4e-dbd2-4135-ad39-f4f992b87f07\") "
Nov 03 02:16:58.830243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.830207    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db97d4e-dbd2-4135-ad39-f4f992b87f07-openshift-service-ca" pod "5db97d4e-dbd2-4135-ad39-f4f992b87f07" (UID: "5db97d4e-dbd2-4135-ad39-f4f992b87f07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:16:58.832514 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.832472    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-metrics-certs" pod "5db97d4e-dbd2-4135-ad39-f4f992b87f07" (UID: "5db97d4e-dbd2-4135-ad39-f4f992b87f07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:16:58.832692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.832666    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db97d4e-dbd2-4135-ad39-f4f992b87f07-kube-api-access-nlprw" pod "5db97d4e-dbd2-4135-ad39-f4f992b87f07" (UID: "5db97d4e-dbd2-4135-ad39-f4f992b87f07"). InnerVolumeSpecName "kube-api-access-nlprw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:16:58.832839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.832817    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-maas-api-tls" pod "5db97d4e-dbd2-4135-ad39-f4f992b87f07" (UID: "5db97d4e-dbd2-4135-ad39-f4f992b87f07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:16:58.931450 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.931416    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db97d4e-dbd2-4135-ad39-f4f992b87f07-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:58.931450 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.931451    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:58.931674 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.931461    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db97d4e-dbd2-4135-ad39-f4f992b87f07-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:58.931674 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:58.931471    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlprw\" (UniqueName: \"kubernetes.io/projected/5db97d4e-dbd2-4135-ad39-f4f992b87f07-kube-api-access-nlprw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:16:59.005691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:59.005586    5508 generic.go:356] "Generic (PLEG): container finished" podID="5db97d4e-dbd2-4135-ad39-f4f992b87f07" containerID="34355e37bcb1098854c98654647cd3978545065f4c10142c61d11d96bb8794c0" exitCode=0
Nov 03 02:16:59.005691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:59.005636    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"
Nov 03 02:16:59.005691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:59.005672    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5" event={"ID":"5db97d4e-dbd2-4135-ad39-f4f992b87f07","Type":"ContainerDied","Data":"34355e37bcb1098854c98654647cd3978545065f4c10142c61d11d96bb8794c0"}
Nov 03 02:16:59.005954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:59.005731    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5" event={"ID":"5db97d4e-dbd2-4135-ad39-f4f992b87f07","Type":"ContainerDied","Data":"696ccc16a3e04f1ac5a46b657904dc1b056a4012a7e090a8511a165248578931"}
Nov 03 02:16:59.005954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:59.005758    5508 scope.go:122] "RemoveContainer" containerID="34355e37bcb1098854c98654647cd3978545065f4c10142c61d11d96bb8794c0"
Nov 03 02:16:59.018231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:59.018215    5508 scope.go:122] "RemoveContainer" containerID="34355e37bcb1098854c98654647cd3978545065f4c10142c61d11d96bb8794c0"
Nov 03 02:16:59.018546 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:16:59.018526    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34355e37bcb1098854c98654647cd3978545065f4c10142c61d11d96bb8794c0\": container with ID starting with 34355e37bcb1098854c98654647cd3978545065f4c10142c61d11d96bb8794c0 not found: ID does not exist" containerID="34355e37bcb1098854c98654647cd3978545065f4c10142c61d11d96bb8794c0"
Nov 03 02:16:59.018640 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:59.018552    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34355e37bcb1098854c98654647cd3978545065f4c10142c61d11d96bb8794c0"} err="failed to get container status \"34355e37bcb1098854c98654647cd3978545065f4c10142c61d11d96bb8794c0\": rpc error: code = NotFound desc = could not find container \"34355e37bcb1098854c98654647cd3978545065f4c10142c61d11d96bb8794c0\": container with ID starting with 34355e37bcb1098854c98654647cd3978545065f4c10142c61d11d96bb8794c0 not found: ID does not exist"
Nov 03 02:16:59.144693 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:59.144656    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"]
Nov 03 02:16:59.157221 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:16:59.157184    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b16721-77f7c668cb-ghmc5"]
Nov 03 02:17:00.692737 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:00.692705    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db97d4e-dbd2-4135-ad39-f4f992b87f07" path="/var/lib/kubelet/pods/5db97d4e-dbd2-4135-ad39-f4f992b87f07/volumes"
Nov 03 02:17:02.228412 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:02.228375    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"]
Nov 03 02:17:02.228829 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:02.228583    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerName="istio-proxy" containerID="cri-o://9d95e9e693b489343c4191ff85897638bd2a0ce1a461adebb4ffd991672d895d" gracePeriod=30
Nov 03 02:17:03.456184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.456146    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"]
Nov 03 02:17:03.457056 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.457035    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5db97d4e-dbd2-4135-ad39-f4f992b87f07" containerName="maas-api"
Nov 03 02:17:03.457113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.457059    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db97d4e-dbd2-4135-ad39-f4f992b87f07" containerName="maas-api"
Nov 03 02:17:03.457113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.457096    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7513eb12-2a38-43d0-b4cc-06a78e5ca89f" containerName="payload-pre-processing"
Nov 03 02:17:03.457113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.457104    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7513eb12-2a38-43d0-b4cc-06a78e5ca89f" containerName="payload-pre-processing"
Nov 03 02:17:03.457213 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.457130    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d9e4a089-412b-4ed3-8dbf-abc2ab20d453" containerName="payload-processing"
Nov 03 02:17:03.457213 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.457137    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e4a089-412b-4ed3-8dbf-abc2ab20d453" containerName="payload-processing"
Nov 03 02:17:03.457308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.457301    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db97d4e-dbd2-4135-ad39-f4f992b87f07" containerName="maas-api"
Nov 03 02:17:03.457372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.457313    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7513eb12-2a38-43d0-b4cc-06a78e5ca89f" containerName="payload-pre-processing"
Nov 03 02:17:03.457372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.457328    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e4a089-412b-4ed3-8dbf-abc2ab20d453" containerName="payload-processing"
Nov 03 02:17:03.462597 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.462575    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.511437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.511408    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-898926-openshift-default-dockercfg-ccwb4\""
Nov 03 02:17:03.560978 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.560946    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"]
Nov 03 02:17:03.598704 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.598666    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-token\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.598906 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.598737    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-credential-socket\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.598906 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.598777    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rczlc\" (UniqueName: \"kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-kube-api-access-rczlc\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.598906 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.598851    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-envoy\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.598906 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.598877    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-data\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.599144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.598926    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-socket\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.599144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.599031    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-podinfo\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.599144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.599087    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istiod-ca-cert\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.599144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.599128    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-certs\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.701212 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.701172    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczlc\" (UniqueName: \"kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-kube-api-access-rczlc\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.701480 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.701295    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-envoy\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.701617 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.701590    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-data\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.701677 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.701660    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-socket\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.701782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.701768    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-podinfo\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.701853 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.701826    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istiod-ca-cert\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.701907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.701870    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-certs\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.701960 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.701944    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-token\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.702005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.701994    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-credential-socket\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.702445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.702422    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-credential-socket\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.703134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.702708    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-data\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.703134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.702913    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-certs\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.703134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.703068    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istiod-ca-cert\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.703134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.703081    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-socket\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.704810 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.704780    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-envoy\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.705544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.705518    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-podinfo\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.759199 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.759122    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rczlc\" (UniqueName: \"kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-kube-api-access-rczlc\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.760007 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.759982    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-token\") pod \"e2e-noten-898926-openshift-default-575b479d9-7p5fn\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") " pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:03.779246 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:03.779219    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:04.264082 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:04.264045    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"]
Nov 03 02:17:04.268511 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:17:04.268483    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ef25b1_3348_465e_b5f6_ea831b7b8717.slice/crio-89b78a9e6a953d2810c53dbf5e759a6e5e443c02cbddcc9fddfd733863d46605 WatchSource:0}: Error finding container 89b78a9e6a953d2810c53dbf5e759a6e5e443c02cbddcc9fddfd733863d46605: Status 404 returned error can't find the container with id 89b78a9e6a953d2810c53dbf5e759a6e5e443c02cbddcc9fddfd733863d46605
Nov 03 02:17:04.271147 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:04.271114    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:17:04.271228 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:04.271185    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:17:04.271228 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:04.271215    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:17:05.050653 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:05.050621    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn" event={"ID":"a5ef25b1-3348-465e-b5f6-ea831b7b8717","Type":"ContainerStarted","Data":"1c2abadb172316b9ede5cb82b7ed8ebfd7c04c23153a681b2decebf0c26eb062"}
Nov 03 02:17:05.051062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:05.050674    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn" event={"ID":"a5ef25b1-3348-465e-b5f6-ea831b7b8717","Type":"ContainerStarted","Data":"89b78a9e6a953d2810c53dbf5e759a6e5e443c02cbddcc9fddfd733863d46605"}
Nov 03 02:17:05.165245 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:05.165185    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn" podStartSLOduration=2.165171247 podStartE2EDuration="2.165171247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:17:05.16384691 +0000 UTC m=+3504.589453395" watchObservedRunningTime="2026-11-03 02:17:05.165171247 +0000 UTC m=+3504.590777765"
Nov 03 02:17:05.779497 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:05.779462    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:05.785234 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:05.785200    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:06.057889 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:06.057817    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:06.058668 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:06.058650    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:17:07.490389 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.490298    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:17:07.551802 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.551770    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-certs\") pod \"e105e299-ee4c-4664-ab9c-39e42f762b8d\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") "
Nov 03 02:17:07.551802 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.551802    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-socket\") pod \"e105e299-ee4c-4664-ab9c-39e42f762b8d\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") "
Nov 03 02:17:07.552058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.551838    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e105e299-ee4c-4664-ab9c-39e42f762b8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e105e299-ee4c-4664-ab9c-39e42f762b8d-istiod-ca-cert\") pod \"e105e299-ee4c-4664-ab9c-39e42f762b8d\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") "
Nov 03 02:17:07.552058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.551865    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-kube-api-access-gcbd9\" (UniqueName: \"kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-kube-api-access-gcbd9\") pod \"e105e299-ee4c-4664-ab9c-39e42f762b8d\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") "
Nov 03 02:17:07.552058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.551899    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-token\" (UniqueName: \"kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-token\") pod \"e105e299-ee4c-4664-ab9c-39e42f762b8d\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") "
Nov 03 02:17:07.552058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552005    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-data\") pod \"e105e299-ee4c-4664-ab9c-39e42f762b8d\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") "
Nov 03 02:17:07.552271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552060    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-podinfo\") pod \"e105e299-ee4c-4664-ab9c-39e42f762b8d\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") "
Nov 03 02:17:07.552271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552080    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-certs" pod "e105e299-ee4c-4664-ab9c-39e42f762b8d" (UID: "e105e299-ee4c-4664-ab9c-39e42f762b8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:17:07.552271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552118    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-envoy\") pod \"e105e299-ee4c-4664-ab9c-39e42f762b8d\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") "
Nov 03 02:17:07.552271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552194    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-credential-socket\") pod \"e105e299-ee4c-4664-ab9c-39e42f762b8d\" (UID: \"e105e299-ee4c-4664-ab9c-39e42f762b8d\") "
Nov 03 02:17:07.552271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552196    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-socket" pod "e105e299-ee4c-4664-ab9c-39e42f762b8d" (UID: "e105e299-ee4c-4664-ab9c-39e42f762b8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:17:07.552585 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552316    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e105e299-ee4c-4664-ab9c-39e42f762b8d-istiod-ca-cert" pod "e105e299-ee4c-4664-ab9c-39e42f762b8d" (UID: "e105e299-ee4c-4664-ab9c-39e42f762b8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:17:07.552585 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552318    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-data" pod "e105e299-ee4c-4664-ab9c-39e42f762b8d" (UID: "e105e299-ee4c-4664-ab9c-39e42f762b8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:17:07.552712 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552592    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-credential-socket" pod "e105e299-ee4c-4664-ab9c-39e42f762b8d" (UID: "e105e299-ee4c-4664-ab9c-39e42f762b8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:17:07.552823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552807    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:07.552897 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552827    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:07.552897 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552841    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:07.552897 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552854    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e105e299-ee4c-4664-ab9c-39e42f762b8d-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:07.552897 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.552862    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:07.554489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.554464    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-kube-api-access-gcbd9" pod "e105e299-ee4c-4664-ab9c-39e42f762b8d" (UID: "e105e299-ee4c-4664-ab9c-39e42f762b8d"). InnerVolumeSpecName "kube-api-access-gcbd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:17:07.554489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.554479    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-envoy" pod "e105e299-ee4c-4664-ab9c-39e42f762b8d" (UID: "e105e299-ee4c-4664-ab9c-39e42f762b8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:17:07.554725 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.554708    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-token" pod "e105e299-ee4c-4664-ab9c-39e42f762b8d" (UID: "e105e299-ee4c-4664-ab9c-39e42f762b8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:17:07.554765 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.554722    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-podinfo" pod "e105e299-ee4c-4664-ab9c-39e42f762b8d" (UID: "e105e299-ee4c-4664-ab9c-39e42f762b8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:17:07.654587 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.654546    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:07.654587 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.654583    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:07.654856 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.654597    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcbd9\" (UniqueName: \"kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-kube-api-access-gcbd9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:07.654856 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:07.654614    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e105e299-ee4c-4664-ab9c-39e42f762b8d-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:17:08.074142 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:08.074107    5508 generic.go:356] "Generic (PLEG): container finished" podID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerID="9d95e9e693b489343c4191ff85897638bd2a0ce1a461adebb4ffd991672d895d" exitCode=0
Nov 03 02:17:08.074335 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:08.074162    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"
Nov 03 02:17:08.074335 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:08.074175    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" event={"ID":"e105e299-ee4c-4664-ab9c-39e42f762b8d","Type":"ContainerDied","Data":"9d95e9e693b489343c4191ff85897638bd2a0ce1a461adebb4ffd991672d895d"}
Nov 03 02:17:08.074335 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:08.074234    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp" event={"ID":"e105e299-ee4c-4664-ab9c-39e42f762b8d","Type":"ContainerDied","Data":"dc4730e65ae4bb0eabb78b46377464b8a9d0d1188c3e9f0ce269776678708be7"}
Nov 03 02:17:08.074335 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:08.074264    5508 scope.go:122] "RemoveContainer" containerID="9d95e9e693b489343c4191ff85897638bd2a0ce1a461adebb4ffd991672d895d"
Nov 03 02:17:08.086061 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:08.086042    5508 scope.go:122] "RemoveContainer" containerID="0fe2152ddfaebebb287fabc443623ab2f23ed7ec3b04d0feb7b94c22dbbb3c04"
Nov 03 02:17:08.095064 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:08.095046    5508 scope.go:122] "RemoveContainer" containerID="9d95e9e693b489343c4191ff85897638bd2a0ce1a461adebb4ffd991672d895d"
Nov 03 02:17:08.095310 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:17:08.095290    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d95e9e693b489343c4191ff85897638bd2a0ce1a461adebb4ffd991672d895d\": container with ID starting with 9d95e9e693b489343c4191ff85897638bd2a0ce1a461adebb4ffd991672d895d not found: ID does not exist" containerID="9d95e9e693b489343c4191ff85897638bd2a0ce1a461adebb4ffd991672d895d"
Nov 03 02:17:08.095390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:08.095319    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d95e9e693b489343c4191ff85897638bd2a0ce1a461adebb4ffd991672d895d"} err="failed to get container status \"9d95e9e693b489343c4191ff85897638bd2a0ce1a461adebb4ffd991672d895d\": rpc error: code = NotFound desc = could not find container \"9d95e9e693b489343c4191ff85897638bd2a0ce1a461adebb4ffd991672d895d\": container with ID starting with 9d95e9e693b489343c4191ff85897638bd2a0ce1a461adebb4ffd991672d895d not found: ID does not exist"
Nov 03 02:17:08.095390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:08.095337    5508 scope.go:122] "RemoveContainer" containerID="0fe2152ddfaebebb287fabc443623ab2f23ed7ec3b04d0feb7b94c22dbbb3c04"
Nov 03 02:17:08.095581 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:17:08.095566    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe2152ddfaebebb287fabc443623ab2f23ed7ec3b04d0feb7b94c22dbbb3c04\": container with ID starting with 0fe2152ddfaebebb287fabc443623ab2f23ed7ec3b04d0feb7b94c22dbbb3c04 not found: ID does not exist" containerID="0fe2152ddfaebebb287fabc443623ab2f23ed7ec3b04d0feb7b94c22dbbb3c04"
Nov 03 02:17:08.095643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:08.095585    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe2152ddfaebebb287fabc443623ab2f23ed7ec3b04d0feb7b94c22dbbb3c04"} err="failed to get container status \"0fe2152ddfaebebb287fabc443623ab2f23ed7ec3b04d0feb7b94c22dbbb3c04\": rpc error: code = NotFound desc = could not find container \"0fe2152ddfaebebb287fabc443623ab2f23ed7ec3b04d0feb7b94c22dbbb3c04\": container with ID starting with 0fe2152ddfaebebb287fabc443623ab2f23ed7ec3b04d0feb7b94c22dbbb3c04 not found: ID does not exist"
Nov 03 02:17:08.229443 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:08.229404    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"]
Nov 03 02:17:08.233739 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:08.233704    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-b16721-openshift-default-74984b654f-xqftp"]
Nov 03 02:17:08.692803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:08.692774    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" path="/var/lib/kubelet/pods/e105e299-ee4c-4664-ab9c-39e42f762b8d/volumes"
Nov 03 02:17:13.812272 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:17:13.812235    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:17:22.446443 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:22.446407    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:17:23.770445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.770405    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"]
Nov 03 02:17:23.771126 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.771108    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerName="istio-proxy"
Nov 03 02:17:23.771191 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.771127    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerName="istio-proxy"
Nov 03 02:17:23.771257 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.771246    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerName="istio-proxy"
Nov 03 02:17:23.774877 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.774858    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.823722 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.823678    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4f87d1-openshift-default-dockercfg-mk9lx\""
Nov 03 02:17:23.856552 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.856509    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-envoy\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.856742 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.856561    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518ce897-7786-44cc-9be3-737bb47e8e50-istiod-ca-cert\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.856742 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.856722    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-certs\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.856862 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.856831    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518ce897-7786-44cc-9be3-737bb47e8e50-istio-podinfo\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.856925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.856884    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb2z8\" (UniqueName: \"kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-kube-api-access-qb2z8\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.856991 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.856922    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.857054 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.857043    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-credential-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.857145 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.857130    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-data\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.857245 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.857225    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-istio-token\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.879065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.879034    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"]
Nov 03 02:17:23.958610 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.958573    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-istio-token\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.958823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.958644    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-envoy\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.958823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.958676    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518ce897-7786-44cc-9be3-737bb47e8e50-istiod-ca-cert\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.958823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.958707    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-certs\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.958823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.958747    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518ce897-7786-44cc-9be3-737bb47e8e50-istio-podinfo\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.958823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.958772    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb2z8\" (UniqueName: \"kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-kube-api-access-qb2z8\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.958823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.958792    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.959169 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.958835    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-credential-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.959169 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.958886    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-data\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.959598 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.959565    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518ce897-7786-44cc-9be3-737bb47e8e50-istiod-ca-cert\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.959801 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.959641    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.959899 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.959687    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-credential-socket\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.959899 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.959718    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-data\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.959899 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.959890    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-certs\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.961526 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.961505    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518ce897-7786-44cc-9be3-737bb47e8e50-istio-podinfo\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:23.961828 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:23.961809    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-envoy\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:24.016119 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.016080    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-istio-token\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:24.016945 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.016920    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb2z8\" (UniqueName: \"kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-kube-api-access-qb2z8\") pod \"e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") " pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:24.092415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.092305    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:24.381726 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.381622    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"]
Nov 03 02:17:24.382417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.382393    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerName="istio-proxy"
Nov 03 02:17:24.382417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.382414    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerName="istio-proxy"
Nov 03 02:17:24.382581 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.382553    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e105e299-ee4c-4664-ab9c-39e42f762b8d" containerName="istio-proxy"
Nov 03 02:17:24.387221 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.387199    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:24.436361 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.436314    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-aab75e-serving-cert\""
Nov 03 02:17:24.444136 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.444102    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"]
Nov 03 02:17:24.466617 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.466573    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e5189f-18d7-405d-a067-ded6fa319c64-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:24.466824 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.466800    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-maas-api-tls\") pod \"maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:24.466916 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.466846    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-metrics-certs\") pod \"maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:24.467012 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.466992    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx4vt\" (UniqueName: \"kubernetes.io/projected/82e5189f-18d7-405d-a067-ded6fa319c64-kube-api-access-kx4vt\") pod \"maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:24.529522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.529483    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"]
Nov 03 02:17:24.545446 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:17:24.545412    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518ce897_7786_44cc_9be3_737bb47e8e50.slice/crio-46157cbf9ff5482a6c903197babef67674fc6b40b5ea5f52c41c18fc0b2e66f7 WatchSource:0}: Error finding container 46157cbf9ff5482a6c903197babef67674fc6b40b5ea5f52c41c18fc0b2e66f7: Status 404 returned error can't find the container with id 46157cbf9ff5482a6c903197babef67674fc6b40b5ea5f52c41c18fc0b2e66f7
Nov 03 02:17:24.548513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.548479    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:17:24.548611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.548549    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:17:24.548611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.548579    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:17:24.568531 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.568503    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e5189f-18d7-405d-a067-ded6fa319c64-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:24.568669 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.568651    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-maas-api-tls\") pod \"maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:24.568743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.568695    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-metrics-certs\") pod \"maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:24.568803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.568776    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx4vt\" (UniqueName: \"kubernetes.io/projected/82e5189f-18d7-405d-a067-ded6fa319c64-kube-api-access-kx4vt\") pod \"maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:24.568863 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:17:24.568808    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-serving-cert: secret "maas-api-e2e-extmod-a-aab75e-serving-cert" not found
Nov 03 02:17:24.568937 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:17:24.568923    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-maas-api-tls podName:82e5189f-18d7-405d-a067-ded6fa319c64 nodeName:}" failed. No retries permitted until 2026-11-03 02:17:25.068899101 +0000 UTC m=+3524.494505564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-maas-api-tls") pod "maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd" (UID: "82e5189f-18d7-405d-a067-ded6fa319c64") : secret "maas-api-e2e-extmod-a-aab75e-serving-cert" not found
Nov 03 02:17:24.569314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.569291    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e5189f-18d7-405d-a067-ded6fa319c64-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:24.571275 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.571249    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-metrics-certs\") pod \"maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:24.641260 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:24.641143    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx4vt\" (UniqueName: \"kubernetes.io/projected/82e5189f-18d7-405d-a067-ded6fa319c64-kube-api-access-kx4vt\") pod \"maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:25.076370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:25.076300    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-maas-api-tls\") pod \"maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:25.078831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:25.078807    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-maas-api-tls\") pod \"maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:25.219576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:25.219534    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" event={"ID":"518ce897-7786-44cc-9be3-737bb47e8e50","Type":"ContainerStarted","Data":"2365d0ad6c101f7b409d4a1cf24ef99e3b8b601c4df9537d7a9f486a78390f7b"}
Nov 03 02:17:25.219798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:25.219603    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" event={"ID":"518ce897-7786-44cc-9be3-737bb47e8e50","Type":"ContainerStarted","Data":"46157cbf9ff5482a6c903197babef67674fc6b40b5ea5f52c41c18fc0b2e66f7"}
Nov 03 02:17:25.301364 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:25.301323    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:25.337020 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:25.336900    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" podStartSLOduration=2.336880302 podStartE2EDuration="2.336880302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:17:25.332042811 +0000 UTC m=+3524.757649278" watchObservedRunningTime="2026-11-03 02:17:25.336880302 +0000 UTC m=+3524.762486768"
Nov 03 02:17:25.685829 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:25.685795    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"]
Nov 03 02:17:25.693162 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:17:25.693125    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e5189f_18d7_405d_a067_ded6fa319c64.slice/crio-a4b703f15769190fa2672cd5ce5ba2ea994ed0ac66c150fbf08c0e5fdb6fb77d WatchSource:0}: Error finding container a4b703f15769190fa2672cd5ce5ba2ea994ed0ac66c150fbf08c0e5fdb6fb77d: Status 404 returned error can't find the container with id a4b703f15769190fa2672cd5ce5ba2ea994ed0ac66c150fbf08c0e5fdb6fb77d
Nov 03 02:17:26.092795 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:26.092708    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:26.099107 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:26.099082    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:26.228055 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:26.228017    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd" event={"ID":"82e5189f-18d7-405d-a067-ded6fa319c64","Type":"ContainerStarted","Data":"a4b703f15769190fa2672cd5ce5ba2ea994ed0ac66c150fbf08c0e5fdb6fb77d"}
Nov 03 02:17:26.228357 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:26.228328    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:26.229442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:26.229425    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:17:27.236433 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:27.236368    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd" event={"ID":"82e5189f-18d7-405d-a067-ded6fa319c64","Type":"ContainerStarted","Data":"56a29572815a3a7ec82264fa54341ea8cdbdaf1cebc7f07d80fe5ce1f4b1b483"}
Nov 03 02:17:27.236433 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:27.236435    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:27.347008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:27.346940    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd" podStartSLOduration=2.311412269 podStartE2EDuration="3.346925756s" totalImagesPullingTime="1.035513487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:17:27.346325399 +0000 UTC m=+3526.771931884" watchObservedRunningTime="2026-11-03 02:17:27.346925756 +0000 UTC m=+3526.772532220"
Nov 03 02:17:33.248995 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:33.248965    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:17:34.463387 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.463237    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"]
Nov 03 02:17:34.469590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.469556    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"
Nov 03 02:17:34.574936 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.574903    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4f87d1-dockercfg-572hr\""
Nov 03 02:17:34.581272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.575469    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4f87d1\""
Nov 03 02:17:34.589015 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.588990    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"]
Nov 03 02:17:34.624759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.618151    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992f07df-b113-42f5-9e9c-e3864e3d9190-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2\" (UID: \"992f07df-b113-42f5-9e9c-e3864e3d9190\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"
Nov 03 02:17:34.626196 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.626149    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zm5p\" (UniqueName: \"kubernetes.io/projected/992f07df-b113-42f5-9e9c-e3864e3d9190-kube-api-access-4zm5p\") pod \"payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2\" (UID: \"992f07df-b113-42f5-9e9c-e3864e3d9190\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"
Nov 03 02:17:34.631105 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.631078    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"]
Nov 03 02:17:34.648586 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.643728    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"
Nov 03 02:17:34.696597 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.696431    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"]
Nov 03 02:17:34.727334 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.727244    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992f07df-b113-42f5-9e9c-e3864e3d9190-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2\" (UID: \"992f07df-b113-42f5-9e9c-e3864e3d9190\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"
Nov 03 02:17:34.727334 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.727294    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96528\" (UniqueName: \"kubernetes.io/projected/ef811d84-4691-43f7-9d23-8033ed536126-kube-api-access-96528\") pod \"payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms\" (UID: \"ef811d84-4691-43f7-9d23-8033ed536126\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"
Nov 03 02:17:34.727334 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.727331    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef811d84-4691-43f7-9d23-8033ed536126-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms\" (UID: \"ef811d84-4691-43f7-9d23-8033ed536126\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"
Nov 03 02:17:34.727598 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.727575    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zm5p\" (UniqueName: \"kubernetes.io/projected/992f07df-b113-42f5-9e9c-e3864e3d9190-kube-api-access-4zm5p\") pod \"payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2\" (UID: \"992f07df-b113-42f5-9e9c-e3864e3d9190\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"
Nov 03 02:17:34.728267 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.728238    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992f07df-b113-42f5-9e9c-e3864e3d9190-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2\" (UID: \"992f07df-b113-42f5-9e9c-e3864e3d9190\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"
Nov 03 02:17:34.785335 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.785301    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zm5p\" (UniqueName: \"kubernetes.io/projected/992f07df-b113-42f5-9e9c-e3864e3d9190-kube-api-access-4zm5p\") pod \"payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2\" (UID: \"992f07df-b113-42f5-9e9c-e3864e3d9190\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"
Nov 03 02:17:34.790151 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.790125    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"
Nov 03 02:17:34.828932 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.828895    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96528\" (UniqueName: \"kubernetes.io/projected/ef811d84-4691-43f7-9d23-8033ed536126-kube-api-access-96528\") pod \"payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms\" (UID: \"ef811d84-4691-43f7-9d23-8033ed536126\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"
Nov 03 02:17:34.829113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.828948    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef811d84-4691-43f7-9d23-8033ed536126-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms\" (UID: \"ef811d84-4691-43f7-9d23-8033ed536126\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"
Nov 03 02:17:34.829935 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.829911    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef811d84-4691-43f7-9d23-8033ed536126-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms\" (UID: \"ef811d84-4691-43f7-9d23-8033ed536126\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"
Nov 03 02:17:34.889417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.889385    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96528\" (UniqueName: \"kubernetes.io/projected/ef811d84-4691-43f7-9d23-8033ed536126-kube-api-access-96528\") pod \"payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms\" (UID: \"ef811d84-4691-43f7-9d23-8033ed536126\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"
Nov 03 02:17:34.975455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:34.975418    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"
Nov 03 02:17:35.175085 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:35.175047    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"]
Nov 03 02:17:35.181144 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:17:35.181115    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992f07df_b113_42f5_9e9c_e3864e3d9190.slice/crio-912aef4d38331857e4e94e681089f687544fe20796ae31f316fb7dd09a2e4f08 WatchSource:0}: Error finding container 912aef4d38331857e4e94e681089f687544fe20796ae31f316fb7dd09a2e4f08: Status 404 returned error can't find the container with id 912aef4d38331857e4e94e681089f687544fe20796ae31f316fb7dd09a2e4f08
Nov 03 02:17:35.303421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:35.303384    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2" event={"ID":"992f07df-b113-42f5-9e9c-e3864e3d9190","Type":"ContainerStarted","Data":"cac5e77c21d5be3a9111280c49351485a6985b3cc26b0ed74e8d59353bd0e93d"}
Nov 03 02:17:35.303617 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:35.303446    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2" event={"ID":"992f07df-b113-42f5-9e9c-e3864e3d9190","Type":"ContainerStarted","Data":"912aef4d38331857e4e94e681089f687544fe20796ae31f316fb7dd09a2e4f08"}
Nov 03 02:17:35.303617 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:35.303515    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"
Nov 03 02:17:35.355370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:35.355328    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"]
Nov 03 02:17:35.361952 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:17:35.361922    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef811d84_4691_43f7_9d23_8033ed536126.slice/crio-0b5918187e2021fe52a5d73e61e3cace5deec71d0613e62d65e203ea69431912 WatchSource:0}: Error finding container 0b5918187e2021fe52a5d73e61e3cace5deec71d0613e62d65e203ea69431912: Status 404 returned error can't find the container with id 0b5918187e2021fe52a5d73e61e3cace5deec71d0613e62d65e203ea69431912
Nov 03 02:17:35.413675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:35.413628    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2" podStartSLOduration=1.413613976 podStartE2EDuration="1.413613976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:17:35.411033941 +0000 UTC m=+3534.836640423" watchObservedRunningTime="2026-11-03 02:17:35.413613976 +0000 UTC m=+3534.839220440"
Nov 03 02:17:36.313082 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:36.313042    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms" event={"ID":"ef811d84-4691-43f7-9d23-8033ed536126","Type":"ContainerStarted","Data":"a1a7947996e8f6f1630989ae71cfe830cb6a8efdf4199ed13ff941629e39c2b1"}
Nov 03 02:17:36.313571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:36.313108    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms" event={"ID":"ef811d84-4691-43f7-9d23-8033ed536126","Type":"ContainerStarted","Data":"0b5918187e2021fe52a5d73e61e3cace5deec71d0613e62d65e203ea69431912"}
Nov 03 02:17:36.313571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:36.313177    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"
Nov 03 02:17:36.423174 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:36.423123    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms" podStartSLOduration=2.423107454 podStartE2EDuration="2.423107454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:17:36.421181759 +0000 UTC m=+3535.846788224" watchObservedRunningTime="2026-11-03 02:17:36.423107454 +0000 UTC m=+3535.848713930"
Nov 03 02:17:43.813135 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:17:43.813095    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:17:46.316304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:46.316275    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"
Nov 03 02:17:47.325046 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:17:47.325016    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"
Nov 03 02:18:12.286572 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:12.286538    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"]
Nov 03 02:18:12.287321 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:12.287289    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn" podUID="a5ef25b1-3348-465e-b5f6-ea831b7b8717" containerName="istio-proxy" containerID="cri-o://1c2abadb172316b9ede5cb82b7ed8ebfd7c04c23153a681b2decebf0c26eb062" gracePeriod=30
Nov 03 02:18:13.833832 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:18:13.833791    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:18:17.579383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.579335    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:18:17.689976 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.689940    5508 generic.go:356] "Generic (PLEG): container finished" podID="a5ef25b1-3348-465e-b5f6-ea831b7b8717" containerID="1c2abadb172316b9ede5cb82b7ed8ebfd7c04c23153a681b2decebf0c26eb062" exitCode=0
Nov 03 02:18:17.690189 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.689985    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"
Nov 03 02:18:17.690189 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.690004    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn" event={"ID":"a5ef25b1-3348-465e-b5f6-ea831b7b8717","Type":"ContainerDied","Data":"1c2abadb172316b9ede5cb82b7ed8ebfd7c04c23153a681b2decebf0c26eb062"}
Nov 03 02:18:17.690189 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.690060    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn" event={"ID":"a5ef25b1-3348-465e-b5f6-ea831b7b8717","Type":"ContainerDied","Data":"89b78a9e6a953d2810c53dbf5e759a6e5e443c02cbddcc9fddfd733863d46605"}
Nov 03 02:18:17.690189 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.690089    5508 scope.go:122] "RemoveContainer" containerID="1c2abadb172316b9ede5cb82b7ed8ebfd7c04c23153a681b2decebf0c26eb062"
Nov 03 02:18:17.701132 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.701113    5508 scope.go:122] "RemoveContainer" containerID="1c2abadb172316b9ede5cb82b7ed8ebfd7c04c23153a681b2decebf0c26eb062"
Nov 03 02:18:17.701478 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:18:17.701457    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2abadb172316b9ede5cb82b7ed8ebfd7c04c23153a681b2decebf0c26eb062\": container with ID starting with 1c2abadb172316b9ede5cb82b7ed8ebfd7c04c23153a681b2decebf0c26eb062 not found: ID does not exist" containerID="1c2abadb172316b9ede5cb82b7ed8ebfd7c04c23153a681b2decebf0c26eb062"
Nov 03 02:18:17.701558 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.701491    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2abadb172316b9ede5cb82b7ed8ebfd7c04c23153a681b2decebf0c26eb062"} err="failed to get container status \"1c2abadb172316b9ede5cb82b7ed8ebfd7c04c23153a681b2decebf0c26eb062\": rpc error: code = NotFound desc = could not find container \"1c2abadb172316b9ede5cb82b7ed8ebfd7c04c23153a681b2decebf0c26eb062\": container with ID starting with 1c2abadb172316b9ede5cb82b7ed8ebfd7c04c23153a681b2decebf0c26eb062 not found: ID does not exist"
Nov 03 02:18:17.719035 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719006    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-socket\") pod \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") "
Nov 03 02:18:17.719184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719051    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-certs\") pod \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") "
Nov 03 02:18:17.719184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719082    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-podinfo\") pod \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") "
Nov 03 02:18:17.719295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719242    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istiod-ca-cert\") pod \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") "
Nov 03 02:18:17.719371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719300    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-socket" pod "a5ef25b1-3348-465e-b5f6-ea831b7b8717" (UID: "a5ef25b1-3348-465e-b5f6-ea831b7b8717"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:18:17.719371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719316    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-token\" (UniqueName: \"kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-token\") pod \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") "
Nov 03 02:18:17.719489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719377    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-certs" pod "a5ef25b1-3348-465e-b5f6-ea831b7b8717" (UID: "a5ef25b1-3348-465e-b5f6-ea831b7b8717"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:18:17.719489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719452    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-data\") pod \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") "
Nov 03 02:18:17.719595 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719485    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-kube-api-access-rczlc\" (UniqueName: \"kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-kube-api-access-rczlc\") pod \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") "
Nov 03 02:18:17.719595 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719572    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-credential-socket\") pod \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") "
Nov 03 02:18:17.719714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719609    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istiod-ca-cert" pod "a5ef25b1-3348-465e-b5f6-ea831b7b8717" (UID: "a5ef25b1-3348-465e-b5f6-ea831b7b8717"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:18:17.719714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719615    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-envoy\") pod \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\" (UID: \"a5ef25b1-3348-465e-b5f6-ea831b7b8717\") "
Nov 03 02:18:17.719714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719637    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-data" pod "a5ef25b1-3348-465e-b5f6-ea831b7b8717" (UID: "a5ef25b1-3348-465e-b5f6-ea831b7b8717"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:18:17.719882 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.719834    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-credential-socket" pod "a5ef25b1-3348-465e-b5f6-ea831b7b8717" (UID: "a5ef25b1-3348-465e-b5f6-ea831b7b8717"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:18:17.720221 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.720198    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:17.720320 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.720225    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:17.720320 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.720238    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:17.720320 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.720248    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:17.720320 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.720260    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:17.721796 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.721767    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-kube-api-access-rczlc" pod "a5ef25b1-3348-465e-b5f6-ea831b7b8717" (UID: "a5ef25b1-3348-465e-b5f6-ea831b7b8717"). InnerVolumeSpecName "kube-api-access-rczlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:18:17.721907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.721812    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-envoy" pod "a5ef25b1-3348-465e-b5f6-ea831b7b8717" (UID: "a5ef25b1-3348-465e-b5f6-ea831b7b8717"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:18:17.721973 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.721954    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-token" pod "a5ef25b1-3348-465e-b5f6-ea831b7b8717" (UID: "a5ef25b1-3348-465e-b5f6-ea831b7b8717"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:18:17.722033 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.722007    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-podinfo" pod "a5ef25b1-3348-465e-b5f6-ea831b7b8717" (UID: "a5ef25b1-3348-465e-b5f6-ea831b7b8717"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:18:17.821325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.821136    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:17.821325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.821175    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rczlc\" (UniqueName: \"kubernetes.io/projected/a5ef25b1-3348-465e-b5f6-ea831b7b8717-kube-api-access-rczlc\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:17.821325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.821191    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:17.821325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:17.821205    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5ef25b1-3348-465e-b5f6-ea831b7b8717-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:18:18.179874 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:18.179827    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"]
Nov 03 02:18:18.191893 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:18.190925    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-898926-openshift-default-575b479d9-7p5fn"]
Nov 03 02:18:18.304189 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:18.304140    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:18:18.695298 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:18.695264    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ef25b1-3348-465e-b5f6-ea831b7b8717" path="/var/lib/kubelet/pods/a5ef25b1-3348-465e-b5f6-ea831b7b8717/volumes"
Nov 03 02:18:41.389870 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:41.389843    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:18:41.460925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:41.460901    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:18:43.813717 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:18:43.813679    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:18:56.682432 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.682391    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"]
Nov 03 02:18:56.683175 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.683148    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a5ef25b1-3348-465e-b5f6-ea831b7b8717" containerName="istio-proxy"
Nov 03 02:18:56.683381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.683340    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ef25b1-3348-465e-b5f6-ea831b7b8717" containerName="istio-proxy"
Nov 03 02:18:56.683544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.683532    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ef25b1-3348-465e-b5f6-ea831b7b8717" containerName="istio-proxy"
Nov 03 02:18:56.687919 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.687892    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:56.736504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.736470    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-4f87d1-serving-cert\""
Nov 03 02:18:56.769728 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.769690    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-metrics-certs\") pod \"maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:56.769923 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.769741    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:56.769994 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.769935    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr8xd\" (UniqueName: \"kubernetes.io/projected/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-kube-api-access-jr8xd\") pod \"maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:56.770092 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.770069    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:56.786101 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.786062    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"]
Nov 03 02:18:56.871284 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.871246    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-metrics-certs\") pod \"maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:56.871493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.871291    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:56.871493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.871379    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr8xd\" (UniqueName: \"kubernetes.io/projected/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-kube-api-access-jr8xd\") pod \"maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:56.871493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.871441    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:56.871610 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:18:56.871571    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-serving-cert: secret "maas-api-e2e-extmod-b-4f87d1-serving-cert" not found
Nov 03 02:18:56.871655 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:18:56.871635    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-maas-api-tls podName:9116085a-5fd8-41cd-9f51-5b5f3e1474c9 nodeName:}" failed. No retries permitted until 2026-11-03 02:18:57.371614804 +0000 UTC m=+3616.797221264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-maas-api-tls") pod "maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg" (UID: "9116085a-5fd8-41cd-9f51-5b5f3e1474c9") : secret "maas-api-e2e-extmod-b-4f87d1-serving-cert" not found
Nov 03 02:18:56.872044 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.872025    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:56.873744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.873725    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-metrics-certs\") pod \"maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:56.929692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.929653    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr8xd\" (UniqueName: \"kubernetes.io/projected/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-kube-api-access-jr8xd\") pod \"maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:56.992411 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.992279    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"]
Nov 03 02:18:56.997363 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:56.997322    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.047762 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.046273    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-bd872a-openshift-default-dockercfg-tp6g6\""
Nov 03 02:18:57.060670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.060624    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"]
Nov 03 02:18:57.077876 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.077845    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-envoy\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.078055 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.077894    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-data\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.078153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.078115    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-istio-token\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.078485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.078216    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438cacfe-f668-4897-992c-1faa9b411850-istiod-ca-cert\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.078485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.078306    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-socket\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.078880 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.078803    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-certs\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.078880 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.078846    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvwg\" (UniqueName: \"kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-kube-api-access-fhvwg\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.079368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.079321    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438cacfe-f668-4897-992c-1faa9b411850-istio-podinfo\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.079714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.079603    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-credential-socket\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.188159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.188110    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-data\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.188527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.188501    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-istio-token\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.189576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.188666    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438cacfe-f668-4897-992c-1faa9b411850-istiod-ca-cert\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.189576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.188709    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-socket\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.189576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.188789    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-certs\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.189576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.188814    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvwg\" (UniqueName: \"kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-kube-api-access-fhvwg\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.189576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.188946    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438cacfe-f668-4897-992c-1faa9b411850-istio-podinfo\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.189576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.189016    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-credential-socket\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.189576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.189058    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-envoy\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.205569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.190408    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-socket\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.205569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.190664    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-data\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.205569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.205101    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-certs\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.206622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.206242    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438cacfe-f668-4897-992c-1faa9b411850-istiod-ca-cert\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.206622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.206552    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-credential-socket\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.220914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.220833    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-envoy\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.223929 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.221647    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438cacfe-f668-4897-992c-1faa9b411850-istio-podinfo\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.258144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.258056    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-istio-token\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.283155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.283115    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvwg\" (UniqueName: \"kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-kube-api-access-fhvwg\") pod \"e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") " pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.312250 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.312218    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:57.395415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.394803    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:57.398238 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.398175    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:57.602374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.602273    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:57.748687 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.748651    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"]
Nov 03 02:18:57.755501 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:18:57.755473    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438cacfe_f668_4897_992c_1faa9b411850.slice/crio-89acb7ebf030d2ac765b683c4ad757307e6b6d1225198ff766c2604948a9402c WatchSource:0}: Error finding container 89acb7ebf030d2ac765b683c4ad757307e6b6d1225198ff766c2604948a9402c: Status 404 returned error can't find the container with id 89acb7ebf030d2ac765b683c4ad757307e6b6d1225198ff766c2604948a9402c
Nov 03 02:18:57.758469 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.758438    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:18:57.758573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.758511    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:18:57.758573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.758542    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:18:57.981075 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.981033    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"]
Nov 03 02:18:57.988170 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:18:57.988135    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9116085a_5fd8_41cd_9f51_5b5f3e1474c9.slice/crio-2596b7bcd67d734f30e5ac04b9f1cdb7c21335ae5190a5e1af58697eedb177e6 WatchSource:0}: Error finding container 2596b7bcd67d734f30e5ac04b9f1cdb7c21335ae5190a5e1af58697eedb177e6: Status 404 returned error can't find the container with id 2596b7bcd67d734f30e5ac04b9f1cdb7c21335ae5190a5e1af58697eedb177e6
Nov 03 02:18:57.989906 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:57.989882    5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:18:58.010047 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:58.009641    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" event={"ID":"438cacfe-f668-4897-992c-1faa9b411850","Type":"ContainerStarted","Data":"b6712fd1a15cf9020e702244c11d10f3d0f815053d7d727cf5490b5fcdfddb36"}
Nov 03 02:18:58.010047 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:58.009708    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" event={"ID":"438cacfe-f668-4897-992c-1faa9b411850","Type":"ContainerStarted","Data":"89acb7ebf030d2ac765b683c4ad757307e6b6d1225198ff766c2604948a9402c"}
Nov 03 02:18:58.011265 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:58.011230    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg" event={"ID":"9116085a-5fd8-41cd-9f51-5b5f3e1474c9","Type":"ContainerStarted","Data":"2596b7bcd67d734f30e5ac04b9f1cdb7c21335ae5190a5e1af58697eedb177e6"}
Nov 03 02:18:58.124615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:58.124562    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" podStartSLOduration=2.124545208 podStartE2EDuration="2.124545208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:58.122198757 +0000 UTC m=+3617.547805241" watchObservedRunningTime="2026-11-03 02:18:58.124545208 +0000 UTC m=+3617.550151673"
Nov 03 02:18:58.313020 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:58.312912    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:58.318993 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:58.318970    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:59.022494 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:59.022446    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg" event={"ID":"9116085a-5fd8-41cd-9f51-5b5f3e1474c9","Type":"ContainerStarted","Data":"5d68efc78df5907f0f2d8bb412c23bc014155c63716c535373429945853eed89"}
Nov 03 02:18:59.022981 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:59.022527    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:18:59.022981 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:59.022734    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:59.024027 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:59.024006    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:18:59.133308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:18:59.133251    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg" podStartSLOduration=2.199796158 podStartE2EDuration="3.133229528s" totalImagesPullingTime="933.43337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:18:59.132154901 +0000 UTC m=+3618.557761386" watchObservedRunningTime="2026-11-03 02:18:59.133229528 +0000 UTC m=+3618.558835993"
Nov 03 02:19:05.035212 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:05.035183    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:19:07.098867 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:07.098833    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"]
Nov 03 02:19:07.105691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:07.105664    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"
Nov 03 02:19:07.152278 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:07.152241    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"]
Nov 03 02:19:07.176495 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:07.176452    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6\" (UID: \"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"
Nov 03 02:19:07.176724 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:07.176606    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mksmt\" (UniqueName: \"kubernetes.io/projected/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-kube-api-access-mksmt\") pod \"payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6\" (UID: \"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"
Nov 03 02:19:07.195089 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:07.195054    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-bd872a\""
Nov 03 02:19:07.195089 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:07.195065    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-bd872a-dockercfg-pq7wt\""
Nov 03 02:19:07.278464 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:07.278424    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mksmt\" (UniqueName: \"kubernetes.io/projected/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-kube-api-access-mksmt\") pod \"payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6\" (UID: \"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"
Nov 03 02:19:07.278685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:07.278647    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6\" (UID: \"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"
Nov 03 02:19:07.279685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:07.279659    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6\" (UID: \"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"
Nov 03 02:19:07.336479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:07.336440    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mksmt\" (UniqueName: \"kubernetes.io/projected/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-kube-api-access-mksmt\") pod \"payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6\" (UID: \"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"
Nov 03 02:19:07.423601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:07.423564    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"
Nov 03 02:19:07.804936 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:07.804858    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"]
Nov 03 02:19:07.811619 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:19:07.811587    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba7f2d3_aebd_4c59_8b12_193c8e88df5d.slice/crio-a6cbedff402e0c6e370c417600784acbb25889a0226c92779db1a858132cdc68 WatchSource:0}: Error finding container a6cbedff402e0c6e370c417600784acbb25889a0226c92779db1a858132cdc68: Status 404 returned error can't find the container with id a6cbedff402e0c6e370c417600784acbb25889a0226c92779db1a858132cdc68
Nov 03 02:19:08.107925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:08.107826    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6" event={"ID":"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d","Type":"ContainerStarted","Data":"960de9e1c1453c4bbb7b4cf968380ef84a3978a6620949968fe84c38477353ac"}
Nov 03 02:19:08.107925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:08.107906    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"
Nov 03 02:19:08.108422 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:08.107933    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6" event={"ID":"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d","Type":"ContainerStarted","Data":"a6cbedff402e0c6e370c417600784acbb25889a0226c92779db1a858132cdc68"}
Nov 03 02:19:08.220268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:08.220200    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6" podStartSLOduration=1.220182575 podStartE2EDuration="1.220182575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:19:08.217040429 +0000 UTC m=+3627.642646893" watchObservedRunningTime="2026-11-03 02:19:08.220182575 +0000 UTC m=+3627.645789040"
Nov 03 02:19:11.520892 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:11.520849    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"]
Nov 03 02:19:11.526147 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:11.526121    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"
Nov 03 02:19:11.589376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:11.582912    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"]
Nov 03 02:19:11.625368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:11.624729    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-3fb07e-dockercfg-jjnzk\""
Nov 03 02:19:11.625368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:11.624809    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-3fb07e\""
Nov 03 02:19:11.694071 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:11.691614    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6nt\" (UniqueName: \"kubernetes.io/projected/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-kube-api-access-5l6nt\") pod \"payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6\" (UID: \"ff52d99c-d352-4ebd-91dc-1b95ecbdf766\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"
Nov 03 02:19:11.694071 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:11.691865    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6\" (UID: \"ff52d99c-d352-4ebd-91dc-1b95ecbdf766\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"
Nov 03 02:19:11.793571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:11.793468    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6nt\" (UniqueName: \"kubernetes.io/projected/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-kube-api-access-5l6nt\") pod \"payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6\" (UID: \"ff52d99c-d352-4ebd-91dc-1b95ecbdf766\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"
Nov 03 02:19:11.793770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:11.793581    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6\" (UID: \"ff52d99c-d352-4ebd-91dc-1b95ecbdf766\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"
Nov 03 02:19:11.794363 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:11.794322    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6\" (UID: \"ff52d99c-d352-4ebd-91dc-1b95ecbdf766\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"
Nov 03 02:19:11.851041 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:11.850997    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6nt\" (UniqueName: \"kubernetes.io/projected/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-kube-api-access-5l6nt\") pod \"payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6\" (UID: \"ff52d99c-d352-4ebd-91dc-1b95ecbdf766\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"
Nov 03 02:19:12.140398 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:12.140291    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"
Nov 03 02:19:12.529076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:12.529036    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"]
Nov 03 02:19:12.535190 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:19:12.535162    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff52d99c_d352_4ebd_91dc_1b95ecbdf766.slice/crio-7ea337e8ccecf366de16f387f2d2397926454ff9b408ecab1bd20869e9de2337 WatchSource:0}: Error finding container 7ea337e8ccecf366de16f387f2d2397926454ff9b408ecab1bd20869e9de2337: Status 404 returned error can't find the container with id 7ea337e8ccecf366de16f387f2d2397926454ff9b408ecab1bd20869e9de2337
Nov 03 02:19:13.149164 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:13.149122    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6" event={"ID":"ff52d99c-d352-4ebd-91dc-1b95ecbdf766","Type":"ContainerStarted","Data":"81cde068f51317ec3a3b6e7375a3c17b796b3106c740da843923ba2f3009c837"}
Nov 03 02:19:13.149401 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:13.149195    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"
Nov 03 02:19:13.149401 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:13.149214    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6" event={"ID":"ff52d99c-d352-4ebd-91dc-1b95ecbdf766","Type":"ContainerStarted","Data":"7ea337e8ccecf366de16f387f2d2397926454ff9b408ecab1bd20869e9de2337"}
Nov 03 02:19:13.259736 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:13.259672    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6" podStartSLOduration=2.259653636 podStartE2EDuration="2.259653636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:19:13.257070947 +0000 UTC m=+3632.682677411" watchObservedRunningTime="2026-11-03 02:19:13.259653636 +0000 UTC m=+3632.685260102"
Nov 03 02:19:13.813692 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:19:13.813654    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:19:14.845864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:14.845828    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:19:19.117907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:19.117877    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"
Nov 03 02:19:24.162122 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:24.162088    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"
Nov 03 02:19:26.229544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:26.229507    5508 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Nov 03 02:19:26.230013 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:26.229579    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Nov 03 02:19:26.291668 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:26.291634    5508 generic.go:356] "Generic (PLEG): container finished" podID="518ce897-7786-44cc-9be3-737bb47e8e50" containerID="2365d0ad6c101f7b409d4a1cf24ef99e3b8b601c4df9537d7a9f486a78390f7b" exitCode=0
Nov 03 02:19:26.291878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:26.291680    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" event={"ID":"518ce897-7786-44cc-9be3-737bb47e8e50","Type":"ContainerDied","Data":"2365d0ad6c101f7b409d4a1cf24ef99e3b8b601c4df9537d7a9f486a78390f7b"}
Nov 03 02:19:26.292195 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:26.292179    5508 scope.go:122] "RemoveContainer" containerID="2365d0ad6c101f7b409d4a1cf24ef99e3b8b601c4df9537d7a9f486a78390f7b"
Nov 03 02:19:26.294048 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:26.294015    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:19:26.294164 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:26.294093    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:19:26.294164 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:26.294125    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:19:27.301714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:27.301666    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" event={"ID":"518ce897-7786-44cc-9be3-737bb47e8e50","Type":"ContainerStarted","Data":"92c4954a9221cc6a0597823243bd2fa368ec33498c6124e01c782b51b4b8388e"}
Nov 03 02:19:28.092742 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:28.092703    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:19:28.092964 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:28.092919    5508 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Nov 03 02:19:28.093052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:28.092975    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Nov 03 02:19:29.092918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:29.092869    5508 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Nov 03 02:19:29.093494 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:29.092947    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Nov 03 02:19:30.093022 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:30.092985    5508 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Nov 03 02:19:30.093616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:30.093048    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Nov 03 02:19:31.092874 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:31.092799    5508 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Nov 03 02:19:31.092874 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:31.092853    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Nov 03 02:19:31.411097 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:31.411061    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"]
Nov 03 02:19:32.093280 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:32.093237    5508 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Nov 03 02:19:32.093517 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:32.093308    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Nov 03 02:19:33.098030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:33.098004    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:19:33.098482 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:33.098464    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:19:33.099262 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:33.099240    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:19:36.412336 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:36.412290    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9" podUID="641e8ee2-7527-4832-9e44-72dbb37b208a" containerName="payload-pre-processing" containerID="cri-o://2abf57990cc49d8e7c6c6ac82e0ea7195b59e84ef114a011c950068720574e13" gracePeriod=25
Nov 03 02:19:36.676870 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:36.676810    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"
Nov 03 02:19:36.770356 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:36.770325    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641e8ee2-7527-4832-9e44-72dbb37b208a-kube-api-access-wt8tc\" (UniqueName: \"kubernetes.io/projected/641e8ee2-7527-4832-9e44-72dbb37b208a-kube-api-access-wt8tc\") pod \"641e8ee2-7527-4832-9e44-72dbb37b208a\" (UID: \"641e8ee2-7527-4832-9e44-72dbb37b208a\") "
Nov 03 02:19:36.770589 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:36.770410    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641e8ee2-7527-4832-9e44-72dbb37b208a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641e8ee2-7527-4832-9e44-72dbb37b208a-plugins-config-volume\") pod \"641e8ee2-7527-4832-9e44-72dbb37b208a\" (UID: \"641e8ee2-7527-4832-9e44-72dbb37b208a\") "
Nov 03 02:19:36.770814 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:36.770791    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641e8ee2-7527-4832-9e44-72dbb37b208a-plugins-config-volume" pod "641e8ee2-7527-4832-9e44-72dbb37b208a" (UID: "641e8ee2-7527-4832-9e44-72dbb37b208a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:19:36.772521 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:36.772493    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641e8ee2-7527-4832-9e44-72dbb37b208a-kube-api-access-wt8tc" pod "641e8ee2-7527-4832-9e44-72dbb37b208a" (UID: "641e8ee2-7527-4832-9e44-72dbb37b208a"). InnerVolumeSpecName "kube-api-access-wt8tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:19:36.871931 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:36.871895    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt8tc\" (UniqueName: \"kubernetes.io/projected/641e8ee2-7527-4832-9e44-72dbb37b208a-kube-api-access-wt8tc\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:19:36.871931 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:36.871926    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/641e8ee2-7527-4832-9e44-72dbb37b208a-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:19:37.376211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:37.376168    5508 generic.go:356] "Generic (PLEG): container finished" podID="641e8ee2-7527-4832-9e44-72dbb37b208a" containerID="2abf57990cc49d8e7c6c6ac82e0ea7195b59e84ef114a011c950068720574e13" exitCode=0
Nov 03 02:19:37.376440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:37.376235    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"
Nov 03 02:19:37.376440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:37.376253    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9" event={"ID":"641e8ee2-7527-4832-9e44-72dbb37b208a","Type":"ContainerDied","Data":"2abf57990cc49d8e7c6c6ac82e0ea7195b59e84ef114a011c950068720574e13"}
Nov 03 02:19:37.376440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:37.376323    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9" event={"ID":"641e8ee2-7527-4832-9e44-72dbb37b208a","Type":"ContainerDied","Data":"ecb3a559b083eafab32c14859a91e1a20f8c5ac319491cbc055ac3b6b8146a6e"}
Nov 03 02:19:37.376440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:37.376372    5508 scope.go:122] "RemoveContainer" containerID="2abf57990cc49d8e7c6c6ac82e0ea7195b59e84ef114a011c950068720574e13"
Nov 03 02:19:37.392519 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:37.392494    5508 scope.go:122] "RemoveContainer" containerID="2abf57990cc49d8e7c6c6ac82e0ea7195b59e84ef114a011c950068720574e13"
Nov 03 02:19:37.392841 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:19:37.392820    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2abf57990cc49d8e7c6c6ac82e0ea7195b59e84ef114a011c950068720574e13\": container with ID starting with 2abf57990cc49d8e7c6c6ac82e0ea7195b59e84ef114a011c950068720574e13 not found: ID does not exist" containerID="2abf57990cc49d8e7c6c6ac82e0ea7195b59e84ef114a011c950068720574e13"
Nov 03 02:19:37.392936 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:37.392853    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2abf57990cc49d8e7c6c6ac82e0ea7195b59e84ef114a011c950068720574e13"} err="failed to get container status \"2abf57990cc49d8e7c6c6ac82e0ea7195b59e84ef114a011c950068720574e13\": rpc error: code = NotFound desc = could not find container \"2abf57990cc49d8e7c6c6ac82e0ea7195b59e84ef114a011c950068720574e13\": container with ID starting with 2abf57990cc49d8e7c6c6ac82e0ea7195b59e84ef114a011c950068720574e13 not found: ID does not exist"
Nov 03 02:19:37.531062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:37.531025    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"]
Nov 03 02:19:37.577616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:37.577579    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-50ca6d-5b8444c955-4jds9"]
Nov 03 02:19:38.694098 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:38.694061    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641e8ee2-7527-4832-9e44-72dbb37b208a" path="/var/lib/kubelet/pods/641e8ee2-7527-4832-9e44-72dbb37b208a/volumes"
Nov 03 02:19:40.322906 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:40.322873    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"]
Nov 03 02:19:40.422881 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:40.422846    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"]
Nov 03 02:19:43.812700 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:19:43.812662    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:19:45.323654 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.323616    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd" podUID="15c2316d-4bf4-47f1-8804-cb1d17ef28ba" containerName="payload-processing" containerID="cri-o://260846b983afef3b8d909b1ccd0baa21f3ce8a87eb2e31f8ff1088e26b02ed58" gracePeriod=25
Nov 03 02:19:45.423821 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.423765    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx" podUID="c5c694d7-f5c9-4288-93ef-ea5a4011e87c" containerName="payload-pre-processing" containerID="cri-o://991b91962f8b3de19919820f40e718c464886825c2fee64e66845f36f8e2373d" gracePeriod=25
Nov 03 02:19:45.441389 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.441343    5508 generic.go:356] "Generic (PLEG): container finished" podID="15c2316d-4bf4-47f1-8804-cb1d17ef28ba" containerID="260846b983afef3b8d909b1ccd0baa21f3ce8a87eb2e31f8ff1088e26b02ed58" exitCode=0
Nov 03 02:19:45.441389 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.441378    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd" event={"ID":"15c2316d-4bf4-47f1-8804-cb1d17ef28ba","Type":"ContainerDied","Data":"260846b983afef3b8d909b1ccd0baa21f3ce8a87eb2e31f8ff1088e26b02ed58"}
Nov 03 02:19:45.624110 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.623737    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"
Nov 03 02:19:45.720204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.720177    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-kube-api-access-9dxzp\" (UniqueName: \"kubernetes.io/projected/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-kube-api-access-9dxzp\") pod \"15c2316d-4bf4-47f1-8804-cb1d17ef28ba\" (UID: \"15c2316d-4bf4-47f1-8804-cb1d17ef28ba\") "
Nov 03 02:19:45.720562 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.720539    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-plugins-config-volume\") pod \"15c2316d-4bf4-47f1-8804-cb1d17ef28ba\" (UID: \"15c2316d-4bf4-47f1-8804-cb1d17ef28ba\") "
Nov 03 02:19:45.720945 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.720921    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-plugins-config-volume" pod "15c2316d-4bf4-47f1-8804-cb1d17ef28ba" (UID: "15c2316d-4bf4-47f1-8804-cb1d17ef28ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:19:45.722610 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.722584    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-kube-api-access-9dxzp" pod "15c2316d-4bf4-47f1-8804-cb1d17ef28ba" (UID: "15c2316d-4bf4-47f1-8804-cb1d17ef28ba"). InnerVolumeSpecName "kube-api-access-9dxzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:19:45.733143 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.733123    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"
Nov 03 02:19:45.821418 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.821380    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-kube-api-access-bcp6c\" (UniqueName: \"kubernetes.io/projected/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-kube-api-access-bcp6c\") pod \"c5c694d7-f5c9-4288-93ef-ea5a4011e87c\" (UID: \"c5c694d7-f5c9-4288-93ef-ea5a4011e87c\") "
Nov 03 02:19:45.821616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.821517    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-plugins-config-volume\") pod \"c5c694d7-f5c9-4288-93ef-ea5a4011e87c\" (UID: \"c5c694d7-f5c9-4288-93ef-ea5a4011e87c\") "
Nov 03 02:19:45.821863 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.821843    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:19:45.821990 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.821856    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-plugins-config-volume" pod "c5c694d7-f5c9-4288-93ef-ea5a4011e87c" (UID: "c5c694d7-f5c9-4288-93ef-ea5a4011e87c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:19:45.821990 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.821864    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dxzp\" (UniqueName: \"kubernetes.io/projected/15c2316d-4bf4-47f1-8804-cb1d17ef28ba-kube-api-access-9dxzp\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:19:45.823649 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.823631    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-kube-api-access-bcp6c" pod "c5c694d7-f5c9-4288-93ef-ea5a4011e87c" (UID: "c5c694d7-f5c9-4288-93ef-ea5a4011e87c"). InnerVolumeSpecName "kube-api-access-bcp6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:19:45.923598 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.923557    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:19:45.923598 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:45.923599    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcp6c\" (UniqueName: \"kubernetes.io/projected/c5c694d7-f5c9-4288-93ef-ea5a4011e87c-kube-api-access-bcp6c\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:19:46.451192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.451149    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd" event={"ID":"15c2316d-4bf4-47f1-8804-cb1d17ef28ba","Type":"ContainerDied","Data":"089e080aaf9ad11519ea1b6abfbe696959265ecdc50f801c937ea14c8af75277"}
Nov 03 02:19:46.451730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.451164    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"
Nov 03 02:19:46.451730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.451264    5508 scope.go:122] "RemoveContainer" containerID="260846b983afef3b8d909b1ccd0baa21f3ce8a87eb2e31f8ff1088e26b02ed58"
Nov 03 02:19:46.453644 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.453490    5508 generic.go:356] "Generic (PLEG): container finished" podID="c5c694d7-f5c9-4288-93ef-ea5a4011e87c" containerID="991b91962f8b3de19919820f40e718c464886825c2fee64e66845f36f8e2373d" exitCode=0
Nov 03 02:19:46.453644 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.453545    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx" event={"ID":"c5c694d7-f5c9-4288-93ef-ea5a4011e87c","Type":"ContainerDied","Data":"991b91962f8b3de19919820f40e718c464886825c2fee64e66845f36f8e2373d"}
Nov 03 02:19:46.453644 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.453563    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"
Nov 03 02:19:46.453644 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.453606    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx" event={"ID":"c5c694d7-f5c9-4288-93ef-ea5a4011e87c","Type":"ContainerDied","Data":"b690fa6bc5ed45d0a5b631fcf41552ed1fafdb277cfb79298cddb1c7edbb02d4"}
Nov 03 02:19:46.464748 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.464724    5508 scope.go:122] "RemoveContainer" containerID="991b91962f8b3de19919820f40e718c464886825c2fee64e66845f36f8e2373d"
Nov 03 02:19:46.476316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.476295    5508 scope.go:122] "RemoveContainer" containerID="991b91962f8b3de19919820f40e718c464886825c2fee64e66845f36f8e2373d"
Nov 03 02:19:46.476674 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:19:46.476651    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991b91962f8b3de19919820f40e718c464886825c2fee64e66845f36f8e2373d\": container with ID starting with 991b91962f8b3de19919820f40e718c464886825c2fee64e66845f36f8e2373d not found: ID does not exist" containerID="991b91962f8b3de19919820f40e718c464886825c2fee64e66845f36f8e2373d"
Nov 03 02:19:46.476759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.476686    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991b91962f8b3de19919820f40e718c464886825c2fee64e66845f36f8e2373d"} err="failed to get container status \"991b91962f8b3de19919820f40e718c464886825c2fee64e66845f36f8e2373d\": rpc error: code = NotFound desc = could not find container \"991b91962f8b3de19919820f40e718c464886825c2fee64e66845f36f8e2373d\": container with ID starting with 991b91962f8b3de19919820f40e718c464886825c2fee64e66845f36f8e2373d not found: ID does not exist"
Nov 03 02:19:46.608756 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.608667    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"]
Nov 03 02:19:46.654893 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.654860    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8cc55-865795f655-qzkdx"]
Nov 03 02:19:46.695664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.695636    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c694d7-f5c9-4288-93ef-ea5a4011e87c" path="/var/lib/kubelet/pods/c5c694d7-f5c9-4288-93ef-ea5a4011e87c/volumes"
Nov 03 02:19:46.713906 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.713876    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"]
Nov 03 02:19:46.718012 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:46.717981    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d8cc55-645848879-2tbqd"]
Nov 03 02:19:48.693023 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:19:48.692992    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c2316d-4bf4-47f1-8804-cb1d17ef28ba" path="/var/lib/kubelet/pods/15c2316d-4bf4-47f1-8804-cb1d17ef28ba/volumes"
Nov 03 02:20:13.811679 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:20:13.811637    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:20:27.047695 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:27.047607    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"]
Nov 03 02:20:27.093867 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:27.093836    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e47db7-jd6vd"]
Nov 03 02:20:28.693335 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:28.693301    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b00055e-a35c-4c0e-8ce7-1758f5dea910" path="/var/lib/kubelet/pods/8b00055e-a35c-4c0e-8ce7-1758f5dea910/volumes"
Nov 03 02:20:33.874495 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:33.874464    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"]
Nov 03 02:20:35.263376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.263314    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"]
Nov 03 02:20:35.264330 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.264305    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c5c694d7-f5c9-4288-93ef-ea5a4011e87c" containerName="payload-pre-processing"
Nov 03 02:20:35.264330 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.264331    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c694d7-f5c9-4288-93ef-ea5a4011e87c" containerName="payload-pre-processing"
Nov 03 02:20:35.266723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.264388    5508 cpu_manager.go:414] "containerMap: removing container" podUID="15c2316d-4bf4-47f1-8804-cb1d17ef28ba" containerName="payload-processing"
Nov 03 02:20:35.266723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.264396    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c2316d-4bf4-47f1-8804-cb1d17ef28ba" containerName="payload-processing"
Nov 03 02:20:35.266723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.264448    5508 cpu_manager.go:414] "containerMap: removing container" podUID="641e8ee2-7527-4832-9e44-72dbb37b208a" containerName="payload-pre-processing"
Nov 03 02:20:35.266723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.264458    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641e8ee2-7527-4832-9e44-72dbb37b208a" containerName="payload-pre-processing"
Nov 03 02:20:35.266723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.264643    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c694d7-f5c9-4288-93ef-ea5a4011e87c" containerName="payload-pre-processing"
Nov 03 02:20:35.266723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.264666    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c2316d-4bf4-47f1-8804-cb1d17ef28ba" containerName="payload-processing"
Nov 03 02:20:35.266723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.264681    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="641e8ee2-7527-4832-9e44-72dbb37b208a" containerName="payload-pre-processing"
Nov 03 02:20:35.269680 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.269658    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:35.319589 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.319564    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-bd872a-serving-cert\""
Nov 03 02:20:35.350596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.350560    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90da2e53-91d3-4e77-b21a-45d8226ab9f5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:35.350782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.350609    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-metrics-certs\") pod \"maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:35.350782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.350703    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:35.350782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.350751    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c2ck\" (UniqueName: \"kubernetes.io/projected/90da2e53-91d3-4e77-b21a-45d8226ab9f5-kube-api-access-7c2ck\") pod \"maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:35.369332 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.369300    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"]
Nov 03 02:20:35.452480 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.452448    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90da2e53-91d3-4e77-b21a-45d8226ab9f5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:35.452696 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.452492    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-metrics-certs\") pod \"maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:35.452696 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.452521    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:35.452696 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.452562    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c2ck\" (UniqueName: \"kubernetes.io/projected/90da2e53-91d3-4e77-b21a-45d8226ab9f5-kube-api-access-7c2ck\") pod \"maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:35.452696 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:20:35.452676    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-serving-cert: secret "maas-api-e2e-extmod-a-bd872a-serving-cert" not found
Nov 03 02:20:35.452922 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:20:35.452772    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-maas-api-tls podName:90da2e53-91d3-4e77-b21a-45d8226ab9f5 nodeName:}" failed. No retries permitted until 2026-11-03 02:20:35.952749994 +0000 UTC m=+3715.378356462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-maas-api-tls") pod "maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h" (UID: "90da2e53-91d3-4e77-b21a-45d8226ab9f5") : secret "maas-api-e2e-extmod-a-bd872a-serving-cert" not found
Nov 03 02:20:35.453106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.453087    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90da2e53-91d3-4e77-b21a-45d8226ab9f5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:35.454974 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.454950    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-metrics-certs\") pod \"maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:35.543426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.537148    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c2ck\" (UniqueName: \"kubernetes.io/projected/90da2e53-91d3-4e77-b21a-45d8226ab9f5-kube-api-access-7c2ck\") pod \"maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:35.963360 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.963320    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:35.965828 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:35.965807    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:36.193376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:36.193330    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:36.570157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:36.570071    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"]
Nov 03 02:20:36.576725 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:20:36.576692    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90da2e53_91d3_4e77_b21a_45d8226ab9f5.slice/crio-a570bf2973757104aa790c0747a3065ce733f8e30ff6192dd6829ef781d41323 WatchSource:0}: Error finding container a570bf2973757104aa790c0747a3065ce733f8e30ff6192dd6829ef781d41323: Status 404 returned error can't find the container with id a570bf2973757104aa790c0747a3065ce733f8e30ff6192dd6829ef781d41323
Nov 03 02:20:36.811904 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:36.811865    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h" event={"ID":"90da2e53-91d3-4e77-b21a-45d8226ab9f5","Type":"ContainerStarted","Data":"a570bf2973757104aa790c0747a3065ce733f8e30ff6192dd6829ef781d41323"}
Nov 03 02:20:37.820465 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:37.820430    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h" event={"ID":"90da2e53-91d3-4e77-b21a-45d8226ab9f5","Type":"ContainerStarted","Data":"f3e187d09c66fd8ae403277d4e0eecba9f13caded2e443ea615fdb90d3c1a704"}
Nov 03 02:20:37.820875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:37.820599    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:37.948029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:37.947901    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h" podStartSLOduration=1.8592464180000001 podStartE2EDuration="2.947884743s" totalImagesPullingTime="1.088638325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:20:37.947286568 +0000 UTC m=+3717.372893058" watchObservedRunningTime="2026-11-03 02:20:37.947884743 +0000 UTC m=+3717.373491207"
Nov 03 02:20:38.044973 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:38.044937    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"]
Nov 03 02:20:38.048239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:38.048213    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28d39226-d46jf"]
Nov 03 02:20:38.693656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:38.693622    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08feef3-9db7-4c18-8c3b-075d6ee44667" path="/var/lib/kubelet/pods/e08feef3-9db7-4c18-8c3b-075d6ee44667/volumes"
Nov 03 02:20:38.874841 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:38.874794    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w" podUID="3a7116ab-aa11-4085-93c1-3d5086a796de" containerName="payload-processing" containerID="cri-o://1c79c08354a37e7cbe3f584ec29a968f1c507f4924f80d88ede6a7fd746f412f" gracePeriod=25
Nov 03 02:20:39.235912 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.235875    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"
Nov 03 02:20:39.314344 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.314256    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7116ab-aa11-4085-93c1-3d5086a796de-kube-api-access-cs457\" (UniqueName: \"kubernetes.io/projected/3a7116ab-aa11-4085-93c1-3d5086a796de-kube-api-access-cs457\") pod \"3a7116ab-aa11-4085-93c1-3d5086a796de\" (UID: \"3a7116ab-aa11-4085-93c1-3d5086a796de\") "
Nov 03 02:20:39.314543 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.314396    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a7116ab-aa11-4085-93c1-3d5086a796de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7116ab-aa11-4085-93c1-3d5086a796de-plugins-config-volume\") pod \"3a7116ab-aa11-4085-93c1-3d5086a796de\" (UID: \"3a7116ab-aa11-4085-93c1-3d5086a796de\") "
Nov 03 02:20:39.314781 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.314760    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7116ab-aa11-4085-93c1-3d5086a796de-plugins-config-volume" pod "3a7116ab-aa11-4085-93c1-3d5086a796de" (UID: "3a7116ab-aa11-4085-93c1-3d5086a796de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:20:39.314987 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.314969    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a7116ab-aa11-4085-93c1-3d5086a796de-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:20:39.316586 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.316565    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7116ab-aa11-4085-93c1-3d5086a796de-kube-api-access-cs457" pod "3a7116ab-aa11-4085-93c1-3d5086a796de" (UID: "3a7116ab-aa11-4085-93c1-3d5086a796de"). InnerVolumeSpecName "kube-api-access-cs457". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:20:39.416395 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.416341    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs457\" (UniqueName: \"kubernetes.io/projected/3a7116ab-aa11-4085-93c1-3d5086a796de-kube-api-access-cs457\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:20:39.835647 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.835610    5508 generic.go:356] "Generic (PLEG): container finished" podID="3a7116ab-aa11-4085-93c1-3d5086a796de" containerID="1c79c08354a37e7cbe3f584ec29a968f1c507f4924f80d88ede6a7fd746f412f" exitCode=0
Nov 03 02:20:39.835835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.835676    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"
Nov 03 02:20:39.835835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.835698    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w" event={"ID":"3a7116ab-aa11-4085-93c1-3d5086a796de","Type":"ContainerDied","Data":"1c79c08354a37e7cbe3f584ec29a968f1c507f4924f80d88ede6a7fd746f412f"}
Nov 03 02:20:39.835835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.835760    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w" event={"ID":"3a7116ab-aa11-4085-93c1-3d5086a796de","Type":"ContainerDied","Data":"2b6da2a4a95b30a1f9c0d30b94552b6964225540cf6ad9f974c59b45c181e20d"}
Nov 03 02:20:39.835835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.835785    5508 scope.go:122] "RemoveContainer" containerID="1c79c08354a37e7cbe3f584ec29a968f1c507f4924f80d88ede6a7fd746f412f"
Nov 03 02:20:39.846229 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.846208    5508 scope.go:122] "RemoveContainer" containerID="1c79c08354a37e7cbe3f584ec29a968f1c507f4924f80d88ede6a7fd746f412f"
Nov 03 02:20:39.846563 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:20:39.846541    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c79c08354a37e7cbe3f584ec29a968f1c507f4924f80d88ede6a7fd746f412f\": container with ID starting with 1c79c08354a37e7cbe3f584ec29a968f1c507f4924f80d88ede6a7fd746f412f not found: ID does not exist" containerID="1c79c08354a37e7cbe3f584ec29a968f1c507f4924f80d88ede6a7fd746f412f"
Nov 03 02:20:39.846637 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.846572    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c79c08354a37e7cbe3f584ec29a968f1c507f4924f80d88ede6a7fd746f412f"} err="failed to get container status \"1c79c08354a37e7cbe3f584ec29a968f1c507f4924f80d88ede6a7fd746f412f\": rpc error: code = NotFound desc = could not find container \"1c79c08354a37e7cbe3f584ec29a968f1c507f4924f80d88ede6a7fd746f412f\": container with ID starting with 1c79c08354a37e7cbe3f584ec29a968f1c507f4924f80d88ede6a7fd746f412f not found: ID does not exist"
Nov 03 02:20:39.993900 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:39.993864    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"]
Nov 03 02:20:40.041386 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:40.041322    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c70365-6ddb79779f-dpn9w"]
Nov 03 02:20:40.693558 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:40.693528    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a7116ab-aa11-4085-93c1-3d5086a796de" path="/var/lib/kubelet/pods/3a7116ab-aa11-4085-93c1-3d5086a796de/volumes"
Nov 03 02:20:43.811883 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:20:43.811843    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:20:43.832338 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:43.832314    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:20:45.584383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.584330    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"]
Nov 03 02:20:45.585299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.585275    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3a7116ab-aa11-4085-93c1-3d5086a796de" containerName="payload-processing"
Nov 03 02:20:45.585299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.585299    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7116ab-aa11-4085-93c1-3d5086a796de" containerName="payload-processing"
Nov 03 02:20:45.585523 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.585497    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7116ab-aa11-4085-93c1-3d5086a796de" containerName="payload-processing"
Nov 03 02:20:45.588831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.588810    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:45.648010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.647976    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-3fb07e-serving-cert\""
Nov 03 02:20:45.694405 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.694112    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-metrics-certs\") pod \"maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:45.694405 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.694228    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdn57\" (UniqueName: \"kubernetes.io/projected/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-kube-api-access-jdn57\") pod \"maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:45.694405 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.694295    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:45.694405 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.694329    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:45.698414 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.698380    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"]
Nov 03 02:20:45.796178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.796137    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-metrics-certs\") pod \"maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:45.796431 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.796253    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdn57\" (UniqueName: \"kubernetes.io/projected/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-kube-api-access-jdn57\") pod \"maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:45.796431 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.796334    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:45.796431 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.796398    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:45.796981 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:20:45.796661    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-serving-cert: secret "maas-api-e2e-extmod-b-3fb07e-serving-cert" not found
Nov 03 02:20:45.796981 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:20:45.796757    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-maas-api-tls podName:beb90e6f-497c-4dc4-80c1-05aa76a82dbd nodeName:}" failed. No retries permitted until 2026-11-03 02:20:46.29673678 +0000 UTC m=+3725.722343225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-maas-api-tls") pod "maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w" (UID: "beb90e6f-497c-4dc4-80c1-05aa76a82dbd") : secret "maas-api-e2e-extmod-b-3fb07e-serving-cert" not found
Nov 03 02:20:45.797190 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.797079    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:45.799015 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.798990    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-metrics-certs\") pod \"maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:45.856653 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:45.856563    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdn57\" (UniqueName: \"kubernetes.io/projected/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-kube-api-access-jdn57\") pod \"maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:46.310387 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:46.310340    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:46.312883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:46.312856    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:46.502711 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:46.502672    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:46.880053 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:46.879967    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"]
Nov 03 02:20:46.886659 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:20:46.886630    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb90e6f_497c_4dc4_80c1_05aa76a82dbd.slice/crio-6c8b64a733790a6bcb7efaa5bc8de74477c1eb42a0d15b23ccaa73b400bacb6f WatchSource:0}: Error finding container 6c8b64a733790a6bcb7efaa5bc8de74477c1eb42a0d15b23ccaa73b400bacb6f: Status 404 returned error can't find the container with id 6c8b64a733790a6bcb7efaa5bc8de74477c1eb42a0d15b23ccaa73b400bacb6f
Nov 03 02:20:46.913872 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:46.913841    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w" event={"ID":"beb90e6f-497c-4dc4-80c1-05aa76a82dbd","Type":"ContainerStarted","Data":"6c8b64a733790a6bcb7efaa5bc8de74477c1eb42a0d15b23ccaa73b400bacb6f"}
Nov 03 02:20:47.922144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:47.922106    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w" event={"ID":"beb90e6f-497c-4dc4-80c1-05aa76a82dbd","Type":"ContainerStarted","Data":"51aaa1c0c67c5930a92594766c1c520441b01a73be168555210c5094bac2c64a"}
Nov 03 02:20:47.922554 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:47.922182    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:48.034335 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:48.034229    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w" podStartSLOduration=2.186639977 podStartE2EDuration="3.034215529s" totalImagesPullingTime="847.575552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:20:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:20:48.031558456 +0000 UTC m=+3727.457164920" watchObservedRunningTime="2026-11-03 02:20:48.034215529 +0000 UTC m=+3727.459821992"
Nov 03 02:20:53.937678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:53.937648    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:20:56.545202 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:20:56.545169    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:21:08.086705 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:08.086670    5508 generic.go:356] "Generic (PLEG): container finished" podID="438cacfe-f668-4897-992c-1faa9b411850" containerID="b6712fd1a15cf9020e702244c11d10f3d0f815053d7d727cf5490b5fcdfddb36" exitCode=0
Nov 03 02:21:08.087277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:08.086738    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" event={"ID":"438cacfe-f668-4897-992c-1faa9b411850","Type":"ContainerDied","Data":"b6712fd1a15cf9020e702244c11d10f3d0f815053d7d727cf5490b5fcdfddb36"}
Nov 03 02:21:08.087277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:08.087209    5508 scope.go:122] "RemoveContainer" containerID="b6712fd1a15cf9020e702244c11d10f3d0f815053d7d727cf5490b5fcdfddb36"
Nov 03 02:21:08.088931 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:08.088896    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:21:08.089034 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:08.088971    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:21:08.089034 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:08.089000    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:21:09.095978 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:09.095941    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" event={"ID":"438cacfe-f668-4897-992c-1faa9b411850","Type":"ContainerStarted","Data":"eed0378bdc45c065bd5814d01a3b2fbf04b63c4ae2951ec2a1bfd0c86dc97508"}
Nov 03 02:21:09.312862 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:09.312826    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:21:09.314191 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:09.314164    5508 patch_prober.go:29] interesting pod/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 03 02:21:09.314381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:09.314217    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" podUID="438cacfe-f668-4897-992c-1faa9b411850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 03 02:21:10.314213 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:10.314169    5508 patch_prober.go:29] interesting pod/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 03 02:21:10.314655 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:10.314224    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" podUID="438cacfe-f668-4897-992c-1faa9b411850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 03 02:21:11.313568 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:11.313533    5508 patch_prober.go:29] interesting pod/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 03 02:21:11.313766 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:11.313591    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" podUID="438cacfe-f668-4897-992c-1faa9b411850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 03 02:21:12.312958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:12.312921    5508 patch_prober.go:29] interesting pod/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 03 02:21:12.313403 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:12.312980    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" podUID="438cacfe-f668-4897-992c-1faa9b411850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 03 02:21:13.313516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:13.313478    5508 patch_prober.go:29] interesting pod/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 03 02:21:13.313954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:13.313544    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" podUID="438cacfe-f668-4897-992c-1faa9b411850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 03 02:21:13.812738 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:21:13.812697    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:21:14.022988 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:14.022917    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:21:14.313560 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:14.313528    5508 patch_prober.go:29] interesting pod/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused" start-of-body=
Nov 03 02:21:14.314037 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:14.313582    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" podUID="438cacfe-f668-4897-992c-1faa9b411850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.51:15021/healthz/ready\": dial tcp 10.133.1.51:15021: connect: connection refused"
Nov 03 02:21:15.318565 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:15.318534    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:21:15.319830 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:15.319810    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:21:30.625969 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:30.625930    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:21:43.812688 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:21:43.812644    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:21:45.959672 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:45.959585    5508 scope.go:122] "RemoveContainer" containerID="4727eff282d10d10c5b62c03454c5f95c71f66a3fe7b5a1c5ab60bc776c905a2"
Nov 03 02:21:45.970782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:45.970759    5508 scope.go:122] "RemoveContainer" containerID="db391f1d3f1d5e5015a5a3d055b3a2bbc870aadfec7be38b4443d42bafb839eb"
Nov 03 02:21:58.291777 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:58.291737    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"]
Nov 03 02:21:58.292268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:58.292045    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" podUID="320b0833-f978-4d66-9d40-8693e461f63b" containerName="istio-proxy" containerID="cri-o://6e6fcba7e2380576729c5fad2e80d64337cbd1dc0a0f316e4d157f5cd0fb3328" gracePeriod=30
Nov 03 02:21:58.703420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:58.703381    5508 patch_prober.go:29] interesting pod/e2e-extmod-a-c70365-openshift-default-5674757df-ng472 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:21:58.703611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:21:58.703448    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" podUID="320b0833-f978-4d66-9d40-8693e461f63b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:22:03.526034 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.525999    5508 generic.go:356] "Generic (PLEG): container finished" podID="320b0833-f978-4d66-9d40-8693e461f63b" containerID="6e6fcba7e2380576729c5fad2e80d64337cbd1dc0a0f316e4d157f5cd0fb3328" exitCode=0
Nov 03 02:22:03.526460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.526058    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" event={"ID":"320b0833-f978-4d66-9d40-8693e461f63b","Type":"ContainerDied","Data":"6e6fcba7e2380576729c5fad2e80d64337cbd1dc0a0f316e4d157f5cd0fb3328"}
Nov 03 02:22:03.526460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.526113    5508 scope.go:122] "RemoveContainer" containerID="f3193bca4642683df7d464ac01275425c055ecece98424cb47e1c1818df965b1"
Nov 03 02:22:03.582505 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.582435    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:22:03.678872 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.678826    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-credential-socket\") pod \"320b0833-f978-4d66-9d40-8693e461f63b\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") "
Nov 03 02:22:03.679099 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.678944    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/320b0833-f978-4d66-9d40-8693e461f63b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320b0833-f978-4d66-9d40-8693e461f63b-istiod-ca-cert\") pod \"320b0833-f978-4d66-9d40-8693e461f63b\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") "
Nov 03 02:22:03.679099 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.678972    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-istio-token\" (UniqueName: \"kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-istio-token\") pod \"320b0833-f978-4d66-9d40-8693e461f63b\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") "
Nov 03 02:22:03.679099 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679029    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-data\") pod \"320b0833-f978-4d66-9d40-8693e461f63b\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") "
Nov 03 02:22:03.679099 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679083    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-certs\") pod \"320b0833-f978-4d66-9d40-8693e461f63b\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") "
Nov 03 02:22:03.679379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679147    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-kube-api-access-k4pms\" (UniqueName: \"kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-kube-api-access-k4pms\") pod \"320b0833-f978-4d66-9d40-8693e461f63b\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") "
Nov 03 02:22:03.679379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679189    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-envoy\") pod \"320b0833-f978-4d66-9d40-8693e461f63b\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") "
Nov 03 02:22:03.679379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679210    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-credential-socket" pod "320b0833-f978-4d66-9d40-8693e461f63b" (UID: "320b0833-f978-4d66-9d40-8693e461f63b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:22:03.679379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679255    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-socket\") pod \"320b0833-f978-4d66-9d40-8693e461f63b\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") "
Nov 03 02:22:03.679379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679300    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/320b0833-f978-4d66-9d40-8693e461f63b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320b0833-f978-4d66-9d40-8693e461f63b-istio-podinfo\") pod \"320b0833-f978-4d66-9d40-8693e461f63b\" (UID: \"320b0833-f978-4d66-9d40-8693e461f63b\") "
Nov 03 02:22:03.679688 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679454    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320b0833-f978-4d66-9d40-8693e461f63b-istiod-ca-cert" pod "320b0833-f978-4d66-9d40-8693e461f63b" (UID: "320b0833-f978-4d66-9d40-8693e461f63b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:22:03.679688 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679459    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-certs" pod "320b0833-f978-4d66-9d40-8693e461f63b" (UID: "320b0833-f978-4d66-9d40-8693e461f63b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:22:03.679688 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679640    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-data" pod "320b0833-f978-4d66-9d40-8693e461f63b" (UID: "320b0833-f978-4d66-9d40-8693e461f63b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:22:03.679902 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679883    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320b0833-f978-4d66-9d40-8693e461f63b-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:03.679963 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679907    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:03.679963 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679922    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:03.680077 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679969    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:03.680077 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.679986    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-socket" pod "320b0833-f978-4d66-9d40-8693e461f63b" (UID: "320b0833-f978-4d66-9d40-8693e461f63b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:22:03.685049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.685020    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-istio-token" pod "320b0833-f978-4d66-9d40-8693e461f63b" (UID: "320b0833-f978-4d66-9d40-8693e461f63b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:22:03.685513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.685482    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-envoy" pod "320b0833-f978-4d66-9d40-8693e461f63b" (UID: "320b0833-f978-4d66-9d40-8693e461f63b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:22:03.685792 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.685770    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/320b0833-f978-4d66-9d40-8693e461f63b-istio-podinfo" pod "320b0833-f978-4d66-9d40-8693e461f63b" (UID: "320b0833-f978-4d66-9d40-8693e461f63b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:22:03.688538 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.688515    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-kube-api-access-k4pms" pod "320b0833-f978-4d66-9d40-8693e461f63b" (UID: "320b0833-f978-4d66-9d40-8693e461f63b"). InnerVolumeSpecName "kube-api-access-k4pms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:22:03.781045 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.781009    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4pms\" (UniqueName: \"kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-kube-api-access-k4pms\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:03.781045 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.781047    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:03.781271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.781061    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320b0833-f978-4d66-9d40-8693e461f63b-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:03.781271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.781073    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320b0833-f978-4d66-9d40-8693e461f63b-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:03.781271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:03.781086    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320b0833-f978-4d66-9d40-8693e461f63b-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:04.534095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:04.534057    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472" event={"ID":"320b0833-f978-4d66-9d40-8693e461f63b","Type":"ContainerDied","Data":"7760f17bfaf73ff5fb57091a77225b3f4f1353f238f752738006fe886fca8f5a"}
Nov 03 02:22:04.534095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:04.534095    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"
Nov 03 02:22:04.534614 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:04.534120    5508 scope.go:122] "RemoveContainer" containerID="6e6fcba7e2380576729c5fad2e80d64337cbd1dc0a0f316e4d157f5cd0fb3328"
Nov 03 02:22:04.735912 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:04.735877    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"]
Nov 03 02:22:04.782026 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:04.781985    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c70365-openshift-default-5674757df-ng472"]
Nov 03 02:22:05.179758 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:05.179718    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"]
Nov 03 02:22:05.294005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:05.293973    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"]
Nov 03 02:22:06.693558 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:06.693522    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320b0833-f978-4d66-9d40-8693e461f63b" path="/var/lib/kubelet/pods/320b0833-f978-4d66-9d40-8693e461f63b/volumes"
Nov 03 02:22:07.462342 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:07.462304    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"]
Nov 03 02:22:10.180674 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.180622    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms" podUID="ef811d84-4691-43f7-9d23-8033ed536126" containerName="payload-processing" containerID="cri-o://a1a7947996e8f6f1630989ae71cfe830cb6a8efdf4199ed13ff941629e39c2b1" gracePeriod=25
Nov 03 02:22:10.294443 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.294224    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2" podUID="992f07df-b113-42f5-9e9c-e3864e3d9190" containerName="payload-pre-processing" containerID="cri-o://cac5e77c21d5be3a9111280c49351485a6985b3cc26b0ed74e8d59353bd0e93d" gracePeriod=25
Nov 03 02:22:10.450460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.450392    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"
Nov 03 02:22:10.591518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.591153    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef811d84-4691-43f7-9d23-8033ed536126-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef811d84-4691-43f7-9d23-8033ed536126-plugins-config-volume\") pod \"ef811d84-4691-43f7-9d23-8033ed536126\" (UID: \"ef811d84-4691-43f7-9d23-8033ed536126\") "
Nov 03 02:22:10.591518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.591391    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef811d84-4691-43f7-9d23-8033ed536126-kube-api-access-96528\" (UniqueName: \"kubernetes.io/projected/ef811d84-4691-43f7-9d23-8033ed536126-kube-api-access-96528\") pod \"ef811d84-4691-43f7-9d23-8033ed536126\" (UID: \"ef811d84-4691-43f7-9d23-8033ed536126\") "
Nov 03 02:22:10.591755 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.591610    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef811d84-4691-43f7-9d23-8033ed536126-plugins-config-volume" pod "ef811d84-4691-43f7-9d23-8033ed536126" (UID: "ef811d84-4691-43f7-9d23-8033ed536126"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:22:10.592024 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.591999    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef811d84-4691-43f7-9d23-8033ed536126-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:10.594259 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.594234    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef811d84-4691-43f7-9d23-8033ed536126-kube-api-access-96528" pod "ef811d84-4691-43f7-9d23-8033ed536126" (UID: "ef811d84-4691-43f7-9d23-8033ed536126"). InnerVolumeSpecName "kube-api-access-96528". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:22:10.596045 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.596023    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"
Nov 03 02:22:10.631698 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.631662    5508 generic.go:356] "Generic (PLEG): container finished" podID="992f07df-b113-42f5-9e9c-e3864e3d9190" containerID="cac5e77c21d5be3a9111280c49351485a6985b3cc26b0ed74e8d59353bd0e93d" exitCode=0
Nov 03 02:22:10.631864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.631718    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"
Nov 03 02:22:10.631864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.631741    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2" event={"ID":"992f07df-b113-42f5-9e9c-e3864e3d9190","Type":"ContainerDied","Data":"cac5e77c21d5be3a9111280c49351485a6985b3cc26b0ed74e8d59353bd0e93d"}
Nov 03 02:22:10.631864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.631787    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2" event={"ID":"992f07df-b113-42f5-9e9c-e3864e3d9190","Type":"ContainerDied","Data":"912aef4d38331857e4e94e681089f687544fe20796ae31f316fb7dd09a2e4f08"}
Nov 03 02:22:10.631864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.631810    5508 scope.go:122] "RemoveContainer" containerID="cac5e77c21d5be3a9111280c49351485a6985b3cc26b0ed74e8d59353bd0e93d"
Nov 03 02:22:10.633263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.633223    5508 generic.go:356] "Generic (PLEG): container finished" podID="ef811d84-4691-43f7-9d23-8033ed536126" containerID="a1a7947996e8f6f1630989ae71cfe830cb6a8efdf4199ed13ff941629e39c2b1" exitCode=0
Nov 03 02:22:10.633368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.633260    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms" event={"ID":"ef811d84-4691-43f7-9d23-8033ed536126","Type":"ContainerDied","Data":"a1a7947996e8f6f1630989ae71cfe830cb6a8efdf4199ed13ff941629e39c2b1"}
Nov 03 02:22:10.633368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.633278    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"
Nov 03 02:22:10.633368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.633309    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms" event={"ID":"ef811d84-4691-43f7-9d23-8033ed536126","Type":"ContainerDied","Data":"0b5918187e2021fe52a5d73e61e3cace5deec71d0613e62d65e203ea69431912"}
Nov 03 02:22:10.642000 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.641930    5508 scope.go:122] "RemoveContainer" containerID="cac5e77c21d5be3a9111280c49351485a6985b3cc26b0ed74e8d59353bd0e93d"
Nov 03 02:22:10.642242 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:22:10.642218    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac5e77c21d5be3a9111280c49351485a6985b3cc26b0ed74e8d59353bd0e93d\": container with ID starting with cac5e77c21d5be3a9111280c49351485a6985b3cc26b0ed74e8d59353bd0e93d not found: ID does not exist" containerID="cac5e77c21d5be3a9111280c49351485a6985b3cc26b0ed74e8d59353bd0e93d"
Nov 03 02:22:10.642298 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.642252    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac5e77c21d5be3a9111280c49351485a6985b3cc26b0ed74e8d59353bd0e93d"} err="failed to get container status \"cac5e77c21d5be3a9111280c49351485a6985b3cc26b0ed74e8d59353bd0e93d\": rpc error: code = NotFound desc = could not find container \"cac5e77c21d5be3a9111280c49351485a6985b3cc26b0ed74e8d59353bd0e93d\": container with ID starting with cac5e77c21d5be3a9111280c49351485a6985b3cc26b0ed74e8d59353bd0e93d not found: ID does not exist"
Nov 03 02:22:10.642298 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.642267    5508 scope.go:122] "RemoveContainer" containerID="a1a7947996e8f6f1630989ae71cfe830cb6a8efdf4199ed13ff941629e39c2b1"
Nov 03 02:22:10.653098 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.653075    5508 scope.go:122] "RemoveContainer" containerID="a1a7947996e8f6f1630989ae71cfe830cb6a8efdf4199ed13ff941629e39c2b1"
Nov 03 02:22:10.653418 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:22:10.653399    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a7947996e8f6f1630989ae71cfe830cb6a8efdf4199ed13ff941629e39c2b1\": container with ID starting with a1a7947996e8f6f1630989ae71cfe830cb6a8efdf4199ed13ff941629e39c2b1 not found: ID does not exist" containerID="a1a7947996e8f6f1630989ae71cfe830cb6a8efdf4199ed13ff941629e39c2b1"
Nov 03 02:22:10.653489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.653429    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a7947996e8f6f1630989ae71cfe830cb6a8efdf4199ed13ff941629e39c2b1"} err="failed to get container status \"a1a7947996e8f6f1630989ae71cfe830cb6a8efdf4199ed13ff941629e39c2b1\": rpc error: code = NotFound desc = could not find container \"a1a7947996e8f6f1630989ae71cfe830cb6a8efdf4199ed13ff941629e39c2b1\": container with ID starting with a1a7947996e8f6f1630989ae71cfe830cb6a8efdf4199ed13ff941629e39c2b1 not found: ID does not exist"
Nov 03 02:22:10.692619 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.692592    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992f07df-b113-42f5-9e9c-e3864e3d9190-kube-api-access-4zm5p\" (UniqueName: \"kubernetes.io/projected/992f07df-b113-42f5-9e9c-e3864e3d9190-kube-api-access-4zm5p\") pod \"992f07df-b113-42f5-9e9c-e3864e3d9190\" (UID: \"992f07df-b113-42f5-9e9c-e3864e3d9190\") "
Nov 03 02:22:10.692799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.692777    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992f07df-b113-42f5-9e9c-e3864e3d9190-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992f07df-b113-42f5-9e9c-e3864e3d9190-plugins-config-volume\") pod \"992f07df-b113-42f5-9e9c-e3864e3d9190\" (UID: \"992f07df-b113-42f5-9e9c-e3864e3d9190\") "
Nov 03 02:22:10.693182 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.693152    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992f07df-b113-42f5-9e9c-e3864e3d9190-plugins-config-volume" pod "992f07df-b113-42f5-9e9c-e3864e3d9190" (UID: "992f07df-b113-42f5-9e9c-e3864e3d9190"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:22:10.693289 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.693220    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96528\" (UniqueName: \"kubernetes.io/projected/ef811d84-4691-43f7-9d23-8033ed536126-kube-api-access-96528\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:10.694776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.694759    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992f07df-b113-42f5-9e9c-e3864e3d9190-kube-api-access-4zm5p" pod "992f07df-b113-42f5-9e9c-e3864e3d9190" (UID: "992f07df-b113-42f5-9e9c-e3864e3d9190"). InnerVolumeSpecName "kube-api-access-4zm5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:22:10.794995 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.794951    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/992f07df-b113-42f5-9e9c-e3864e3d9190-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:10.794995 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.794993    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zm5p\" (UniqueName: \"kubernetes.io/projected/992f07df-b113-42f5-9e9c-e3864e3d9190-kube-api-access-4zm5p\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:10.810436 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.810399    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"]
Nov 03 02:22:10.814827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:10.814801    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4f87d1-599556f77c-98qms"]
Nov 03 02:22:11.039409 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:11.039312    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"]
Nov 03 02:22:11.043859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:11.043834    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-4f87d1-65b66d649f-b48k2"]
Nov 03 02:22:11.509574 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:11.509540    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:22:12.462720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:12.462664    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6" podUID="1ba7f2d3-aebd-4c59-8b12-193c8e88df5d" containerName="payload-processing" containerID="cri-o://960de9e1c1453c4bbb7b4cf968380ef84a3978a6620949968fe84c38477353ac" gracePeriod=25
Nov 03 02:22:12.652481 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:12.652447    5508 generic.go:356] "Generic (PLEG): container finished" podID="1ba7f2d3-aebd-4c59-8b12-193c8e88df5d" containerID="960de9e1c1453c4bbb7b4cf968380ef84a3978a6620949968fe84c38477353ac" exitCode=0
Nov 03 02:22:12.652982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:12.652616    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6" event={"ID":"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d","Type":"ContainerDied","Data":"960de9e1c1453c4bbb7b4cf968380ef84a3978a6620949968fe84c38477353ac"}
Nov 03 02:22:12.693017 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:12.692976    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992f07df-b113-42f5-9e9c-e3864e3d9190" path="/var/lib/kubelet/pods/992f07df-b113-42f5-9e9c-e3864e3d9190/volumes"
Nov 03 02:22:12.693554 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:12.693532    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef811d84-4691-43f7-9d23-8033ed536126" path="/var/lib/kubelet/pods/ef811d84-4691-43f7-9d23-8033ed536126/volumes"
Nov 03 02:22:12.749359 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:12.749327    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"
Nov 03 02:22:12.819246 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:12.819215    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-kube-api-access-mksmt\" (UniqueName: \"kubernetes.io/projected/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-kube-api-access-mksmt\") pod \"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d\" (UID: \"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d\") "
Nov 03 02:22:12.819455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:12.819275    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-plugins-config-volume\") pod \"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d\" (UID: \"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d\") "
Nov 03 02:22:12.819716 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:12.819683    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-plugins-config-volume" pod "1ba7f2d3-aebd-4c59-8b12-193c8e88df5d" (UID: "1ba7f2d3-aebd-4c59-8b12-193c8e88df5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:22:12.821380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:12.821359    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-kube-api-access-mksmt" pod "1ba7f2d3-aebd-4c59-8b12-193c8e88df5d" (UID: "1ba7f2d3-aebd-4c59-8b12-193c8e88df5d"). InnerVolumeSpecName "kube-api-access-mksmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:22:12.920904 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:12.920863    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mksmt\" (UniqueName: \"kubernetes.io/projected/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-kube-api-access-mksmt\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:12.920904 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:12.920896    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:22:13.660095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:13.660062    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"
Nov 03 02:22:13.660095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:13.660091    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6" event={"ID":"1ba7f2d3-aebd-4c59-8b12-193c8e88df5d","Type":"ContainerDied","Data":"a6cbedff402e0c6e370c417600784acbb25889a0226c92779db1a858132cdc68"}
Nov 03 02:22:13.660653 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:13.660156    5508 scope.go:122] "RemoveContainer" containerID="960de9e1c1453c4bbb7b4cf968380ef84a3978a6620949968fe84c38477353ac"
Nov 03 02:22:13.815097 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:13.815064    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"]
Nov 03 02:22:13.815677 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:22:13.815649    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:22:13.860866 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:13.860833    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bd872a-7bdf9cfc9-hrpn6"]
Nov 03 02:22:14.694172 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:22:14.694138    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba7f2d3-aebd-4c59-8b12-193c8e88df5d" path="/var/lib/kubelet/pods/1ba7f2d3-aebd-4c59-8b12-193c8e88df5d/volumes"
Nov 03 02:22:43.833910 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:22:43.833862    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:23:13.813074 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:13.813034    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:23:18.662312 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:18.662213    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:23:25.235537 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.235496    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"]
Nov 03 02:23:25.236021 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.235732    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd" podUID="82e5189f-18d7-405d-a067-ded6fa319c64" containerName="maas-api" containerID="cri-o://56a29572815a3a7ec82264fa54341ea8cdbdaf1cebc7f07d80fe5ce1f4b1b483" gracePeriod=30
Nov 03 02:23:25.534858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.534824    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:23:25.622335 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.622296    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e5189f-18d7-405d-a067-ded6fa319c64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e5189f-18d7-405d-a067-ded6fa319c64-openshift-service-ca\") pod \"82e5189f-18d7-405d-a067-ded6fa319c64\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") "
Nov 03 02:23:25.633529 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.633492    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e5189f-18d7-405d-a067-ded6fa319c64-kube-api-access-kx4vt\" (UniqueName: \"kubernetes.io/projected/82e5189f-18d7-405d-a067-ded6fa319c64-kube-api-access-kx4vt\") pod \"82e5189f-18d7-405d-a067-ded6fa319c64\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") "
Nov 03 02:23:25.633971 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.633944    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-maas-api-tls\") pod \"82e5189f-18d7-405d-a067-ded6fa319c64\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") "
Nov 03 02:23:25.634362 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.634326    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-metrics-certs\") pod \"82e5189f-18d7-405d-a067-ded6fa319c64\" (UID: \"82e5189f-18d7-405d-a067-ded6fa319c64\") "
Nov 03 02:23:25.644798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.629780    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e5189f-18d7-405d-a067-ded6fa319c64-openshift-service-ca" pod "82e5189f-18d7-405d-a067-ded6fa319c64" (UID: "82e5189f-18d7-405d-a067-ded6fa319c64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:23:25.674112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.649591    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-maas-api-tls" pod "82e5189f-18d7-405d-a067-ded6fa319c64" (UID: "82e5189f-18d7-405d-a067-ded6fa319c64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:23:25.674112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.651776    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-metrics-certs" pod "82e5189f-18d7-405d-a067-ded6fa319c64" (UID: "82e5189f-18d7-405d-a067-ded6fa319c64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:23:25.674112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.657485    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e5189f-18d7-405d-a067-ded6fa319c64-kube-api-access-kx4vt" pod "82e5189f-18d7-405d-a067-ded6fa319c64" (UID: "82e5189f-18d7-405d-a067-ded6fa319c64"). InnerVolumeSpecName "kube-api-access-kx4vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:23:25.736806 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.736768    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:25.736806 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.736806    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82e5189f-18d7-405d-a067-ded6fa319c64-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:25.737022 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.736825    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82e5189f-18d7-405d-a067-ded6fa319c64-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:25.737022 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:25.736842    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx4vt\" (UniqueName: \"kubernetes.io/projected/82e5189f-18d7-405d-a067-ded6fa319c64-kube-api-access-kx4vt\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:26.228688 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.228652    5508 generic.go:356] "Generic (PLEG): container finished" podID="82e5189f-18d7-405d-a067-ded6fa319c64" containerID="56a29572815a3a7ec82264fa54341ea8cdbdaf1cebc7f07d80fe5ce1f4b1b483" exitCode=0
Nov 03 02:23:26.228891 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.228722    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd" event={"ID":"82e5189f-18d7-405d-a067-ded6fa319c64","Type":"ContainerDied","Data":"56a29572815a3a7ec82264fa54341ea8cdbdaf1cebc7f07d80fe5ce1f4b1b483"}
Nov 03 02:23:26.228891 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.228726    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"
Nov 03 02:23:26.228891 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.228765    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd" event={"ID":"82e5189f-18d7-405d-a067-ded6fa319c64","Type":"ContainerDied","Data":"a4b703f15769190fa2672cd5ce5ba2ea994ed0ac66c150fbf08c0e5fdb6fb77d"}
Nov 03 02:23:26.228891 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.228788    5508 scope.go:122] "RemoveContainer" containerID="56a29572815a3a7ec82264fa54341ea8cdbdaf1cebc7f07d80fe5ce1f4b1b483"
Nov 03 02:23:26.246132 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.245963    5508 scope.go:122] "RemoveContainer" containerID="56a29572815a3a7ec82264fa54341ea8cdbdaf1cebc7f07d80fe5ce1f4b1b483"
Nov 03 02:23:26.246446 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:26.246307    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a29572815a3a7ec82264fa54341ea8cdbdaf1cebc7f07d80fe5ce1f4b1b483\": container with ID starting with 56a29572815a3a7ec82264fa54341ea8cdbdaf1cebc7f07d80fe5ce1f4b1b483 not found: ID does not exist" containerID="56a29572815a3a7ec82264fa54341ea8cdbdaf1cebc7f07d80fe5ce1f4b1b483"
Nov 03 02:23:26.246446 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.246360    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a29572815a3a7ec82264fa54341ea8cdbdaf1cebc7f07d80fe5ce1f4b1b483"} err="failed to get container status \"56a29572815a3a7ec82264fa54341ea8cdbdaf1cebc7f07d80fe5ce1f4b1b483\": rpc error: code = NotFound desc = could not find container \"56a29572815a3a7ec82264fa54341ea8cdbdaf1cebc7f07d80fe5ce1f4b1b483\": container with ID starting with 56a29572815a3a7ec82264fa54341ea8cdbdaf1cebc7f07d80fe5ce1f4b1b483 not found: ID does not exist"
Nov 03 02:23:26.334085 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.334046    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"]
Nov 03 02:23:26.334290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.334267    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg" podUID="9116085a-5fd8-41cd-9f51-5b5f3e1474c9" containerName="maas-api" containerID="cri-o://5d68efc78df5907f0f2d8bb412c23bc014155c63716c535373429945853eed89" gracePeriod=30
Nov 03 02:23:26.402028 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.401984    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"]
Nov 03 02:23:26.405110 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.405075    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-aab75e-57f6cd79d9-zdlxd"]
Nov 03 02:23:26.649890 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.649860    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:23:26.693477 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.693444    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e5189f-18d7-405d-a067-ded6fa319c64" path="/var/lib/kubelet/pods/82e5189f-18d7-405d-a067-ded6fa319c64/volumes"
Nov 03 02:23:26.754537 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.754505    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-openshift-service-ca\") pod \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") "
Nov 03 02:23:26.754767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.754588    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-kube-api-access-jr8xd\" (UniqueName: \"kubernetes.io/projected/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-kube-api-access-jr8xd\") pod \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") "
Nov 03 02:23:26.754767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.754719    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-maas-api-tls\") pod \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") "
Nov 03 02:23:26.754907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.754892    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-metrics-certs\") pod \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\" (UID: \"9116085a-5fd8-41cd-9f51-5b5f3e1474c9\") "
Nov 03 02:23:26.754965 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.754929    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-openshift-service-ca" pod "9116085a-5fd8-41cd-9f51-5b5f3e1474c9" (UID: "9116085a-5fd8-41cd-9f51-5b5f3e1474c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:23:26.755288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.755267    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:26.757012 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.756989    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-kube-api-access-jr8xd" pod "9116085a-5fd8-41cd-9f51-5b5f3e1474c9" (UID: "9116085a-5fd8-41cd-9f51-5b5f3e1474c9"). InnerVolumeSpecName "kube-api-access-jr8xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:23:26.757279 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.757243    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-maas-api-tls" pod "9116085a-5fd8-41cd-9f51-5b5f3e1474c9" (UID: "9116085a-5fd8-41cd-9f51-5b5f3e1474c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:23:26.757400 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.757277    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-metrics-certs" pod "9116085a-5fd8-41cd-9f51-5b5f3e1474c9" (UID: "9116085a-5fd8-41cd-9f51-5b5f3e1474c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:23:26.856204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.856166    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:26.856204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.856197    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:26.856204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:26.856212    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr8xd\" (UniqueName: \"kubernetes.io/projected/9116085a-5fd8-41cd-9f51-5b5f3e1474c9-kube-api-access-jr8xd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:27.256980 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.256936    5508 generic.go:356] "Generic (PLEG): container finished" podID="9116085a-5fd8-41cd-9f51-5b5f3e1474c9" containerID="5d68efc78df5907f0f2d8bb412c23bc014155c63716c535373429945853eed89" exitCode=0
Nov 03 02:23:27.257527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.257193    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"
Nov 03 02:23:27.258937 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.258711    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg" event={"ID":"9116085a-5fd8-41cd-9f51-5b5f3e1474c9","Type":"ContainerDied","Data":"5d68efc78df5907f0f2d8bb412c23bc014155c63716c535373429945853eed89"}
Nov 03 02:23:27.258937 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.258777    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg" event={"ID":"9116085a-5fd8-41cd-9f51-5b5f3e1474c9","Type":"ContainerDied","Data":"2596b7bcd67d734f30e5ac04b9f1cdb7c21335ae5190a5e1af58697eedb177e6"}
Nov 03 02:23:27.258937 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.258807    5508 scope.go:122] "RemoveContainer" containerID="5d68efc78df5907f0f2d8bb412c23bc014155c63716c535373429945853eed89"
Nov 03 02:23:27.290134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.290097    5508 scope.go:122] "RemoveContainer" containerID="5d68efc78df5907f0f2d8bb412c23bc014155c63716c535373429945853eed89"
Nov 03 02:23:27.290651 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:27.290608    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d68efc78df5907f0f2d8bb412c23bc014155c63716c535373429945853eed89\": container with ID starting with 5d68efc78df5907f0f2d8bb412c23bc014155c63716c535373429945853eed89 not found: ID does not exist" containerID="5d68efc78df5907f0f2d8bb412c23bc014155c63716c535373429945853eed89"
Nov 03 02:23:27.290768 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.290648    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d68efc78df5907f0f2d8bb412c23bc014155c63716c535373429945853eed89"} err="failed to get container status \"5d68efc78df5907f0f2d8bb412c23bc014155c63716c535373429945853eed89\": rpc error: code = NotFound desc = could not find container \"5d68efc78df5907f0f2d8bb412c23bc014155c63716c535373429945853eed89\": container with ID starting with 5d68efc78df5907f0f2d8bb412c23bc014155c63716c535373429945853eed89 not found: ID does not exist"
Nov 03 02:23:27.414622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.414589    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"]
Nov 03 02:23:27.421523 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.421496    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4f87d1-54f6d5c877-5kbcg"]
Nov 03 02:23:27.428378 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.428339    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"]
Nov 03 02:23:27.428563 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.428542    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h" podUID="90da2e53-91d3-4e77-b21a-45d8226ab9f5" containerName="maas-api" containerID="cri-o://f3e187d09c66fd8ae403277d4e0eecba9f13caded2e443ea615fdb90d3c1a704" gracePeriod=30
Nov 03 02:23:27.708056 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.708029    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:23:27.870988 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.870900    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90da2e53-91d3-4e77-b21a-45d8226ab9f5-kube-api-access-7c2ck\" (UniqueName: \"kubernetes.io/projected/90da2e53-91d3-4e77-b21a-45d8226ab9f5-kube-api-access-7c2ck\") pod \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") "
Nov 03 02:23:27.871166 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.871010    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-metrics-certs\") pod \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") "
Nov 03 02:23:27.871265 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.871185    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-maas-api-tls\") pod \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") "
Nov 03 02:23:27.871265 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.871227    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90da2e53-91d3-4e77-b21a-45d8226ab9f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90da2e53-91d3-4e77-b21a-45d8226ab9f5-openshift-service-ca\") pod \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\" (UID: \"90da2e53-91d3-4e77-b21a-45d8226ab9f5\") "
Nov 03 02:23:27.871654 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.871622    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90da2e53-91d3-4e77-b21a-45d8226ab9f5-openshift-service-ca" pod "90da2e53-91d3-4e77-b21a-45d8226ab9f5" (UID: "90da2e53-91d3-4e77-b21a-45d8226ab9f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:23:27.871796 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.871766    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90da2e53-91d3-4e77-b21a-45d8226ab9f5-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:27.873419 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.873341    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-metrics-certs" pod "90da2e53-91d3-4e77-b21a-45d8226ab9f5" (UID: "90da2e53-91d3-4e77-b21a-45d8226ab9f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:23:27.873738 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.873701    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90da2e53-91d3-4e77-b21a-45d8226ab9f5-kube-api-access-7c2ck" pod "90da2e53-91d3-4e77-b21a-45d8226ab9f5" (UID: "90da2e53-91d3-4e77-b21a-45d8226ab9f5"). InnerVolumeSpecName "kube-api-access-7c2ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:23:27.874055 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.873917    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-maas-api-tls" pod "90da2e53-91d3-4e77-b21a-45d8226ab9f5" (UID: "90da2e53-91d3-4e77-b21a-45d8226ab9f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:23:27.975632 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.975373    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:27.975632 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.975411    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c2ck\" (UniqueName: \"kubernetes.io/projected/90da2e53-91d3-4e77-b21a-45d8226ab9f5-kube-api-access-7c2ck\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:27.975632 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:27.975428    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90da2e53-91d3-4e77-b21a-45d8226ab9f5-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:28.268648 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:28.268611    5508 generic.go:356] "Generic (PLEG): container finished" podID="90da2e53-91d3-4e77-b21a-45d8226ab9f5" containerID="f3e187d09c66fd8ae403277d4e0eecba9f13caded2e443ea615fdb90d3c1a704" exitCode=0
Nov 03 02:23:28.269156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:28.268650    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h" event={"ID":"90da2e53-91d3-4e77-b21a-45d8226ab9f5","Type":"ContainerDied","Data":"f3e187d09c66fd8ae403277d4e0eecba9f13caded2e443ea615fdb90d3c1a704"}
Nov 03 02:23:28.269156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:28.268725    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h" event={"ID":"90da2e53-91d3-4e77-b21a-45d8226ab9f5","Type":"ContainerDied","Data":"a570bf2973757104aa790c0747a3065ce733f8e30ff6192dd6829ef781d41323"}
Nov 03 02:23:28.269156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:28.268664    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"
Nov 03 02:23:28.269156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:28.268755    5508 scope.go:122] "RemoveContainer" containerID="f3e187d09c66fd8ae403277d4e0eecba9f13caded2e443ea615fdb90d3c1a704"
Nov 03 02:23:28.282414 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:28.282392    5508 scope.go:122] "RemoveContainer" containerID="f3e187d09c66fd8ae403277d4e0eecba9f13caded2e443ea615fdb90d3c1a704"
Nov 03 02:23:28.282720 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:28.282700    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e187d09c66fd8ae403277d4e0eecba9f13caded2e443ea615fdb90d3c1a704\": container with ID starting with f3e187d09c66fd8ae403277d4e0eecba9f13caded2e443ea615fdb90d3c1a704 not found: ID does not exist" containerID="f3e187d09c66fd8ae403277d4e0eecba9f13caded2e443ea615fdb90d3c1a704"
Nov 03 02:23:28.282805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:28.282731    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e187d09c66fd8ae403277d4e0eecba9f13caded2e443ea615fdb90d3c1a704"} err="failed to get container status \"f3e187d09c66fd8ae403277d4e0eecba9f13caded2e443ea615fdb90d3c1a704\": rpc error: code = NotFound desc = could not find container \"f3e187d09c66fd8ae403277d4e0eecba9f13caded2e443ea615fdb90d3c1a704\": container with ID starting with f3e187d09c66fd8ae403277d4e0eecba9f13caded2e443ea615fdb90d3c1a704 not found: ID does not exist"
Nov 03 02:23:28.433371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:28.432405    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"]
Nov 03 02:23:28.435776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:28.434897    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bd872a-c54db8596-h4s5h"]
Nov 03 02:23:28.693487 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:28.693455    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90da2e53-91d3-4e77-b21a-45d8226ab9f5" path="/var/lib/kubelet/pods/90da2e53-91d3-4e77-b21a-45d8226ab9f5/volumes"
Nov 03 02:23:28.693904 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:28.693891    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9116085a-5fd8-41cd-9f51-5b5f3e1474c9" path="/var/lib/kubelet/pods/9116085a-5fd8-41cd-9f51-5b5f3e1474c9/volumes"
Nov 03 02:23:30.532322 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532272    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"]
Nov 03 02:23:30.532904 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532888    5508 cpu_manager.go:414] "containerMap: removing container" podUID="320b0833-f978-4d66-9d40-8693e461f63b" containerName="istio-proxy"
Nov 03 02:23:30.532958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532905    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320b0833-f978-4d66-9d40-8693e461f63b" containerName="istio-proxy"
Nov 03 02:23:30.532958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532917    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ef811d84-4691-43f7-9d23-8033ed536126" containerName="payload-processing"
Nov 03 02:23:30.532958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532922    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef811d84-4691-43f7-9d23-8033ed536126" containerName="payload-processing"
Nov 03 02:23:30.532958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532930    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9116085a-5fd8-41cd-9f51-5b5f3e1474c9" containerName="maas-api"
Nov 03 02:23:30.532958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532935    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9116085a-5fd8-41cd-9f51-5b5f3e1474c9" containerName="maas-api"
Nov 03 02:23:30.532958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532946    5508 cpu_manager.go:414] "containerMap: removing container" podUID="82e5189f-18d7-405d-a067-ded6fa319c64" containerName="maas-api"
Nov 03 02:23:30.532958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532950    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e5189f-18d7-405d-a067-ded6fa319c64" containerName="maas-api"
Nov 03 02:23:30.532958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532958    5508 cpu_manager.go:414] "containerMap: removing container" podUID="992f07df-b113-42f5-9e9c-e3864e3d9190" containerName="payload-pre-processing"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532963    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992f07df-b113-42f5-9e9c-e3864e3d9190" containerName="payload-pre-processing"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532975    5508 cpu_manager.go:414] "containerMap: removing container" podUID="90da2e53-91d3-4e77-b21a-45d8226ab9f5" containerName="maas-api"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532980    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90da2e53-91d3-4e77-b21a-45d8226ab9f5" containerName="maas-api"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532995    5508 cpu_manager.go:414] "containerMap: removing container" podUID="1ba7f2d3-aebd-4c59-8b12-193c8e88df5d" containerName="payload-processing"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.532999    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba7f2d3-aebd-4c59-8b12-193c8e88df5d" containerName="payload-processing"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.533008    5508 cpu_manager.go:414] "containerMap: removing container" podUID="320b0833-f978-4d66-9d40-8693e461f63b" containerName="istio-proxy"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.533013    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320b0833-f978-4d66-9d40-8693e461f63b" containerName="istio-proxy"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.533117    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba7f2d3-aebd-4c59-8b12-193c8e88df5d" containerName="payload-processing"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.533128    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="320b0833-f978-4d66-9d40-8693e461f63b" containerName="istio-proxy"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.533139    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef811d84-4691-43f7-9d23-8033ed536126" containerName="payload-processing"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.533145    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9116085a-5fd8-41cd-9f51-5b5f3e1474c9" containerName="maas-api"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.533153    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e5189f-18d7-405d-a067-ded6fa319c64" containerName="maas-api"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.533162    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="992f07df-b113-42f5-9e9c-e3864e3d9190" containerName="payload-pre-processing"
Nov 03 02:23:30.533218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.533171    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="90da2e53-91d3-4e77-b21a-45d8226ab9f5" containerName="maas-api"
Nov 03 02:23:30.538155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.538137    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"
Nov 03 02:23:30.586687 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.586657    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:23:30.631062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.631024    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96423399-82sn4\" (UID: \"dbc61062-0645-4d53-8c11-eb88238a0d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"
Nov 03 02:23:30.631257 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.631183    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96423399-82sn4\" (UID: \"dbc61062-0645-4d53-8c11-eb88238a0d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"
Nov 03 02:23:30.631425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.631402    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgvpc\" (UniqueName: \"kubernetes.io/projected/dbc61062-0645-4d53-8c11-eb88238a0d1c-kube-api-access-mgvpc\") pod \"maas-api-revoke-sub-96423399-82sn4\" (UID: \"dbc61062-0645-4d53-8c11-eb88238a0d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"
Nov 03 02:23:30.632823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.632805    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:23:30.637699 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.637671    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"]
Nov 03 02:23:30.733118 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.733076    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgvpc\" (UniqueName: \"kubernetes.io/projected/dbc61062-0645-4d53-8c11-eb88238a0d1c-kube-api-access-mgvpc\") pod \"maas-api-revoke-sub-96423399-82sn4\" (UID: \"dbc61062-0645-4d53-8c11-eb88238a0d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"
Nov 03 02:23:30.733293 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.733172    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96423399-82sn4\" (UID: \"dbc61062-0645-4d53-8c11-eb88238a0d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"
Nov 03 02:23:30.733293 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.733283    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96423399-82sn4\" (UID: \"dbc61062-0645-4d53-8c11-eb88238a0d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"
Nov 03 02:23:30.733914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.733884    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96423399-82sn4\" (UID: \"dbc61062-0645-4d53-8c11-eb88238a0d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"
Nov 03 02:23:30.735753 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.735729    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96423399-82sn4\" (UID: \"dbc61062-0645-4d53-8c11-eb88238a0d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"
Nov 03 02:23:30.790585 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.790502    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgvpc\" (UniqueName: \"kubernetes.io/projected/dbc61062-0645-4d53-8c11-eb88238a0d1c-kube-api-access-mgvpc\") pod \"maas-api-revoke-sub-96423399-82sn4\" (UID: \"dbc61062-0645-4d53-8c11-eb88238a0d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"
Nov 03 02:23:30.854433 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:30.854390    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"
Nov 03 02:23:31.093113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:31.093023    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"]
Nov 03 02:23:31.093303 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:31.093239    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy" containerID="cri-o://92c4954a9221cc6a0597823243bd2fa368ec33498c6124e01c782b51b4b8388e" gracePeriod=30
Nov 03 02:23:31.304295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:31.304257    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"]
Nov 03 02:23:31.316571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:31.316535    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4" event={"ID":"dbc61062-0645-4d53-8c11-eb88238a0d1c","Type":"ContainerStarted","Data":"3ce0abcb4bd6e96b6e2ee768292b03e8c9d54eb325c9b4ec32f219ed0558c3c4"}
Nov 03 02:23:32.325239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:32.325201    5508 generic.go:356] "Generic (PLEG): container finished" podID="dbc61062-0645-4d53-8c11-eb88238a0d1c" containerID="9f14fd652f7b9a2c31508fb1449deb15003613a2beff9c2529ef8cc87b43d20d" exitCode=0
Nov 03 02:23:32.325696 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:32.325300    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4" event={"ID":"dbc61062-0645-4d53-8c11-eb88238a0d1c","Type":"ContainerDied","Data":"9f14fd652f7b9a2c31508fb1449deb15003613a2beff9c2529ef8cc87b43d20d"}
Nov 03 02:23:32.426136 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:32.426098    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"]
Nov 03 02:23:32.426768 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:32.426376    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" podUID="438cacfe-f668-4897-992c-1faa9b411850" containerName="istio-proxy" containerID="cri-o://eed0378bdc45c065bd5814d01a3b2fbf04b63c4ae2951ec2a1bfd0c86dc97508" gracePeriod=30
Nov 03 02:23:33.101275 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:33.101228    5508 patch_prober.go:29] interesting pod/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:23:33.101497 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:33.101299    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:23:33.488066 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:33.488036    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"
Nov 03 02:23:33.522331 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:33.522297    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-xks-ca\") pod \"dbc61062-0645-4d53-8c11-eb88238a0d1c\" (UID: \"dbc61062-0645-4d53-8c11-eb88238a0d1c\") "
Nov 03 02:23:33.522508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:33.522393    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-openshift-ca\") pod \"dbc61062-0645-4d53-8c11-eb88238a0d1c\" (UID: \"dbc61062-0645-4d53-8c11-eb88238a0d1c\") "
Nov 03 02:23:33.522508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:33.522480    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbc61062-0645-4d53-8c11-eb88238a0d1c-kube-api-access-mgvpc\" (UniqueName: \"kubernetes.io/projected/dbc61062-0645-4d53-8c11-eb88238a0d1c-kube-api-access-mgvpc\") pod \"dbc61062-0645-4d53-8c11-eb88238a0d1c\" (UID: \"dbc61062-0645-4d53-8c11-eb88238a0d1c\") "
Nov 03 02:23:33.522908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:33.522830    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-openshift-ca" pod "dbc61062-0645-4d53-8c11-eb88238a0d1c" (UID: "dbc61062-0645-4d53-8c11-eb88238a0d1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:23:33.523124 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:33.523093    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:33.524520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:33.524499    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-xks-ca" pod "dbc61062-0645-4d53-8c11-eb88238a0d1c" (UID: "dbc61062-0645-4d53-8c11-eb88238a0d1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:23:33.524841 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:33.524819    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc61062-0645-4d53-8c11-eb88238a0d1c-kube-api-access-mgvpc" pod "dbc61062-0645-4d53-8c11-eb88238a0d1c" (UID: "dbc61062-0645-4d53-8c11-eb88238a0d1c"). InnerVolumeSpecName "kube-api-access-mgvpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:23:33.624577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:33.624547    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc61062-0645-4d53-8c11-eb88238a0d1c-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:33.624577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:33.624577    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgvpc\" (UniqueName: \"kubernetes.io/projected/dbc61062-0645-4d53-8c11-eb88238a0d1c-kube-api-access-mgvpc\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:34.341835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.341804    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"
Nov 03 02:23:34.342025 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.341836    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4" event={"ID":"dbc61062-0645-4d53-8c11-eb88238a0d1c","Type":"ContainerDied","Data":"3ce0abcb4bd6e96b6e2ee768292b03e8c9d54eb325c9b4ec32f219ed0558c3c4"}
Nov 03 02:23:34.342025 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.341887    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ce0abcb4bd6e96b6e2ee768292b03e8c9d54eb325c9b4ec32f219ed0558c3c4"
Nov 03 02:23:34.822516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.822482    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"]
Nov 03 02:23:34.823279 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.823246    5508 cpu_manager.go:414] "containerMap: removing container" podUID="dbc61062-0645-4d53-8c11-eb88238a0d1c" containerName="revoke-keys"
Nov 03 02:23:34.823279 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.823274    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbc61062-0645-4d53-8c11-eb88238a0d1c" containerName="revoke-keys"
Nov 03 02:23:34.823528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.823483    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbc61062-0645-4d53-8c11-eb88238a0d1c" containerName="revoke-keys"
Nov 03 02:23:34.823528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.823503    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="320b0833-f978-4d66-9d40-8693e461f63b" containerName="istio-proxy"
Nov 03 02:23:34.827086 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.827066    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:34.875948 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.875918    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-w2lkg\""
Nov 03 02:23:34.925381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.925335    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"]
Nov 03 02:23:34.941152 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.941104    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:34.941329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.941220    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:34.941329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.941274    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:34.941329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.941290    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:34.941553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.941430    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fefda07-12d0-4158-a08d-4e0772428124-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:34.941615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.941563    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fefda07-12d0-4158-a08d-4e0772428124-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:34.941615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.941599    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqml9\" (UniqueName: \"kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-kube-api-access-gqml9\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:34.941705 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.941656    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:34.941742 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:34.941703    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.043163 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.043121    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fefda07-12d0-4158-a08d-4e0772428124-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.043163 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.043167    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqml9\" (UniqueName: \"kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-kube-api-access-gqml9\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.043506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.043204    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.043506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.043246    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.043506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.043409    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.043506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.043460    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.043506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.043494    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.043770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.043519    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.043770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.043574    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fefda07-12d0-4158-a08d-4e0772428124-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.043770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.043728    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.043958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.043814    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.043958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.043840    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.043958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.043923    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.044337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.044315    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fefda07-12d0-4158-a08d-4e0772428124-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.045700 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.045671    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.045794 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.045775    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fefda07-12d0-4158-a08d-4e0772428124-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.100807 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.100745    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqml9\" (UniqueName: \"kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-kube-api-access-gqml9\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.100807 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.100759    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-mjk8j\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.150678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.144672    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:35.579799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.579763    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"]
Nov 03 02:23:35.586564 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:23:35.586539    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fefda07_12d0_4158_a08d_4e0772428124.slice/crio-b4aebbe5a6c3401f65e2be1fa3d1f74f776bbc757a4914969b2c5476236c13e7 WatchSource:0}: Error finding container b4aebbe5a6c3401f65e2be1fa3d1f74f776bbc757a4914969b2c5476236c13e7: Status 404 returned error can't find the container with id b4aebbe5a6c3401f65e2be1fa3d1f74f776bbc757a4914969b2c5476236c13e7
Nov 03 02:23:35.589226 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.589192    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:23:35.589333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.589271    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:23:35.589333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:35.589300    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:23:36.392113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.391463    5508 generic.go:356] "Generic (PLEG): container finished" podID="518ce897-7786-44cc-9be3-737bb47e8e50" containerID="92c4954a9221cc6a0597823243bd2fa368ec33498c6124e01c782b51b4b8388e" exitCode=0
Nov 03 02:23:36.392113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.391566    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" event={"ID":"518ce897-7786-44cc-9be3-737bb47e8e50","Type":"ContainerDied","Data":"92c4954a9221cc6a0597823243bd2fa368ec33498c6124e01c782b51b4b8388e"}
Nov 03 02:23:36.392113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.391632    5508 scope.go:122] "RemoveContainer" containerID="2365d0ad6c101f7b409d4a1cf24ef99e3b8b601c4df9537d7a9f486a78390f7b"
Nov 03 02:23:36.395732 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.395699    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" event={"ID":"9fefda07-12d0-4158-a08d-4e0772428124","Type":"ContainerStarted","Data":"578a7bfd301a84d7d0c278f49385fcee1abcca9806a00ddf806ced907d4c3db6"}
Nov 03 02:23:36.395856 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.395758    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" event={"ID":"9fefda07-12d0-4158-a08d-4e0772428124","Type":"ContainerStarted","Data":"b4aebbe5a6c3401f65e2be1fa3d1f74f776bbc757a4914969b2c5476236c13e7"}
Nov 03 02:23:36.469377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.466803    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:23:36.536744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.536683    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" podStartSLOduration=2.536666347 podStartE2EDuration="2.536666347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:36.534875078 +0000 UTC m=+3895.960481542" watchObservedRunningTime="2026-11-03 02:23:36.536666347 +0000 UTC m=+3895.962272812"
Nov 03 02:23:36.602518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.602478    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-istio-token\" (UniqueName: \"kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-istio-token\") pod \"518ce897-7786-44cc-9be3-737bb47e8e50\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") "
Nov 03 02:23:36.602693 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.602531    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-credential-socket\") pod \"518ce897-7786-44cc-9be3-737bb47e8e50\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") "
Nov 03 02:23:36.602693 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.602635    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/518ce897-7786-44cc-9be3-737bb47e8e50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518ce897-7786-44cc-9be3-737bb47e8e50-istio-podinfo\") pod \"518ce897-7786-44cc-9be3-737bb47e8e50\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") "
Nov 03 02:23:36.602787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.602768    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/518ce897-7786-44cc-9be3-737bb47e8e50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518ce897-7786-44cc-9be3-737bb47e8e50-istiod-ca-cert\") pod \"518ce897-7786-44cc-9be3-737bb47e8e50\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") "
Nov 03 02:23:36.602826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.602809    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-envoy\") pod \"518ce897-7786-44cc-9be3-737bb47e8e50\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") "
Nov 03 02:23:36.602879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.602858    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-socket\") pod \"518ce897-7786-44cc-9be3-737bb47e8e50\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") "
Nov 03 02:23:36.602975 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.602959    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-data\") pod \"518ce897-7786-44cc-9be3-737bb47e8e50\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") "
Nov 03 02:23:36.603019 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.602999    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-certs\") pod \"518ce897-7786-44cc-9be3-737bb47e8e50\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") "
Nov 03 02:23:36.603058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.603040    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-kube-api-access-qb2z8\" (UniqueName: \"kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-kube-api-access-qb2z8\") pod \"518ce897-7786-44cc-9be3-737bb47e8e50\" (UID: \"518ce897-7786-44cc-9be3-737bb47e8e50\") "
Nov 03 02:23:36.603730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.603704    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-data" pod "518ce897-7786-44cc-9be3-737bb47e8e50" (UID: "518ce897-7786-44cc-9be3-737bb47e8e50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:36.604183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.603947    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518ce897-7786-44cc-9be3-737bb47e8e50-istiod-ca-cert" pod "518ce897-7786-44cc-9be3-737bb47e8e50" (UID: "518ce897-7786-44cc-9be3-737bb47e8e50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:23:36.604183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.604022    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-socket" pod "518ce897-7786-44cc-9be3-737bb47e8e50" (UID: "518ce897-7786-44cc-9be3-737bb47e8e50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:36.604183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.604158    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-certs" pod "518ce897-7786-44cc-9be3-737bb47e8e50" (UID: "518ce897-7786-44cc-9be3-737bb47e8e50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:36.604585 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.604561    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-credential-socket" pod "518ce897-7786-44cc-9be3-737bb47e8e50" (UID: "518ce897-7786-44cc-9be3-737bb47e8e50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:36.605676 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.605652    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/518ce897-7786-44cc-9be3-737bb47e8e50-istio-podinfo" pod "518ce897-7786-44cc-9be3-737bb47e8e50" (UID: "518ce897-7786-44cc-9be3-737bb47e8e50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:23:36.606332 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.605967    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-envoy" pod "518ce897-7786-44cc-9be3-737bb47e8e50" (UID: "518ce897-7786-44cc-9be3-737bb47e8e50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:36.606332 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.606295    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-kube-api-access-qb2z8" pod "518ce897-7786-44cc-9be3-737bb47e8e50" (UID: "518ce897-7786-44cc-9be3-737bb47e8e50"). InnerVolumeSpecName "kube-api-access-qb2z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:23:36.606786 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.606761    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-istio-token" pod "518ce897-7786-44cc-9be3-737bb47e8e50" (UID: "518ce897-7786-44cc-9be3-737bb47e8e50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:23:36.707376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.705221    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:36.707376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.705261    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/518ce897-7786-44cc-9be3-737bb47e8e50-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:36.707376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.705279    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/518ce897-7786-44cc-9be3-737bb47e8e50-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:36.707376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.705292    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:36.707376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.705306    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:36.707376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.705319    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:36.707376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.705332    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/518ce897-7786-44cc-9be3-737bb47e8e50-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:36.707376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.705359    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb2z8\" (UniqueName: \"kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-kube-api-access-qb2z8\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:36.707376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.705375    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/518ce897-7786-44cc-9be3-737bb47e8e50-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:36.911691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.911651    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"]
Nov 03 02:23:36.912378 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.912341    5508 cpu_manager.go:414] "containerMap: removing container" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy"
Nov 03 02:23:36.912378 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.912375    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy"
Nov 03 02:23:36.912543 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.912515    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy"
Nov 03 02:23:36.916450 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.916429    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:36.967013 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.966432    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4fa7cdd724cd\""
Nov 03 02:23:36.967013 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.966567    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-t2v42\""
Nov 03 02:23:36.967013 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.966636    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 02:23:36.967013 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.966712    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 02:23:36.967013 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.966770    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 02:23:36.967013 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.966844    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-07b1025d12d7\""
Nov 03 02:23:36.967013 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.966906    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 02:23:36.979377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:36.979327    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"]
Nov 03 02:23:37.009701 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.009666    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl9w9\" (UniqueName: \"kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-kube-api-access-dl9w9\") pod \"payload-processing-external-model-7976ff5d9b-2vq8z\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") " pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:37.009884 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.009735    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-routing-overlay-volume\") pod \"payload-processing-external-model-7976ff5d9b-2vq8z\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") " pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:37.009884 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.009864    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-plugins-config-volume\") pod \"payload-processing-external-model-7976ff5d9b-2vq8z\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") " pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:37.010018 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.009902    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials\") pod \"payload-processing-external-model-7976ff5d9b-2vq8z\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") " pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:37.111265 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.111232    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl9w9\" (UniqueName: \"kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-kube-api-access-dl9w9\") pod \"payload-processing-external-model-7976ff5d9b-2vq8z\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") " pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:37.111470 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.111305    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-routing-overlay-volume\") pod \"payload-processing-external-model-7976ff5d9b-2vq8z\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") " pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:37.111470 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.111389    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-plugins-config-volume\") pod \"payload-processing-external-model-7976ff5d9b-2vq8z\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") " pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:37.111470 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.111411    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials\") pod \"payload-processing-external-model-7976ff5d9b-2vq8z\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") " pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:37.111988 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.111963    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-routing-overlay-volume\") pod \"payload-processing-external-model-7976ff5d9b-2vq8z\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") " pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:37.112164 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.112141    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-plugins-config-volume\") pod \"payload-processing-external-model-7976ff5d9b-2vq8z\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") " pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:37.113718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.113693    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials\") pod \"payload-processing-external-model-7976ff5d9b-2vq8z\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") " pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:37.169433 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.168713    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:37.175449 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.175401    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mjk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Nov 03 02:23:37.175636 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.175465    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" podUID="9fefda07-12d0-4158-a08d-4e0772428124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Nov 03 02:23:37.176918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.176888    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl9w9\" (UniqueName: \"kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-kube-api-access-dl9w9\") pod \"payload-processing-external-model-7976ff5d9b-2vq8z\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") " pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:37.230476 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.230396    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:37.409627 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.408770    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"
Nov 03 02:23:37.409627 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.409208    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2" event={"ID":"518ce897-7786-44cc-9be3-737bb47e8e50","Type":"ContainerDied","Data":"46157cbf9ff5482a6c903197babef67674fc6b40b5ea5f52c41c18fc0b2e66f7"}
Nov 03 02:23:37.409627 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.409256    5508 scope.go:122] "RemoveContainer" containerID="92c4954a9221cc6a0597823243bd2fa368ec33498c6124e01c782b51b4b8388e"
Nov 03 02:23:37.579770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.579557    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"]
Nov 03 02:23:37.583709 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.583674    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4f87d1-openshift-default-7cb54948df-sd8x2"]
Nov 03 02:23:37.693858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.693821    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"]
Nov 03 02:23:37.695935 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:23:37.695910    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11452e9d_9223_49c8_9baf_5ead6c181ae7.slice/crio-a9d42fe161e3f92ea6efc109bd947bf1681035a321543a3ec5089afb7c2f72b7 WatchSource:0}: Error finding container a9d42fe161e3f92ea6efc109bd947bf1681035a321543a3ec5089afb7c2f72b7: Status 404 returned error can't find the container with id a9d42fe161e3f92ea6efc109bd947bf1681035a321543a3ec5089afb7c2f72b7
Nov 03 02:23:37.738741 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.738721    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:23:37.819813 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.819778    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-data\") pod \"438cacfe-f668-4897-992c-1faa9b411850\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") "
Nov 03 02:23:37.819949 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.819850    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-socket\") pod \"438cacfe-f668-4897-992c-1faa9b411850\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") "
Nov 03 02:23:37.819949 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.819883    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-certs\") pod \"438cacfe-f668-4897-992c-1faa9b411850\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") "
Nov 03 02:23:37.820089 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.819990    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/438cacfe-f668-4897-992c-1faa9b411850-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438cacfe-f668-4897-992c-1faa9b411850-istiod-ca-cert\") pod \"438cacfe-f668-4897-992c-1faa9b411850\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") "
Nov 03 02:23:37.820089 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.820048    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/438cacfe-f668-4897-992c-1faa9b411850-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438cacfe-f668-4897-992c-1faa9b411850-istio-podinfo\") pod \"438cacfe-f668-4897-992c-1faa9b411850\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") "
Nov 03 02:23:37.820194 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.820082    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-data" pod "438cacfe-f668-4897-992c-1faa9b411850" (UID: "438cacfe-f668-4897-992c-1faa9b411850"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:37.820194 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.820136    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-socket" pod "438cacfe-f668-4897-992c-1faa9b411850" (UID: "438cacfe-f668-4897-992c-1faa9b411850"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:37.820194 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.820141    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-kube-api-access-fhvwg\" (UniqueName: \"kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-kube-api-access-fhvwg\") pod \"438cacfe-f668-4897-992c-1faa9b411850\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") "
Nov 03 02:23:37.820386 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.820237    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-credential-socket\") pod \"438cacfe-f668-4897-992c-1faa9b411850\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") "
Nov 03 02:23:37.820386 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.820298    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-envoy\") pod \"438cacfe-f668-4897-992c-1faa9b411850\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") "
Nov 03 02:23:37.820508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.820393    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438cacfe-f668-4897-992c-1faa9b411850-istiod-ca-cert" pod "438cacfe-f668-4897-992c-1faa9b411850" (UID: "438cacfe-f668-4897-992c-1faa9b411850"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:23:37.820508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.820458    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-istio-token\" (UniqueName: \"kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-istio-token\") pod \"438cacfe-f668-4897-992c-1faa9b411850\" (UID: \"438cacfe-f668-4897-992c-1faa9b411850\") "
Nov 03 02:23:37.820616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.820521    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-credential-socket" pod "438cacfe-f668-4897-992c-1faa9b411850" (UID: "438cacfe-f668-4897-992c-1faa9b411850"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:37.820675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.820652    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-certs" pod "438cacfe-f668-4897-992c-1faa9b411850" (UID: "438cacfe-f668-4897-992c-1faa9b411850"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:37.821006 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.820984    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:37.821116 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.821009    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:37.821116 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.821023    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:37.821116 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.821037    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:37.821116 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.821051    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438cacfe-f668-4897-992c-1faa9b411850-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:37.822275 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.822253    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/438cacfe-f668-4897-992c-1faa9b411850-istio-podinfo" pod "438cacfe-f668-4897-992c-1faa9b411850" (UID: "438cacfe-f668-4897-992c-1faa9b411850"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:23:37.822801 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.822777    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-istio-token" pod "438cacfe-f668-4897-992c-1faa9b411850" (UID: "438cacfe-f668-4897-992c-1faa9b411850"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:23:37.822954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.822928    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-envoy" pod "438cacfe-f668-4897-992c-1faa9b411850" (UID: "438cacfe-f668-4897-992c-1faa9b411850"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:23:37.823319 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.823054    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-kube-api-access-fhvwg" pod "438cacfe-f668-4897-992c-1faa9b411850" (UID: "438cacfe-f668-4897-992c-1faa9b411850"). InnerVolumeSpecName "kube-api-access-fhvwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:23:37.922378 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.922332    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438cacfe-f668-4897-992c-1faa9b411850-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:37.922378 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.922372    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:37.922573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.922391    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438cacfe-f668-4897-992c-1faa9b411850-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:37.922573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:37.922405    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhvwg\" (UniqueName: \"kubernetes.io/projected/438cacfe-f668-4897-992c-1faa9b411850-kube-api-access-fhvwg\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:38.150146 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.149716    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mjk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Nov 03 02:23:38.150146 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.149774    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" podUID="9fefda07-12d0-4158-a08d-4e0772428124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Nov 03 02:23:38.346284 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.346199    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"]
Nov 03 02:23:38.417815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.417777    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z" event={"ID":"11452e9d-9223-49c8-9baf-5ead6c181ae7","Type":"ContainerStarted","Data":"95ec9ae0d66348c95cf7919a76ceca221a3c7113d43451cb6c5d345039966349"}
Nov 03 02:23:38.418324 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.417843    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:38.418324 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.417866    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z" event={"ID":"11452e9d-9223-49c8-9baf-5ead6c181ae7","Type":"ContainerStarted","Data":"a9d42fe161e3f92ea6efc109bd947bf1681035a321543a3ec5089afb7c2f72b7"}
Nov 03 02:23:38.419643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.419617    5508 generic.go:356] "Generic (PLEG): container finished" podID="438cacfe-f668-4897-992c-1faa9b411850" containerID="eed0378bdc45c065bd5814d01a3b2fbf04b63c4ae2951ec2a1bfd0c86dc97508" exitCode=0
Nov 03 02:23:38.419782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.419666    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"
Nov 03 02:23:38.419782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.419694    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" event={"ID":"438cacfe-f668-4897-992c-1faa9b411850","Type":"ContainerDied","Data":"eed0378bdc45c065bd5814d01a3b2fbf04b63c4ae2951ec2a1bfd0c86dc97508"}
Nov 03 02:23:38.419782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.419743    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq" event={"ID":"438cacfe-f668-4897-992c-1faa9b411850","Type":"ContainerDied","Data":"89acb7ebf030d2ac765b683c4ad757307e6b6d1225198ff766c2604948a9402c"}
Nov 03 02:23:38.419782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.419773    5508 scope.go:122] "RemoveContainer" containerID="eed0378bdc45c065bd5814d01a3b2fbf04b63c4ae2951ec2a1bfd0c86dc97508"
Nov 03 02:23:38.430177 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.430157    5508 scope.go:122] "RemoveContainer" containerID="b6712fd1a15cf9020e702244c11d10f3d0f815053d7d727cf5490b5fcdfddb36"
Nov 03 02:23:38.442703 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.442686    5508 scope.go:122] "RemoveContainer" containerID="eed0378bdc45c065bd5814d01a3b2fbf04b63c4ae2951ec2a1bfd0c86dc97508"
Nov 03 02:23:38.443045 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:38.443018    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed0378bdc45c065bd5814d01a3b2fbf04b63c4ae2951ec2a1bfd0c86dc97508\": container with ID starting with eed0378bdc45c065bd5814d01a3b2fbf04b63c4ae2951ec2a1bfd0c86dc97508 not found: ID does not exist" containerID="eed0378bdc45c065bd5814d01a3b2fbf04b63c4ae2951ec2a1bfd0c86dc97508"
Nov 03 02:23:38.443142 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.443055    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed0378bdc45c065bd5814d01a3b2fbf04b63c4ae2951ec2a1bfd0c86dc97508"} err="failed to get container status \"eed0378bdc45c065bd5814d01a3b2fbf04b63c4ae2951ec2a1bfd0c86dc97508\": rpc error: code = NotFound desc = could not find container \"eed0378bdc45c065bd5814d01a3b2fbf04b63c4ae2951ec2a1bfd0c86dc97508\": container with ID starting with eed0378bdc45c065bd5814d01a3b2fbf04b63c4ae2951ec2a1bfd0c86dc97508 not found: ID does not exist"
Nov 03 02:23:38.443142 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.443077    5508 scope.go:122] "RemoveContainer" containerID="b6712fd1a15cf9020e702244c11d10f3d0f815053d7d727cf5490b5fcdfddb36"
Nov 03 02:23:38.443478 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:38.443444    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6712fd1a15cf9020e702244c11d10f3d0f815053d7d727cf5490b5fcdfddb36\": container with ID starting with b6712fd1a15cf9020e702244c11d10f3d0f815053d7d727cf5490b5fcdfddb36 not found: ID does not exist" containerID="b6712fd1a15cf9020e702244c11d10f3d0f815053d7d727cf5490b5fcdfddb36"
Nov 03 02:23:38.443559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.443489    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6712fd1a15cf9020e702244c11d10f3d0f815053d7d727cf5490b5fcdfddb36"} err="failed to get container status \"b6712fd1a15cf9020e702244c11d10f3d0f815053d7d727cf5490b5fcdfddb36\": rpc error: code = NotFound desc = could not find container \"b6712fd1a15cf9020e702244c11d10f3d0f815053d7d727cf5490b5fcdfddb36\": container with ID starting with b6712fd1a15cf9020e702244c11d10f3d0f815053d7d727cf5490b5fcdfddb36 not found: ID does not exist"
Nov 03 02:23:38.585517 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.585451    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z" podStartSLOduration=2.585432215 podStartE2EDuration="2.585432215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:38.583281165 +0000 UTC m=+3898.008887631" watchObservedRunningTime="2026-11-03 02:23:38.585432215 +0000 UTC m=+3898.011038681"
Nov 03 02:23:38.709507 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.708831    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" path="/var/lib/kubelet/pods/518ce897-7786-44cc-9be3-737bb47e8e50/volumes"
Nov 03 02:23:38.717083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.717050    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"]
Nov 03 02:23:38.722030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:38.722003    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bd872a-openshift-default-59fdfc8f-wjwzq"]
Nov 03 02:23:39.177374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:39.165434    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mjk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Nov 03 02:23:39.177374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:39.165620    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" podUID="9fefda07-12d0-4158-a08d-4e0772428124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Nov 03 02:23:39.567559 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:39.567462    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-07b1025d12d7: secret "agc-cred-07b1025d12d7" not found
Nov 03 02:23:39.573487 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:39.573458    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z: secret "agc-cred-07b1025d12d7" not found
Nov 03 02:23:39.573681 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:39.573551    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials podName:11452e9d-9223-49c8-9baf-5ead6c181ae7 nodeName:}" failed. No retries permitted until 2026-11-03 02:23:40.073528153 +0000 UTC m=+3899.499134596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials") pod "payload-processing-external-model-7976ff5d9b-2vq8z" (UID: "11452e9d-9223-49c8-9baf-5ead6c181ae7") : secret "agc-cred-07b1025d12d7" not found
Nov 03 02:23:40.095691 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:40.095653    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-07b1025d12d7: secret "agc-cred-07b1025d12d7" not found
Nov 03 02:23:40.095691 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:40.095686    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z: secret "agc-cred-07b1025d12d7" not found
Nov 03 02:23:40.095932 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:40.095770    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials podName:11452e9d-9223-49c8-9baf-5ead6c181ae7 nodeName:}" failed. No retries permitted until 2026-11-03 02:23:41.095754473 +0000 UTC m=+3900.521360920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials") pod "payload-processing-external-model-7976ff5d9b-2vq8z" (UID: "11452e9d-9223-49c8-9baf-5ead6c181ae7") : secret "agc-cred-07b1025d12d7" not found
Nov 03 02:23:40.153337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:40.153300    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mjk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Nov 03 02:23:40.153635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:40.153604    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" podUID="9fefda07-12d0-4158-a08d-4e0772428124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Nov 03 02:23:40.693927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:40.693894    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="438cacfe-f668-4897-992c-1faa9b411850" path="/var/lib/kubelet/pods/438cacfe-f668-4897-992c-1faa9b411850/volumes"
Nov 03 02:23:41.111317 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:41.111213    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-07b1025d12d7: secret "agc-cred-07b1025d12d7" not found
Nov 03 02:23:41.111317 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:41.111244    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4fa7cdd724cd: secret "agc-cred-4fa7cdd724cd" not found
Nov 03 02:23:41.111317 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:41.111257    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z: [secret "agc-cred-07b1025d12d7" not found, secret "agc-cred-4fa7cdd724cd" not found]
Nov 03 02:23:41.111675 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:41.111328    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials podName:11452e9d-9223-49c8-9baf-5ead6c181ae7 nodeName:}" failed. No retries permitted until 2026-11-03 02:23:43.111308747 +0000 UTC m=+3902.536915212 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials") pod "payload-processing-external-model-7976ff5d9b-2vq8z" (UID: "11452e9d-9223-49c8-9baf-5ead6c181ae7") : [secret "agc-cred-07b1025d12d7" not found, secret "agc-cred-4fa7cdd724cd" not found]
Nov 03 02:23:41.156372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:41.152731    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mjk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Nov 03 02:23:41.156372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:41.152792    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" podUID="9fefda07-12d0-4158-a08d-4e0772428124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Nov 03 02:23:41.465472 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:41.450984    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:23:41.565372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:41.562613    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:23:41.713648 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:41.713613    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"]
Nov 03 02:23:42.149495 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:42.149454    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mjk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Nov 03 02:23:42.149700 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:42.149514    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" podUID="9fefda07-12d0-4158-a08d-4e0772428124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Nov 03 02:23:43.145270 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:43.145220    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-07b1025d12d7: secret "agc-cred-07b1025d12d7" not found
Nov 03 02:23:43.145270 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:43.145261    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4fa7cdd724cd: secret "agc-cred-4fa7cdd724cd" not found
Nov 03 02:23:43.145270 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:43.145271    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z: [secret "agc-cred-07b1025d12d7" not found, secret "agc-cred-4fa7cdd724cd" not found]
Nov 03 02:23:43.145966 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:43.145330    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials podName:11452e9d-9223-49c8-9baf-5ead6c181ae7 nodeName:}" failed. No retries permitted until 2026-11-03 02:23:47.145314268 +0000 UTC m=+3906.570920711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials") pod "payload-processing-external-model-7976ff5d9b-2vq8z" (UID: "11452e9d-9223-49c8-9baf-5ead6c181ae7") : [secret "agc-cred-07b1025d12d7" not found, secret "agc-cred-4fa7cdd724cd" not found]
Nov 03 02:23:43.149947 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.149920    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mjk8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused" start-of-body=
Nov 03 02:23:43.150109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.149965    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" podUID="9fefda07-12d0-4158-a08d-4e0772428124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.57:15021/healthz/ready\": dial tcp 10.133.1.57:15021: connect: connection refused"
Nov 03 02:23:43.325964 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.325930    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"]
Nov 03 02:23:43.326519 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.326501    5508 cpu_manager.go:414] "containerMap: removing container" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy"
Nov 03 02:23:43.326519 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.326515    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy"
Nov 03 02:23:43.326745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.326549    5508 cpu_manager.go:414] "containerMap: removing container" podUID="438cacfe-f668-4897-992c-1faa9b411850" containerName="istio-proxy"
Nov 03 02:23:43.326745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.326554    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="438cacfe-f668-4897-992c-1faa9b411850" containerName="istio-proxy"
Nov 03 02:23:43.326745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.326567    5508 cpu_manager.go:414] "containerMap: removing container" podUID="438cacfe-f668-4897-992c-1faa9b411850" containerName="istio-proxy"
Nov 03 02:23:43.326745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.326572    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="438cacfe-f668-4897-992c-1faa9b411850" containerName="istio-proxy"
Nov 03 02:23:43.326745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.326658    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="438cacfe-f668-4897-992c-1faa9b411850" containerName="istio-proxy"
Nov 03 02:23:43.326745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.326670    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="518ce897-7786-44cc-9be3-737bb47e8e50" containerName="istio-proxy"
Nov 03 02:23:43.330759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.330738    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.347417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.347363    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6" podUID="ff52d99c-d352-4ebd-91dc-1b95ecbdf766" containerName="payload-pre-processing" containerID="cri-o://81cde068f51317ec3a3b6e7375a3c17b796b3106c740da843923ba2f3009c837" gracePeriod=25
Nov 03 02:23:43.395169 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.386439    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d6fbd7-openshift-default-dockercfg-tqg6d\""
Nov 03 02:23:43.423792 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.423760    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"]
Nov 03 02:23:43.497440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.472073    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-token\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.497440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.472127    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-data\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.497440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.472153    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-envoy\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.497440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.472189    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-socket\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.497440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.472217    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhvj7\" (UniqueName: \"kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-kube-api-access-bhvj7\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.497440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.472248    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df719dd5-eb51-4427-9a9d-626b1c277cb5-istiod-ca-cert\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.497440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.472300    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-credential-socket\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.497440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.472407    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-podinfo\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.497440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.472561    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-certs\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.576848    5508 generic.go:356] "Generic (PLEG): container finished" podID="ff52d99c-d352-4ebd-91dc-1b95ecbdf766" containerID="81cde068f51317ec3a3b6e7375a3c17b796b3106c740da843923ba2f3009c837" exitCode=0
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.579135    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6" event={"ID":"ff52d99c-d352-4ebd-91dc-1b95ecbdf766","Type":"ContainerDied","Data":"81cde068f51317ec3a3b6e7375a3c17b796b3106c740da843923ba2f3009c837"}
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.579921    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-token\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.579959    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-data\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.579985    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-envoy\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.580019    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-socket\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.580043    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhvj7\" (UniqueName: \"kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-kube-api-access-bhvj7\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.580071    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df719dd5-eb51-4427-9a9d-626b1c277cb5-istiod-ca-cert\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.580113    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-credential-socket\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.580182    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-podinfo\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.580295    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-certs\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.580838    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-certs\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.581004    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-socket\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.581689    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-data\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.581805    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df719dd5-eb51-4427-9a9d-626b1c277cb5-istiod-ca-cert\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.582048    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-credential-socket\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.585368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.585372    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-podinfo\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.596601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.596546    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-envoy\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.641404 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.639963    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhvj7\" (UniqueName: \"kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-kube-api-access-bhvj7\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.644060 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.644033    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-token\") pod \"e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") " pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.645184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.645158    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:43.771437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.770802    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"
Nov 03 02:23:43.796302 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.794461    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-plugins-config-volume\") pod \"ff52d99c-d352-4ebd-91dc-1b95ecbdf766\" (UID: \"ff52d99c-d352-4ebd-91dc-1b95ecbdf766\") "
Nov 03 02:23:43.796302 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.794696    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-kube-api-access-5l6nt\" (UniqueName: \"kubernetes.io/projected/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-kube-api-access-5l6nt\") pod \"ff52d99c-d352-4ebd-91dc-1b95ecbdf766\" (UID: \"ff52d99c-d352-4ebd-91dc-1b95ecbdf766\") "
Nov 03 02:23:43.796604 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.796523    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-plugins-config-volume" pod "ff52d99c-d352-4ebd-91dc-1b95ecbdf766" (UID: "ff52d99c-d352-4ebd-91dc-1b95ecbdf766"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:23:43.806380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.805319    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-kube-api-access-5l6nt" pod "ff52d99c-d352-4ebd-91dc-1b95ecbdf766" (UID: "ff52d99c-d352-4ebd-91dc-1b95ecbdf766"). InnerVolumeSpecName "kube-api-access-5l6nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:23:43.843158 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:43.842611    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:23:43.897761 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.897581    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l6nt\" (UniqueName: \"kubernetes.io/projected/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-kube-api-access-5l6nt\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:43.897761 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:43.897658    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff52d99c-d352-4ebd-91dc-1b95ecbdf766-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:44.186369 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.184115    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"]
Nov 03 02:23:44.206615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.204198    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:23:44.206615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.204305    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:23:44.206615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.204371    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:23:44.594490 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.593108    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"
Nov 03 02:23:44.594490 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.593136    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6" event={"ID":"ff52d99c-d352-4ebd-91dc-1b95ecbdf766","Type":"ContainerDied","Data":"7ea337e8ccecf366de16f387f2d2397926454ff9b408ecab1bd20869e9de2337"}
Nov 03 02:23:44.594490 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.593230    5508 scope.go:122] "RemoveContainer" containerID="81cde068f51317ec3a3b6e7375a3c17b796b3106c740da843923ba2f3009c837"
Nov 03 02:23:44.601104 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.601066    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx" event={"ID":"df719dd5-eb51-4427-9a9d-626b1c277cb5","Type":"ContainerStarted","Data":"9e291a49da0ae12a84e63e1ae01e31e16275937e02525062c131efe8a3944e46"}
Nov 03 02:23:44.601267 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.601134    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx" event={"ID":"df719dd5-eb51-4427-9a9d-626b1c277cb5","Type":"ContainerStarted","Data":"ba728e2f4709a9154e5f0b7ef37c4738594c2a806d14548fc9fa9892133eb906"}
Nov 03 02:23:44.663371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.659676    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:44.778411 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.777672    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx" podStartSLOduration=1.77763812 podStartE2EDuration="1.77763812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:44.776407686 +0000 UTC m=+3904.202014152" watchObservedRunningTime="2026-11-03 02:23:44.77763812 +0000 UTC m=+3904.203244586"
Nov 03 02:23:44.882177 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.882077    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"]
Nov 03 02:23:44.887559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.887450    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3fb07e-6fc88ff66c-ttpn6"]
Nov 03 02:23:44.957914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.957883    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:44.959326 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.958873    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:44.959872 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:44.959854    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:23:45.652287 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:45.652253    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:46.616829 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:46.616800    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:46.617892 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:46.617870    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:23:46.700157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:46.700115    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff52d99c-d352-4ebd-91dc-1b95ecbdf766" path="/var/lib/kubelet/pods/ff52d99c-d352-4ebd-91dc-1b95ecbdf766/volumes"
Nov 03 02:23:46.714767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:46.714723    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z" podUID="11452e9d-9223-49c8-9baf-5ead6c181ae7" containerName="payload-processing" containerID="cri-o://95ec9ae0d66348c95cf7919a76ceca221a3c7113d43451cb6c5d345039966349" gracePeriod=25
Nov 03 02:23:47.135652 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.135119    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:47.170382 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:47.165843    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-07b1025d12d7: secret "agc-cred-07b1025d12d7" not found
Nov 03 02:23:47.170382 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:47.165873    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-4fa7cdd724cd: secret "agc-cred-4fa7cdd724cd" not found
Nov 03 02:23:47.170382 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:47.165886    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z: [secret "agc-cred-07b1025d12d7" not found, secret "agc-cred-4fa7cdd724cd" not found]
Nov 03 02:23:47.170382 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:47.165948    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials podName:11452e9d-9223-49c8-9baf-5ead6c181ae7 nodeName:}" failed. No retries permitted until 2026-11-03 02:23:55.165929378 +0000 UTC m=+3914.591535841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials") pod "payload-processing-external-model-7976ff5d9b-2vq8z" (UID: "11452e9d-9223-49c8-9baf-5ead6c181ae7") : [secret "agc-cred-07b1025d12d7" not found, secret "agc-cred-4fa7cdd724cd" not found]
Nov 03 02:23:47.275428 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.267018    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-kube-api-access-dl9w9\" (UniqueName: \"kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-kube-api-access-dl9w9\") pod \"11452e9d-9223-49c8-9baf-5ead6c181ae7\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") "
Nov 03 02:23:47.275428 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.267091    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-plugins-config-volume\") pod \"11452e9d-9223-49c8-9baf-5ead6c181ae7\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") "
Nov 03 02:23:47.275428 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.267151    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials\") pod \"11452e9d-9223-49c8-9baf-5ead6c181ae7\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") "
Nov 03 02:23:47.275428 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.267361    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-routing-overlay-volume\") pod \"11452e9d-9223-49c8-9baf-5ead6c181ae7\" (UID: \"11452e9d-9223-49c8-9baf-5ead6c181ae7\") "
Nov 03 02:23:47.275825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.275622    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-plugins-config-volume" pod "11452e9d-9223-49c8-9baf-5ead6c181ae7" (UID: "11452e9d-9223-49c8-9baf-5ead6c181ae7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:23:47.276138 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.276090    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-routing-overlay-volume" pod "11452e9d-9223-49c8-9baf-5ead6c181ae7" (UID: "11452e9d-9223-49c8-9baf-5ead6c181ae7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:23:47.288370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.283820    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-kube-api-access-dl9w9" pod "11452e9d-9223-49c8-9baf-5ead6c181ae7" (UID: "11452e9d-9223-49c8-9baf-5ead6c181ae7"). InnerVolumeSpecName "kube-api-access-dl9w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:23:47.320390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.319300    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials" pod "11452e9d-9223-49c8-9baf-5ead6c181ae7" (UID: "11452e9d-9223-49c8-9baf-5ead6c181ae7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:23:47.360596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.355005    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"]
Nov 03 02:23:47.360596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.355962    5508 cpu_manager.go:414] "containerMap: removing container" podUID="11452e9d-9223-49c8-9baf-5ead6c181ae7" containerName="payload-processing"
Nov 03 02:23:47.360596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.355985    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11452e9d-9223-49c8-9baf-5ead6c181ae7" containerName="payload-processing"
Nov 03 02:23:47.360596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.356008    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ff52d99c-d352-4ebd-91dc-1b95ecbdf766" containerName="payload-pre-processing"
Nov 03 02:23:47.360596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.356019    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff52d99c-d352-4ebd-91dc-1b95ecbdf766" containerName="payload-pre-processing"
Nov 03 02:23:47.360596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.356463    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="438cacfe-f668-4897-992c-1faa9b411850" containerName="istio-proxy"
Nov 03 02:23:47.360596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.356492    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="11452e9d-9223-49c8-9baf-5ead6c181ae7" containerName="payload-processing"
Nov 03 02:23:47.360596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.356505    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff52d99c-d352-4ebd-91dc-1b95ecbdf766" containerName="payload-pre-processing"
Nov 03 02:23:47.362639 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.362616    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"
Nov 03 02:23:47.377961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.375963    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9nl\" (UniqueName: \"kubernetes.io/projected/845bcb6d-d2e0-4806-be2a-5feed5fb7796-kube-api-access-tf9nl\") pod \"payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74\" (UID: \"845bcb6d-d2e0-4806-be2a-5feed5fb7796\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"
Nov 03 02:23:47.377961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.376063    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845bcb6d-d2e0-4806-be2a-5feed5fb7796-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74\" (UID: \"845bcb6d-d2e0-4806-be2a-5feed5fb7796\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"
Nov 03 02:23:47.377961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.376134    5508 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-routing-overlay-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:47.377961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.376146    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl9w9\" (UniqueName: \"kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-kube-api-access-dl9w9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:47.377961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.376161    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11452e9d-9223-49c8-9baf-5ead6c181ae7-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:47.377961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.376172    5508 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11452e9d-9223-49c8-9baf-5ead6c181ae7-provider-credentials\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:23:47.411944 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.411643    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e051f3\""
Nov 03 02:23:47.411944 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.411791    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e051f3-dockercfg-v6bnh\""
Nov 03 02:23:47.420522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.420492    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"]
Nov 03 02:23:47.511178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.495184    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845bcb6d-d2e0-4806-be2a-5feed5fb7796-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74\" (UID: \"845bcb6d-d2e0-4806-be2a-5feed5fb7796\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"
Nov 03 02:23:47.511178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.495339    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9nl\" (UniqueName: \"kubernetes.io/projected/845bcb6d-d2e0-4806-be2a-5feed5fb7796-kube-api-access-tf9nl\") pod \"payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74\" (UID: \"845bcb6d-d2e0-4806-be2a-5feed5fb7796\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"
Nov 03 02:23:47.511178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.508968    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845bcb6d-d2e0-4806-be2a-5feed5fb7796-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74\" (UID: \"845bcb6d-d2e0-4806-be2a-5feed5fb7796\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"
Nov 03 02:23:47.559706 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.552941    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"]
Nov 03 02:23:47.575798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.575421    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"
Nov 03 02:23:47.578062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.578024    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9nl\" (UniqueName: \"kubernetes.io/projected/845bcb6d-d2e0-4806-be2a-5feed5fb7796-kube-api-access-tf9nl\") pod \"payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74\" (UID: \"845bcb6d-d2e0-4806-be2a-5feed5fb7796\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"
Nov 03 02:23:47.597584 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.597554    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjrqd\" (UniqueName: \"kubernetes.io/projected/e9983db5-4103-41af-98ac-4d3847ade519-kube-api-access-fjrqd\") pod \"payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g\" (UID: \"e9983db5-4103-41af-98ac-4d3847ade519\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"
Nov 03 02:23:47.597754 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.597716    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9983db5-4103-41af-98ac-4d3847ade519-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g\" (UID: \"e9983db5-4103-41af-98ac-4d3847ade519\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"
Nov 03 02:23:47.603776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.600724    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"]
Nov 03 02:23:47.629474 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.626032    5508 generic.go:356] "Generic (PLEG): container finished" podID="11452e9d-9223-49c8-9baf-5ead6c181ae7" containerID="95ec9ae0d66348c95cf7919a76ceca221a3c7113d43451cb6c5d345039966349" exitCode=0
Nov 03 02:23:47.629474 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.626339    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"
Nov 03 02:23:47.629474 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.626398    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z" event={"ID":"11452e9d-9223-49c8-9baf-5ead6c181ae7","Type":"ContainerDied","Data":"95ec9ae0d66348c95cf7919a76ceca221a3c7113d43451cb6c5d345039966349"}
Nov 03 02:23:47.629474 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.626453    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z" event={"ID":"11452e9d-9223-49c8-9baf-5ead6c181ae7","Type":"ContainerDied","Data":"a9d42fe161e3f92ea6efc109bd947bf1681035a321543a3ec5089afb7c2f72b7"}
Nov 03 02:23:47.629474 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.626490    5508 scope.go:122] "RemoveContainer" containerID="95ec9ae0d66348c95cf7919a76ceca221a3c7113d43451cb6c5d345039966349"
Nov 03 02:23:47.645180 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.645085    5508 scope.go:122] "RemoveContainer" containerID="95ec9ae0d66348c95cf7919a76ceca221a3c7113d43451cb6c5d345039966349"
Nov 03 02:23:47.645750 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:47.645722    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ec9ae0d66348c95cf7919a76ceca221a3c7113d43451cb6c5d345039966349\": container with ID starting with 95ec9ae0d66348c95cf7919a76ceca221a3c7113d43451cb6c5d345039966349 not found: ID does not exist" containerID="95ec9ae0d66348c95cf7919a76ceca221a3c7113d43451cb6c5d345039966349"
Nov 03 02:23:47.645841 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.645760    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ec9ae0d66348c95cf7919a76ceca221a3c7113d43451cb6c5d345039966349"} err="failed to get container status \"95ec9ae0d66348c95cf7919a76ceca221a3c7113d43451cb6c5d345039966349\": rpc error: code = NotFound desc = could not find container \"95ec9ae0d66348c95cf7919a76ceca221a3c7113d43451cb6c5d345039966349\": container with ID starting with 95ec9ae0d66348c95cf7919a76ceca221a3c7113d43451cb6c5d345039966349 not found: ID does not exist"
Nov 03 02:23:47.686984 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.686582    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"
Nov 03 02:23:47.708557 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.707198    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9983db5-4103-41af-98ac-4d3847ade519-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g\" (UID: \"e9983db5-4103-41af-98ac-4d3847ade519\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"
Nov 03 02:23:47.708557 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.707446    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjrqd\" (UniqueName: \"kubernetes.io/projected/e9983db5-4103-41af-98ac-4d3847ade519-kube-api-access-fjrqd\") pod \"payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g\" (UID: \"e9983db5-4103-41af-98ac-4d3847ade519\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"
Nov 03 02:23:47.708557 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.708457    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9983db5-4103-41af-98ac-4d3847ade519-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g\" (UID: \"e9983db5-4103-41af-98ac-4d3847ade519\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"
Nov 03 02:23:47.779120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.769187    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjrqd\" (UniqueName: \"kubernetes.io/projected/e9983db5-4103-41af-98ac-4d3847ade519-kube-api-access-fjrqd\") pod \"payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g\" (UID: \"e9983db5-4103-41af-98ac-4d3847ade519\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"
Nov 03 02:23:47.811567 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.807550    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"]
Nov 03 02:23:47.813077 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.812501    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7976ff5d9b-2vq8z"]
Nov 03 02:23:47.904226 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:47.904194    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"
Nov 03 02:23:48.120483 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:48.120405    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"]
Nov 03 02:23:48.127235 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:23:48.127204    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845bcb6d_d2e0_4806_be2a_5feed5fb7796.slice/crio-395a7cb1281c7f0e217cdd82a6e44736a392da1b0d39affe681871eb81020f9a WatchSource:0}: Error finding container 395a7cb1281c7f0e217cdd82a6e44736a392da1b0d39affe681871eb81020f9a: Status 404 returned error can't find the container with id 395a7cb1281c7f0e217cdd82a6e44736a392da1b0d39affe681871eb81020f9a
Nov 03 02:23:48.289620 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:48.289584    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"]
Nov 03 02:23:48.297173 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:23:48.297142    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9983db5_4103_41af_98ac_4d3847ade519.slice/crio-f1d03b533a833a4d09e7578e7aa3e96f1071471426018059ca0010bbf2b9ecd3 WatchSource:0}: Error finding container f1d03b533a833a4d09e7578e7aa3e96f1071471426018059ca0010bbf2b9ecd3: Status 404 returned error can't find the container with id f1d03b533a833a4d09e7578e7aa3e96f1071471426018059ca0010bbf2b9ecd3
Nov 03 02:23:48.637149 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:48.637110    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g" event={"ID":"e9983db5-4103-41af-98ac-4d3847ade519","Type":"ContainerStarted","Data":"9ceebcb1779117c5adb9f38ccca26a31df4418d83c76c1d641a7bf4bb8f77667"}
Nov 03 02:23:48.637396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:48.637167    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g" event={"ID":"e9983db5-4103-41af-98ac-4d3847ade519","Type":"ContainerStarted","Data":"f1d03b533a833a4d09e7578e7aa3e96f1071471426018059ca0010bbf2b9ecd3"}
Nov 03 02:23:48.637396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:48.637340    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"
Nov 03 02:23:48.638889 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:48.638860    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74" event={"ID":"845bcb6d-d2e0-4806-be2a-5feed5fb7796","Type":"ContainerStarted","Data":"3a039fa6a28bd34aa30357499740be0d6f6a8c6b938d7b29c2db214426212043"}
Nov 03 02:23:48.638989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:48.638911    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74" event={"ID":"845bcb6d-d2e0-4806-be2a-5feed5fb7796","Type":"ContainerStarted","Data":"395a7cb1281c7f0e217cdd82a6e44736a392da1b0d39affe681871eb81020f9a"}
Nov 03 02:23:48.639114 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:48.639025    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"
Nov 03 02:23:48.694924 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:48.694891    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11452e9d-9223-49c8-9baf-5ead6c181ae7" path="/var/lib/kubelet/pods/11452e9d-9223-49c8-9baf-5ead6c181ae7/volumes"
Nov 03 02:23:48.749155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:48.749094    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g" podStartSLOduration=1.749074615 podStartE2EDuration="1.749074615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:48.746980563 +0000 UTC m=+3908.172587028" watchObservedRunningTime="2026-11-03 02:23:48.749074615 +0000 UTC m=+3908.174681085"
Nov 03 02:23:48.857500 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:48.857433    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74" podStartSLOduration=1.857414505 podStartE2EDuration="1.857414505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:48.855129325 +0000 UTC m=+3908.280735791" watchObservedRunningTime="2026-11-03 02:23:48.857414505 +0000 UTC m=+3908.283020971"
Nov 03 02:23:49.736028 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.731888    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"]
Nov 03 02:23:49.738372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.738239    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:49.789602 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.789558    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3f45679e4378\""
Nov 03 02:23:49.790063 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.789597    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 02:23:49.790063 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.789624    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 02:23:49.833544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.833510    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials\") pod \"payload-processing-external-model-7578fcd544-t7thb\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:49.833723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.833636    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-plugins-config-volume\") pod \"payload-processing-external-model-7578fcd544-t7thb\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:49.833723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.833679    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-routing-overlay-volume\") pod \"payload-processing-external-model-7578fcd544-t7thb\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:49.833723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.833715    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtdp8\" (UniqueName: \"kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-kube-api-access-mtdp8\") pod \"payload-processing-external-model-7578fcd544-t7thb\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:49.835715 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.835693    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 02:23:49.835715 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.835693    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vjs22\""
Nov 03 02:23:49.835883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.835701    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 02:23:49.839059 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.839039    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"]
Nov 03 02:23:49.934465 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.934431    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-plugins-config-volume\") pod \"payload-processing-external-model-7578fcd544-t7thb\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:49.934465 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.934470    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-routing-overlay-volume\") pod \"payload-processing-external-model-7578fcd544-t7thb\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:49.934733 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.934490    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtdp8\" (UniqueName: \"kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-kube-api-access-mtdp8\") pod \"payload-processing-external-model-7578fcd544-t7thb\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:49.934733 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.934613    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials\") pod \"payload-processing-external-model-7578fcd544-t7thb\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:49.935233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.935205    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-routing-overlay-volume\") pod \"payload-processing-external-model-7578fcd544-t7thb\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:49.935387 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.935256    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-plugins-config-volume\") pod \"payload-processing-external-model-7578fcd544-t7thb\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:49.937085 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.937064    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials\") pod \"payload-processing-external-model-7578fcd544-t7thb\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:49.991984 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:49.991915    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtdp8\" (UniqueName: \"kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-kube-api-access-mtdp8\") pod \"payload-processing-external-model-7578fcd544-t7thb\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:50.055555 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:50.055518    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:50.505404 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:50.505362    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"]
Nov 03 02:23:50.512560 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:23:50.512530    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode381f68e_d9a6_4a17_9e0a_2a0daa958cc3.slice/crio-f99de20b671f1b2d7b45f78b3cc79e9ca83a4ca850b11aae6a590ec7bc901a94 WatchSource:0}: Error finding container f99de20b671f1b2d7b45f78b3cc79e9ca83a4ca850b11aae6a590ec7bc901a94: Status 404 returned error can't find the container with id f99de20b671f1b2d7b45f78b3cc79e9ca83a4ca850b11aae6a590ec7bc901a94
Nov 03 02:23:50.653887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:50.653853    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb" event={"ID":"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3","Type":"ContainerStarted","Data":"ffe1857f847a92b710434a0911d722b3f3eb46137235c2382868ce40fa6d9ec5"}
Nov 03 02:23:50.654094 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:50.653907    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb" event={"ID":"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3","Type":"ContainerStarted","Data":"f99de20b671f1b2d7b45f78b3cc79e9ca83a4ca850b11aae6a590ec7bc901a94"}
Nov 03 02:23:50.654094 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:50.654059    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:50.783609 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:50.783504    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3f45679e4378: secret "agc-cred-3f45679e4378" not found
Nov 03 02:23:50.783609 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:50.783531    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7578fcd544-t7thb: secret "agc-cred-3f45679e4378" not found
Nov 03 02:23:50.784014 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:50.783976    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials podName:e381f68e-d9a6-4a17-9e0a-2a0daa958cc3 nodeName:}" failed. No retries permitted until 2026-11-03 02:23:51.283953247 +0000 UTC m=+3910.709559714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials") pod "payload-processing-external-model-7578fcd544-t7thb" (UID: "e381f68e-d9a6-4a17-9e0a-2a0daa958cc3") : secret "agc-cred-3f45679e4378" not found
Nov 03 02:23:51.293765 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:51.293732    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3f45679e4378: secret "agc-cred-3f45679e4378" not found
Nov 03 02:23:51.293765 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:51.293761    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7578fcd544-t7thb: secret "agc-cred-3f45679e4378" not found
Nov 03 02:23:51.294238 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:51.293811    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials podName:e381f68e-d9a6-4a17-9e0a-2a0daa958cc3 nodeName:}" failed. No retries permitted until 2026-11-03 02:23:52.293797708 +0000 UTC m=+3911.719404165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials") pod "payload-processing-external-model-7578fcd544-t7thb" (UID: "e381f68e-d9a6-4a17-9e0a-2a0daa958cc3") : secret "agc-cred-3f45679e4378" not found
Nov 03 02:23:52.307723 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:52.307681    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3f45679e4378: secret "agc-cred-3f45679e4378" not found
Nov 03 02:23:52.307723 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:52.307706    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7578fcd544-t7thb: secret "agc-cred-3f45679e4378" not found
Nov 03 02:23:52.308362 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:52.307763    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials podName:e381f68e-d9a6-4a17-9e0a-2a0daa958cc3 nodeName:}" failed. No retries permitted until 2026-11-03 02:23:54.307749198 +0000 UTC m=+3913.733355657 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials") pod "payload-processing-external-model-7578fcd544-t7thb" (UID: "e381f68e-d9a6-4a17-9e0a-2a0daa958cc3") : secret "agc-cred-3f45679e4378" not found
Nov 03 02:23:54.338271 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:54.338240    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3f45679e4378: secret "agc-cred-3f45679e4378" not found
Nov 03 02:23:54.338271 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:54.338265    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7578fcd544-t7thb: secret "agc-cred-3f45679e4378" not found
Nov 03 02:23:54.338734 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:23:54.338320    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials podName:e381f68e-d9a6-4a17-9e0a-2a0daa958cc3 nodeName:}" failed. No retries permitted until 2026-11-03 02:23:58.338302828 +0000 UTC m=+3917.763909282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials") pod "payload-processing-external-model-7578fcd544-t7thb" (UID: "e381f68e-d9a6-4a17-9e0a-2a0daa958cc3") : secret "agc-cred-3f45679e4378" not found
Nov 03 02:23:54.551264 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:54.551189    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb" podStartSLOduration=5.55116865 podStartE2EDuration="5.55116865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:50.777634185 +0000 UTC m=+3910.203240657" watchObservedRunningTime="2026-11-03 02:23:54.55116865 +0000 UTC m=+3913.976775110"
Nov 03 02:23:54.552703 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:54.552654    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"]
Nov 03 02:23:54.559389 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:54.559368    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"
Nov 03 02:23:54.643609 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:54.643524    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de8b9a97-5095-4a9a-a64e-6053fdf408ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr\" (UID: \"de8b9a97-5095-4a9a-a64e-6053fdf408ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"
Nov 03 02:23:54.643609 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:54.643584    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrpd2\" (UniqueName: \"kubernetes.io/projected/de8b9a97-5095-4a9a-a64e-6053fdf408ca-kube-api-access-jrpd2\") pod \"payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr\" (UID: \"de8b9a97-5095-4a9a-a64e-6053fdf408ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"
Nov 03 02:23:54.654817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:54.654736    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d6fbd7\""
Nov 03 02:23:54.654817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:54.654763    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d6fbd7-dockercfg-6x4cw\""
Nov 03 02:23:54.658136 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:54.658109    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"]
Nov 03 02:23:54.745080 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:54.745050    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de8b9a97-5095-4a9a-a64e-6053fdf408ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr\" (UID: \"de8b9a97-5095-4a9a-a64e-6053fdf408ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"
Nov 03 02:23:54.745267 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:54.745113    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrpd2\" (UniqueName: \"kubernetes.io/projected/de8b9a97-5095-4a9a-a64e-6053fdf408ca-kube-api-access-jrpd2\") pod \"payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr\" (UID: \"de8b9a97-5095-4a9a-a64e-6053fdf408ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"
Nov 03 02:23:54.745982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:54.745957    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de8b9a97-5095-4a9a-a64e-6053fdf408ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr\" (UID: \"de8b9a97-5095-4a9a-a64e-6053fdf408ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"
Nov 03 02:23:54.802915 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:54.802882    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrpd2\" (UniqueName: \"kubernetes.io/projected/de8b9a97-5095-4a9a-a64e-6053fdf408ca-kube-api-access-jrpd2\") pod \"payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr\" (UID: \"de8b9a97-5095-4a9a-a64e-6053fdf408ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"
Nov 03 02:23:54.879842 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:54.879808    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"
Nov 03 02:23:55.256077 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:55.256042    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"]
Nov 03 02:23:55.263407 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:23:55.263374    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8b9a97_5095_4a9a_a64e_6053fdf408ca.slice/crio-3220a3603676794d2d45e4c2aa75fb9479ddb21ec11f99f34ae830da3d52a9a3 WatchSource:0}: Error finding container 3220a3603676794d2d45e4c2aa75fb9479ddb21ec11f99f34ae830da3d52a9a3: Status 404 returned error can't find the container with id 3220a3603676794d2d45e4c2aa75fb9479ddb21ec11f99f34ae830da3d52a9a3
Nov 03 02:23:55.277506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:55.277476    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"]
Nov 03 02:23:55.280868 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:55.280850    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:23:55.691961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:55.691924    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr" event={"ID":"de8b9a97-5095-4a9a-a64e-6053fdf408ca","Type":"ContainerStarted","Data":"76a6f6a12d3fe9cb58a48d573949644daf976ae048da995f4345e162f035ba4b"}
Nov 03 02:23:55.692420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:55.691993    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"
Nov 03 02:23:55.692420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:55.692016    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr" event={"ID":"de8b9a97-5095-4a9a-a64e-6053fdf408ca","Type":"ContainerStarted","Data":"3220a3603676794d2d45e4c2aa75fb9479ddb21ec11f99f34ae830da3d52a9a3"}
Nov 03 02:23:55.804260 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:55.804195    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr" podStartSLOduration=1.804180465 podStartE2EDuration="1.804180465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:23:55.800400407 +0000 UTC m=+3915.226006897" watchObservedRunningTime="2026-11-03 02:23:55.804180465 +0000 UTC m=+3915.229786930"
Nov 03 02:23:59.648880 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:59.648849    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"
Nov 03 02:23:59.649393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:23:59.649004    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"
Nov 03 02:24:00.278493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.278406    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb" podUID="e381f68e-d9a6-4a17-9e0a-2a0daa958cc3" containerName="payload-processing" containerID="cri-o://ffe1857f847a92b710434a0911d722b3f3eb46137235c2382868ce40fa6d9ec5" gracePeriod=25
Nov 03 02:24:00.554990 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.554964    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:24:00.643574 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.643539    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-routing-overlay-volume\") pod \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") "
Nov 03 02:24:00.643763 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.643620    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-kube-api-access-mtdp8\" (UniqueName: \"kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-kube-api-access-mtdp8\") pod \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") "
Nov 03 02:24:00.643763 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.643646    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials\") pod \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") "
Nov 03 02:24:00.643763 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.643715    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-plugins-config-volume\") pod \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\" (UID: \"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3\") "
Nov 03 02:24:00.644079 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.643981    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-routing-overlay-volume" pod "e381f68e-d9a6-4a17-9e0a-2a0daa958cc3" (UID: "e381f68e-d9a6-4a17-9e0a-2a0daa958cc3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:24:00.644195 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.644172    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-plugins-config-volume" pod "e381f68e-d9a6-4a17-9e0a-2a0daa958cc3" (UID: "e381f68e-d9a6-4a17-9e0a-2a0daa958cc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:24:00.644397 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.644379    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:24:00.644498 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.644399    5508 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-routing-overlay-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:24:00.645993 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.645966    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials" pod "e381f68e-d9a6-4a17-9e0a-2a0daa958cc3" (UID: "e381f68e-d9a6-4a17-9e0a-2a0daa958cc3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:24:00.645993 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.645980    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-kube-api-access-mtdp8" pod "e381f68e-d9a6-4a17-9e0a-2a0daa958cc3" (UID: "e381f68e-d9a6-4a17-9e0a-2a0daa958cc3"). InnerVolumeSpecName "kube-api-access-mtdp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:24:00.727928 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.727898    5508 generic.go:356] "Generic (PLEG): container finished" podID="e381f68e-d9a6-4a17-9e0a-2a0daa958cc3" containerID="ffe1857f847a92b710434a0911d722b3f3eb46137235c2382868ce40fa6d9ec5" exitCode=0
Nov 03 02:24:00.728317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.727941    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"
Nov 03 02:24:00.728317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.727942    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb" event={"ID":"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3","Type":"ContainerDied","Data":"ffe1857f847a92b710434a0911d722b3f3eb46137235c2382868ce40fa6d9ec5"}
Nov 03 02:24:00.728317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.728062    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7578fcd544-t7thb" event={"ID":"e381f68e-d9a6-4a17-9e0a-2a0daa958cc3","Type":"ContainerDied","Data":"f99de20b671f1b2d7b45f78b3cc79e9ca83a4ca850b11aae6a590ec7bc901a94"}
Nov 03 02:24:00.728317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.728091    5508 scope.go:122] "RemoveContainer" containerID="ffe1857f847a92b710434a0911d722b3f3eb46137235c2382868ce40fa6d9ec5"
Nov 03 02:24:00.738226 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.738205    5508 scope.go:122] "RemoveContainer" containerID="ffe1857f847a92b710434a0911d722b3f3eb46137235c2382868ce40fa6d9ec5"
Nov 03 02:24:00.738490 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:00.738462    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe1857f847a92b710434a0911d722b3f3eb46137235c2382868ce40fa6d9ec5\": container with ID starting with ffe1857f847a92b710434a0911d722b3f3eb46137235c2382868ce40fa6d9ec5 not found: ID does not exist" containerID="ffe1857f847a92b710434a0911d722b3f3eb46137235c2382868ce40fa6d9ec5"
Nov 03 02:24:00.738588 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.738498    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe1857f847a92b710434a0911d722b3f3eb46137235c2382868ce40fa6d9ec5"} err="failed to get container status \"ffe1857f847a92b710434a0911d722b3f3eb46137235c2382868ce40fa6d9ec5\": rpc error: code = NotFound desc = could not find container \"ffe1857f847a92b710434a0911d722b3f3eb46137235c2382868ce40fa6d9ec5\": container with ID starting with ffe1857f847a92b710434a0911d722b3f3eb46137235c2382868ce40fa6d9ec5 not found: ID does not exist"
Nov 03 02:24:00.746108 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.745528    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtdp8\" (UniqueName: \"kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-kube-api-access-mtdp8\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:24:00.746108 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.745558    5508 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3-provider-credentials\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:24:00.885447 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.885357    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"]
Nov 03 02:24:00.891401 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:00.891373    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7578fcd544-t7thb"]
Nov 03 02:24:02.693479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:02.693449    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e381f68e-d9a6-4a17-9e0a-2a0daa958cc3" path="/var/lib/kubelet/pods/e381f68e-d9a6-4a17-9e0a-2a0daa958cc3/volumes"
Nov 03 02:24:03.960126 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:03.960090    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"]
Nov 03 02:24:03.961177 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:03.961152    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e381f68e-d9a6-4a17-9e0a-2a0daa958cc3" containerName="payload-processing"
Nov 03 02:24:03.961284 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:03.961177    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e381f68e-d9a6-4a17-9e0a-2a0daa958cc3" containerName="payload-processing"
Nov 03 02:24:03.961362 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:03.961309    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e381f68e-d9a6-4a17-9e0a-2a0daa958cc3" containerName="payload-processing"
Nov 03 02:24:03.966887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:03.966861    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:04.025308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.025269    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"]
Nov 03 02:24:04.063693 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.063658    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3f45679e4378\""
Nov 03 02:24:04.064563 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.064520    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mff27\""
Nov 03 02:24:04.064777 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.064622    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 02:24:04.064777 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.064650    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 02:24:04.064777 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.064699    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 02:24:04.064962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.064759    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 02:24:04.096618 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.096580    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-routing-overlay-volume\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:04.096875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.096634    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:04.096875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.096782    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh4sk\" (UniqueName: \"kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-kube-api-access-dh4sk\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:04.096875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.096851    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-plugins-config-volume\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:04.198461 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.198426    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-routing-overlay-volume\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:04.198678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.198466    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:04.198678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.198521    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh4sk\" (UniqueName: \"kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-kube-api-access-dh4sk\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:04.198678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.198565    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-plugins-config-volume\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:04.198678 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:04.198667    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3f45679e4378: secret "agc-cred-3f45679e4378" not found
Nov 03 02:24:04.198914 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:04.198691    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz: secret "agc-cred-3f45679e4378" not found
Nov 03 02:24:04.198914 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:04.198777    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials podName:4c7c4145-4696-4ea4-ba68-99aebbc4d42a nodeName:}" failed. No retries permitted until 2026-11-03 02:24:04.698753078 +0000 UTC m=+3924.124359532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials") pod "payload-processing-external-model-7578fcd544-wlnwz" (UID: "4c7c4145-4696-4ea4-ba68-99aebbc4d42a") : secret "agc-cred-3f45679e4378" not found
Nov 03 02:24:04.199248 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.199227    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-routing-overlay-volume\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:04.199401 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.199339    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-plugins-config-volume\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:04.258474 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.258401    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh4sk\" (UniqueName: \"kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-kube-api-access-dh4sk\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:04.706456 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:04.706411    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:04.706642 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:04.706548    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3f45679e4378: secret "agc-cred-3f45679e4378" not found
Nov 03 02:24:04.706642 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:04.706567    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz: secret "agc-cred-3f45679e4378" not found
Nov 03 02:24:04.706642 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:04.706629    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials podName:4c7c4145-4696-4ea4-ba68-99aebbc4d42a nodeName:}" failed. No retries permitted until 2026-11-03 02:24:05.706614877 +0000 UTC m=+3925.132221321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials") pod "payload-processing-external-model-7578fcd544-wlnwz" (UID: "4c7c4145-4696-4ea4-ba68-99aebbc4d42a") : secret "agc-cred-3f45679e4378" not found
Nov 03 02:24:05.724311 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:05.724263    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:05.724743 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:05.724429    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3f45679e4378: secret "agc-cred-3f45679e4378" not found
Nov 03 02:24:05.724743 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:05.724451    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz: secret "agc-cred-3f45679e4378" not found
Nov 03 02:24:05.724743 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:05.724509    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials podName:4c7c4145-4696-4ea4-ba68-99aebbc4d42a nodeName:}" failed. No retries permitted until 2026-11-03 02:24:07.724493763 +0000 UTC m=+3927.150100206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials") pod "payload-processing-external-model-7578fcd544-wlnwz" (UID: "4c7c4145-4696-4ea4-ba68-99aebbc4d42a") : secret "agc-cred-3f45679e4378" not found
Nov 03 02:24:06.192034 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.191995    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"]
Nov 03 02:24:06.196823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.196800    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.231171 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.231135    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-credential-socket\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.231383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.231252    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-socket\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.231383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.231282    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-token\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.231383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.231318    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-certs\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.231564 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.231543    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb908f8a-7347-4ef4-b66c-b43b313cf509-istiod-ca-cert\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.231600 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.231591    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-data\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.231635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.231608    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-podinfo\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.231690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.231676    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-envoy\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.231725 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.231700    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz84c\" (UniqueName: \"kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-kube-api-access-kz84c\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.245938 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.245907    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c56865-openshift-default-dockercfg-wdwlt\""
Nov 03 02:24:06.300045 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.300006    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"]
Nov 03 02:24:06.333287 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.333257    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb908f8a-7347-4ef4-b66c-b43b313cf509-istiod-ca-cert\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.333528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.333302    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-data\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.333528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.333329    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-podinfo\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.333528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.333397    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-envoy\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.333528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.333423    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz84c\" (UniqueName: \"kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-kube-api-access-kz84c\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.333528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.333459    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-credential-socket\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.333528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.333523    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-socket\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.333931 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.333549    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-token\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.333931 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.333596    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-certs\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.333931 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.333728    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-data\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.334096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.333980    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-certs\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.334154 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.334132    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb908f8a-7347-4ef4-b66c-b43b313cf509-istiod-ca-cert\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.334233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.334214    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-credential-socket\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.334419 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.334395    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-socket\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.337045 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.336923    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-podinfo\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.337045 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.336992    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-envoy\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.421145 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.421108    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz84c\" (UniqueName: \"kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-kube-api-access-kz84c\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.429694 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.425966    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-token\") pod \"e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") " pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.513292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.513197    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:06.704641 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.704612    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"
Nov 03 02:24:06.946858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.946818    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"]
Nov 03 02:24:06.953255 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:24:06.953213    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb908f8a_7347_4ef4_b66c_b43b313cf509.slice/crio-2af3ceddf290ff3c1250da9c948c6ddad3b31f78ad2bc4811be633cb6b59b6be WatchSource:0}: Error finding container 2af3ceddf290ff3c1250da9c948c6ddad3b31f78ad2bc4811be633cb6b59b6be: Status 404 returned error can't find the container with id 2af3ceddf290ff3c1250da9c948c6ddad3b31f78ad2bc4811be633cb6b59b6be
Nov 03 02:24:06.956621 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.956587    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:24:06.956751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.956676    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:24:06.956751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.956722    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:24:06.959357 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:06.959308    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"]
Nov 03 02:24:06.959814 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:06.959784    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz" podUID="4c7c4145-4696-4ea4-ba68-99aebbc4d42a"
Nov 03 02:24:07.757579 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.757546    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials\") pod \"payload-processing-external-model-7578fcd544-wlnwz\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") " pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:07.772029 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:07.758532    5508 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3f45679e4378: secret "agc-cred-3f45679e4378" not found
Nov 03 02:24:07.772288 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:07.772267    5508 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz: secret "agc-cred-3f45679e4378" not found
Nov 03 02:24:07.772495 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:07.772482    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials podName:4c7c4145-4696-4ea4-ba68-99aebbc4d42a nodeName:}" failed. No retries permitted until 2026-11-03 02:24:11.77245861 +0000 UTC m=+3931.198065073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials") pod "payload-processing-external-model-7578fcd544-wlnwz" (UID: "4c7c4145-4696-4ea4-ba68-99aebbc4d42a") : secret "agc-cred-3f45679e4378" not found
Nov 03 02:24:07.811396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.809872    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:07.811396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.810316    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" event={"ID":"eb908f8a-7347-4ef4-b66c-b43b313cf509","Type":"ContainerStarted","Data":"ce84032d70badf666dba728e6251c100de1a78080fb7b194f46a347c779c1c45"}
Nov 03 02:24:07.811396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.810384    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" event={"ID":"eb908f8a-7347-4ef4-b66c-b43b313cf509","Type":"ContainerStarted","Data":"2af3ceddf290ff3c1250da9c948c6ddad3b31f78ad2bc4811be633cb6b59b6be"}
Nov 03 02:24:07.820103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.820077    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:07.858803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.858714    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-routing-overlay-volume\") pod \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") "
Nov 03 02:24:07.859002 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.858869    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-plugins-config-volume\") pod \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") "
Nov 03 02:24:07.859002 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.858906    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-kube-api-access-dh4sk\" (UniqueName: \"kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-kube-api-access-dh4sk\") pod \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\" (UID: \"4c7c4145-4696-4ea4-ba68-99aebbc4d42a\") "
Nov 03 02:24:07.859170 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.859072    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-routing-overlay-volume" pod "4c7c4145-4696-4ea4-ba68-99aebbc4d42a" (UID: "4c7c4145-4696-4ea4-ba68-99aebbc4d42a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:24:07.859319 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.859241    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-plugins-config-volume" pod "4c7c4145-4696-4ea4-ba68-99aebbc4d42a" (UID: "4c7c4145-4696-4ea4-ba68-99aebbc4d42a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:24:07.859781 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.859762    5508 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-routing-overlay-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:24:07.859879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.859783    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:24:07.861624 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.861603    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-kube-api-access-dh4sk" pod "4c7c4145-4696-4ea4-ba68-99aebbc4d42a" (UID: "4c7c4145-4696-4ea4-ba68-99aebbc4d42a"). InnerVolumeSpecName "kube-api-access-dh4sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:24:07.927316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.927247    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" podStartSLOduration=1.927227461 podStartE2EDuration="1.927227461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:07.925156936 +0000 UTC m=+3927.350763403" watchObservedRunningTime="2026-11-03 02:24:07.927227461 +0000 UTC m=+3927.352833928"
Nov 03 02:24:07.961008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:07.960970    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh4sk\" (UniqueName: \"kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-kube-api-access-dh4sk\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:24:08.513760 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:08.513724    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:08.520214 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:08.520187    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:08.816409 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:08.816297    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"
Nov 03 02:24:08.816794 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:08.816760    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:08.817906 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:08.817878    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:24:09.151379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:09.151304    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"]
Nov 03 02:24:09.155586 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:09.155558    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7578fcd544-wlnwz"]
Nov 03 02:24:09.182157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:09.182119    5508 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c7c4145-4696-4ea4-ba68-99aebbc4d42a-provider-credentials\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:24:10.693423 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:10.693391    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c7c4145-4696-4ea4-ba68-99aebbc4d42a" path="/var/lib/kubelet/pods/4c7c4145-4696-4ea4-ba68-99aebbc4d42a/volumes"
Nov 03 02:24:13.814545 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:13.814512    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:24:14.308090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.308049    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"]
Nov 03 02:24:14.312597 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.312571    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"
Nov 03 02:24:14.361274 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.361233    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c56865\""
Nov 03 02:24:14.407735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.407699    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c56865-dockercfg-wlf6h\""
Nov 03 02:24:14.413314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.413055    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"]
Nov 03 02:24:14.480370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.478175    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxrmh\" (UniqueName: \"kubernetes.io/projected/8a750072-53df-42b4-94ea-0a30718c7207-kube-api-access-zxrmh\") pod \"payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4\" (UID: \"8a750072-53df-42b4-94ea-0a30718c7207\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"
Nov 03 02:24:14.480370 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.478258    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a750072-53df-42b4-94ea-0a30718c7207-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4\" (UID: \"8a750072-53df-42b4-94ea-0a30718c7207\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"
Nov 03 02:24:14.522934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.522824    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"]
Nov 03 02:24:14.531082 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.531045    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"
Nov 03 02:24:14.552390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.543818    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"]
Nov 03 02:24:14.579770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.579668    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxrmh\" (UniqueName: \"kubernetes.io/projected/8a750072-53df-42b4-94ea-0a30718c7207-kube-api-access-zxrmh\") pod \"payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4\" (UID: \"8a750072-53df-42b4-94ea-0a30718c7207\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"
Nov 03 02:24:14.579770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.579747    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a750072-53df-42b4-94ea-0a30718c7207-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4\" (UID: \"8a750072-53df-42b4-94ea-0a30718c7207\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"
Nov 03 02:24:14.580796 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.580768    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a750072-53df-42b4-94ea-0a30718c7207-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4\" (UID: \"8a750072-53df-42b4-94ea-0a30718c7207\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"
Nov 03 02:24:14.637181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.637150    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxrmh\" (UniqueName: \"kubernetes.io/projected/8a750072-53df-42b4-94ea-0a30718c7207-kube-api-access-zxrmh\") pod \"payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4\" (UID: \"8a750072-53df-42b4-94ea-0a30718c7207\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"
Nov 03 02:24:14.680671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.680632    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4s7k\" (UniqueName: \"kubernetes.io/projected/0a945f17-b9a9-4bf8-93a9-497c58ef528f-kube-api-access-x4s7k\") pod \"payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj\" (UID: \"0a945f17-b9a9-4bf8-93a9-497c58ef528f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"
Nov 03 02:24:14.680878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.680695    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a945f17-b9a9-4bf8-93a9-497c58ef528f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj\" (UID: \"0a945f17-b9a9-4bf8-93a9-497c58ef528f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"
Nov 03 02:24:14.782612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.782561    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a945f17-b9a9-4bf8-93a9-497c58ef528f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj\" (UID: \"0a945f17-b9a9-4bf8-93a9-497c58ef528f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"
Nov 03 02:24:14.782867 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.782850    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4s7k\" (UniqueName: \"kubernetes.io/projected/0a945f17-b9a9-4bf8-93a9-497c58ef528f-kube-api-access-x4s7k\") pod \"payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj\" (UID: \"0a945f17-b9a9-4bf8-93a9-497c58ef528f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"
Nov 03 02:24:14.783316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.783294    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a945f17-b9a9-4bf8-93a9-497c58ef528f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj\" (UID: \"0a945f17-b9a9-4bf8-93a9-497c58ef528f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"
Nov 03 02:24:14.840747 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.840676    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4s7k\" (UniqueName: \"kubernetes.io/projected/0a945f17-b9a9-4bf8-93a9-497c58ef528f-kube-api-access-x4s7k\") pod \"payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj\" (UID: \"0a945f17-b9a9-4bf8-93a9-497c58ef528f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"
Nov 03 02:24:14.848674 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.848646    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"
Nov 03 02:24:14.926259 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:14.926215    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"
Nov 03 02:24:15.239720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:15.239688    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"]
Nov 03 02:24:15.241042 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:24:15.241016    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a945f17_b9a9_4bf8_93a9_497c58ef528f.slice/crio-6b262e8db75fda935819943c27e1e06cc3a082ae2b01e80f90d852ac63b6238a WatchSource:0}: Error finding container 6b262e8db75fda935819943c27e1e06cc3a082ae2b01e80f90d852ac63b6238a: Status 404 returned error can't find the container with id 6b262e8db75fda935819943c27e1e06cc3a082ae2b01e80f90d852ac63b6238a
Nov 03 02:24:15.341774 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:15.341741    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"]
Nov 03 02:24:15.348129 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:24:15.348102    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a750072_53df_42b4_94ea_0a30718c7207.slice/crio-48fb3b56499f26545914a068eacc9258d7b83c2d1279b1e4d7aca1e91477842c WatchSource:0}: Error finding container 48fb3b56499f26545914a068eacc9258d7b83c2d1279b1e4d7aca1e91477842c: Status 404 returned error can't find the container with id 48fb3b56499f26545914a068eacc9258d7b83c2d1279b1e4d7aca1e91477842c
Nov 03 02:24:15.867228 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:15.867191    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj" event={"ID":"0a945f17-b9a9-4bf8-93a9-497c58ef528f","Type":"ContainerStarted","Data":"4e854503415f4a0c5986b67eca564b281c250e66c746ff86d05ec42a7f6cba8b"}
Nov 03 02:24:15.867759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:15.867256    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj" event={"ID":"0a945f17-b9a9-4bf8-93a9-497c58ef528f","Type":"ContainerStarted","Data":"6b262e8db75fda935819943c27e1e06cc3a082ae2b01e80f90d852ac63b6238a"}
Nov 03 02:24:15.867759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:15.867308    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"
Nov 03 02:24:15.869000 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:15.868976    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4" event={"ID":"8a750072-53df-42b4-94ea-0a30718c7207","Type":"ContainerStarted","Data":"505b37cd5d13ce758d0a75f6569337bf76fe52123dd07f15be71bab140946284"}
Nov 03 02:24:15.869154 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:15.869022    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4" event={"ID":"8a750072-53df-42b4-94ea-0a30718c7207","Type":"ContainerStarted","Data":"48fb3b56499f26545914a068eacc9258d7b83c2d1279b1e4d7aca1e91477842c"}
Nov 03 02:24:15.869154 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:15.869058    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"
Nov 03 02:24:15.978010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:15.977956    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj" podStartSLOduration=1.977940286 podStartE2EDuration="1.977940286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:15.974893219 +0000 UTC m=+3935.400499697" watchObservedRunningTime="2026-11-03 02:24:15.977940286 +0000 UTC m=+3935.403546751"
Nov 03 02:24:16.105377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:16.099596    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4" podStartSLOduration=2.099570938 podStartE2EDuration="2.099570938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:16.086791693 +0000 UTC m=+3935.512398159" watchObservedRunningTime="2026-11-03 02:24:16.099570938 +0000 UTC m=+3935.525177403"
Nov 03 02:24:19.812685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:19.812649    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:24:20.191680 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.191646    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"]
Nov 03 02:24:20.197590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.197553    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.273127 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.273088    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.273391 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.273138    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.273391 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.273261    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b767dd-0d0d-4736-931c-29debe051aaf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.273391 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.273388    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.273597 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.273446    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kmvk\" (UniqueName: \"kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-kube-api-access-2kmvk\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.273597 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.273483    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.273597 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.273528    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.273597 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.273565    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.273597 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.273592    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.374658 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.374618    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.374874 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.374668    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.374874 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.374789    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b767dd-0d0d-4736-931c-29debe051aaf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.375011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.374879    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.375011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.374957    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kmvk\" (UniqueName: \"kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-kube-api-access-2kmvk\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.375153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.375018    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.375153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.375084    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.375153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.375089    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.375153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.375125    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.375373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.375160    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.376137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.375815    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.376137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.375883    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.376137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.376031    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.376400 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.376136    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b767dd-0d0d-4736-931c-29debe051aaf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.377824 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.377802    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.378388 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.378313    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.398528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.398496    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"]
Nov 03 02:24:20.433111 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.433061    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.433409 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.433386    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kmvk\" (UniqueName: \"kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-kube-api-access-2kmvk\") pod \"maas-default-gateway-openshift-default-59f957fdd7-5t2x2\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.514780 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.514696    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:20.960093 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.960048    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"]
Nov 03 02:24:20.966814 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:24:20.966786    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b767dd_0d0d_4736_931c_29debe051aaf.slice/crio-310fb2e07a551c327659edbce07cbc1f0b3f4a6c182df41c12869ffa9f4b7bbe WatchSource:0}: Error finding container 310fb2e07a551c327659edbce07cbc1f0b3f4a6c182df41c12869ffa9f4b7bbe: Status 404 returned error can't find the container with id 310fb2e07a551c327659edbce07cbc1f0b3f4a6c182df41c12869ffa9f4b7bbe
Nov 03 02:24:20.970208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.970173    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:24:20.970314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.970246    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:24:20.970314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:20.970280    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:24:21.917626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:21.917584    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2" event={"ID":"d9b767dd-0d0d-4736-931c-29debe051aaf","Type":"ContainerStarted","Data":"a66cecf7c1fcdd2541a17f766f60f08e86e29b9bf66135485d57f91989ebe2f4"}
Nov 03 02:24:21.917832 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:21.917649    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2" event={"ID":"d9b767dd-0d0d-4736-931c-29debe051aaf","Type":"ContainerStarted","Data":"310fb2e07a551c327659edbce07cbc1f0b3f4a6c182df41c12869ffa9f4b7bbe"}
Nov 03 02:24:22.090390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:22.089710    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2" podStartSLOduration=2.08969439 podStartE2EDuration="2.08969439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:22.085503955 +0000 UTC m=+3941.511110431" watchObservedRunningTime="2026-11-03 02:24:22.08969439 +0000 UTC m=+3941.515300854"
Nov 03 02:24:22.515045 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:22.515004    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:22.517154 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:22.517130    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-5t2x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 03 02:24:22.517241 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:22.517181    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2" podUID="d9b767dd-0d0d-4736-931c-29debe051aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 03 02:24:23.515196 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:23.515165    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-5t2x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 03 02:24:23.515720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:23.515690    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2" podUID="d9b767dd-0d0d-4736-931c-29debe051aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 03 02:24:24.515316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:24.515270    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-5t2x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 03 02:24:24.515776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:24.515364    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2" podUID="d9b767dd-0d0d-4736-931c-29debe051aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 03 02:24:25.515532 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:25.515484    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-5t2x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 03 02:24:25.515937 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:25.515560    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2" podUID="d9b767dd-0d0d-4736-931c-29debe051aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 03 02:24:26.515407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:26.515368    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-5t2x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 03 02:24:26.515637 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:26.515439    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2" podUID="d9b767dd-0d0d-4736-931c-29debe051aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 03 02:24:26.880333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:26.880254    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"
Nov 03 02:24:26.880831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:26.880813    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"
Nov 03 02:24:27.515451 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:27.515413    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-5t2x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 03 02:24:27.515683 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:27.515476    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2" podUID="d9b767dd-0d0d-4736-931c-29debe051aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 03 02:24:28.515640 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:28.515596    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-5t2x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused" start-of-body=
Nov 03 02:24:28.515836 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:28.515669    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2" podUID="d9b767dd-0d0d-4736-931c-29debe051aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.68:15021/healthz/ready\": dial tcp 10.133.1.68:15021: connect: connection refused"
Nov 03 02:24:29.965454 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:29.957268    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:29.965454 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:29.960090    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:29.966021 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:29.965712    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:24:43.813664 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:43.813625    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:24:50.434339 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.434296    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"]
Nov 03 02:24:50.441336 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.441311    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.534146 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.534108    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"]
Nov 03 02:24:50.613647 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.613602    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsz72\" (UniqueName: \"kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-kube-api-access-vsz72\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.613854 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.613673    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.613854 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.613718    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.613854 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.613743    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.614087 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.613901    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.614087 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.613948    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.614209 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.614123    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.614209 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.614178    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.614282 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.614229    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.715489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.715402    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.715489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.715451    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.715743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.715529    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.715743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.715558    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.715743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.715576    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.715743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.715638    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsz72\" (UniqueName: \"kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-kube-api-access-vsz72\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.715743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.715694    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.715743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.715726    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.716137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.715750    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.716137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.715953    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.716137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.715958    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.716291 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.716164    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.716291 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.716220    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.716540 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.716509    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.718449 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.718427    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.718602 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.718583    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.773308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.773277    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsz72\" (UniqueName: \"kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-kube-api-access-vsz72\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:50.773515 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:50.773493    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-c4d57\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:51.056189 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:51.056102    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:51.501717 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:51.501683    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"]
Nov 03 02:24:51.508043 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:24:51.508011    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4194050_7f6a_49eb_ade8_fdd65ce6603c.slice/crio-6ac693e038e6750a33f86e5aebd41336061629ee46802d1cbd83e663ea587a9c WatchSource:0}: Error finding container 6ac693e038e6750a33f86e5aebd41336061629ee46802d1cbd83e663ea587a9c: Status 404 returned error can't find the container with id 6ac693e038e6750a33f86e5aebd41336061629ee46802d1cbd83e663ea587a9c
Nov 03 02:24:51.510748 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:51.510711    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:24:51.510839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:51.510784    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:24:51.510839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:51.510813    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:24:52.278424 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:52.278386    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" event={"ID":"a4194050-7f6a-49eb-ade8-fdd65ce6603c","Type":"ContainerStarted","Data":"77a64726ad3c51be97e98419c6cd9527f3e402f227bda1a5d2d25400ed628b3b"}
Nov 03 02:24:52.278767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:52.278454    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" event={"ID":"a4194050-7f6a-49eb-ade8-fdd65ce6603c","Type":"ContainerStarted","Data":"6ac693e038e6750a33f86e5aebd41336061629ee46802d1cbd83e663ea587a9c"}
Nov 03 02:24:52.442129 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:52.442074    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" podStartSLOduration=2.442059468 podStartE2EDuration="2.442059468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:24:52.439155059 +0000 UTC m=+3971.864761535" watchObservedRunningTime="2026-11-03 02:24:52.442059468 +0000 UTC m=+3971.867665932"
Nov 03 02:24:53.064372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.062410    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:24:53.064372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.062596    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-c4d57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Nov 03 02:24:53.064372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.062650    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" podUID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Nov 03 02:24:53.416956 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.416917    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"]
Nov 03 02:24:53.417194 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.417170    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w" podUID="beb90e6f-497c-4dc4-80c1-05aa76a82dbd" containerName="maas-api" containerID="cri-o://51aaa1c0c67c5930a92594766c1c520441b01a73be168555210c5094bac2c64a" gracePeriod=30
Nov 03 02:24:53.886122 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.885747    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:24:53.909713 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.909671    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-metrics-certs\") pod \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") "
Nov 03 02:24:53.910077 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.910055    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-maas-api-tls\") pod \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") "
Nov 03 02:24:53.911753 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.910297    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-openshift-service-ca\") pod \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") "
Nov 03 02:24:53.911753 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.910337    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-kube-api-access-jdn57\" (UniqueName: \"kubernetes.io/projected/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-kube-api-access-jdn57\") pod \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\" (UID: \"beb90e6f-497c-4dc4-80c1-05aa76a82dbd\") "
Nov 03 02:24:53.913298 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.913269    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-openshift-service-ca" pod "beb90e6f-497c-4dc4-80c1-05aa76a82dbd" (UID: "beb90e6f-497c-4dc4-80c1-05aa76a82dbd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:24:53.916190 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.916166    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-kube-api-access-jdn57" pod "beb90e6f-497c-4dc4-80c1-05aa76a82dbd" (UID: "beb90e6f-497c-4dc4-80c1-05aa76a82dbd"). InnerVolumeSpecName "kube-api-access-jdn57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:24:53.916543 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.916516    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-metrics-certs" pod "beb90e6f-497c-4dc4-80c1-05aa76a82dbd" (UID: "beb90e6f-497c-4dc4-80c1-05aa76a82dbd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:24:53.916871 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:53.916838    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-maas-api-tls" pod "beb90e6f-497c-4dc4-80c1-05aa76a82dbd" (UID: "beb90e6f-497c-4dc4-80c1-05aa76a82dbd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:24:54.013329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.013087    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:24:54.013329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.013122    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:24:54.013329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.013138    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:24:54.013329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.013154    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdn57\" (UniqueName: \"kubernetes.io/projected/beb90e6f-497c-4dc4-80c1-05aa76a82dbd-kube-api-access-jdn57\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:24:54.070850 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.070754    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-c4d57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Nov 03 02:24:54.072231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.072197    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" podUID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Nov 03 02:24:54.305385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.305098    5508 generic.go:356] "Generic (PLEG): container finished" podID="beb90e6f-497c-4dc4-80c1-05aa76a82dbd" containerID="51aaa1c0c67c5930a92594766c1c520441b01a73be168555210c5094bac2c64a" exitCode=0
Nov 03 02:24:54.305385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.305173    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w" event={"ID":"beb90e6f-497c-4dc4-80c1-05aa76a82dbd","Type":"ContainerDied","Data":"51aaa1c0c67c5930a92594766c1c520441b01a73be168555210c5094bac2c64a"}
Nov 03 02:24:54.305385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.305259    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"
Nov 03 02:24:54.305385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.305274    5508 scope.go:122] "RemoveContainer" containerID="51aaa1c0c67c5930a92594766c1c520441b01a73be168555210c5094bac2c64a"
Nov 03 02:24:54.305385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.305259    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w" event={"ID":"beb90e6f-497c-4dc4-80c1-05aa76a82dbd","Type":"ContainerDied","Data":"6c8b64a733790a6bcb7efaa5bc8de74477c1eb42a0d15b23ccaa73b400bacb6f"}
Nov 03 02:24:54.329773 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.325069    5508 scope.go:122] "RemoveContainer" containerID="51aaa1c0c67c5930a92594766c1c520441b01a73be168555210c5094bac2c64a"
Nov 03 02:24:54.329773 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:24:54.325563    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51aaa1c0c67c5930a92594766c1c520441b01a73be168555210c5094bac2c64a\": container with ID starting with 51aaa1c0c67c5930a92594766c1c520441b01a73be168555210c5094bac2c64a not found: ID does not exist" containerID="51aaa1c0c67c5930a92594766c1c520441b01a73be168555210c5094bac2c64a"
Nov 03 02:24:54.329773 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.325622    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51aaa1c0c67c5930a92594766c1c520441b01a73be168555210c5094bac2c64a"} err="failed to get container status \"51aaa1c0c67c5930a92594766c1c520441b01a73be168555210c5094bac2c64a\": rpc error: code = NotFound desc = could not find container \"51aaa1c0c67c5930a92594766c1c520441b01a73be168555210c5094bac2c64a\": container with ID starting with 51aaa1c0c67c5930a92594766c1c520441b01a73be168555210c5094bac2c64a not found: ID does not exist"
Nov 03 02:24:54.490372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.489394    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"]
Nov 03 02:24:54.504367 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.504318    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3fb07e-577b7dfd54-4rf8w"]
Nov 03 02:24:54.693714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:54.693676    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb90e6f-497c-4dc4-80c1-05aa76a82dbd" path="/var/lib/kubelet/pods/beb90e6f-497c-4dc4-80c1-05aa76a82dbd/volumes"
Nov 03 02:24:55.058434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:55.057790    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-c4d57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Nov 03 02:24:55.058434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:55.057851    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" podUID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Nov 03 02:24:56.061372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:56.059807    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-c4d57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Nov 03 02:24:56.061372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:56.059873    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" podUID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Nov 03 02:24:57.067382 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:57.059520    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-c4d57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Nov 03 02:24:57.067382 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:57.059595    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" podUID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Nov 03 02:24:58.057557 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:58.057518    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-c4d57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Nov 03 02:24:58.057756 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:58.057581    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" podUID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Nov 03 02:24:59.057167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:59.057124    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-c4d57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused" start-of-body=
Nov 03 02:24:59.057620 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:59.057196    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" podUID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.69:15021/healthz/ready\": dial tcp 10.133.1.69:15021: connect: connection refused"
Nov 03 02:24:59.819714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:59.819680    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"]
Nov 03 02:24:59.820456 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:59.820429    5508 cpu_manager.go:414] "containerMap: removing container" podUID="beb90e6f-497c-4dc4-80c1-05aa76a82dbd" containerName="maas-api"
Nov 03 02:24:59.820701 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:59.820613    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb90e6f-497c-4dc4-80c1-05aa76a82dbd" containerName="maas-api"
Nov 03 02:24:59.820887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:59.820872    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb90e6f-497c-4dc4-80c1-05aa76a82dbd" containerName="maas-api"
Nov 03 02:24:59.825851 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:59.825830    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:24:59.874907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:59.874872    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e051f3-serving-cert\""
Nov 03 02:24:59.907682 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:59.907640    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bbb584-5431-4d36-a98c-e9ed8be3114b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e051f3-7d699948b-spctr\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:24:59.907985 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:59.907958    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-metrics-certs\") pod \"maas-api-e2e-inf-a-e051f3-7d699948b-spctr\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:24:59.908281 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:59.908258    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e051f3-7d699948b-spctr\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:24:59.908544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:59.908522    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c6r8\" (UniqueName: \"kubernetes.io/projected/d5bbb584-5431-4d36-a98c-e9ed8be3114b-kube-api-access-2c6r8\") pod \"maas-api-e2e-inf-a-e051f3-7d699948b-spctr\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:24:59.926919 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:24:59.926886    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"]
Nov 03 02:25:00.010314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:00.010278    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bbb584-5431-4d36-a98c-e9ed8be3114b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e051f3-7d699948b-spctr\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:25:00.010534 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:00.010323    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-metrics-certs\") pod \"maas-api-e2e-inf-a-e051f3-7d699948b-spctr\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:25:00.010534 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:00.010437    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e051f3-7d699948b-spctr\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:25:00.010534 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:00.010518    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c6r8\" (UniqueName: \"kubernetes.io/projected/d5bbb584-5431-4d36-a98c-e9ed8be3114b-kube-api-access-2c6r8\") pod \"maas-api-e2e-inf-a-e051f3-7d699948b-spctr\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:25:00.011119 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:25:00.010596    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-serving-cert: secret "maas-api-e2e-inf-a-e051f3-serving-cert" not found
Nov 03 02:25:00.011119 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:25:00.010665    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-maas-api-tls podName:d5bbb584-5431-4d36-a98c-e9ed8be3114b nodeName:}" failed. No retries permitted until 2026-11-03 02:25:00.510648884 +0000 UTC m=+3979.936255331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-maas-api-tls") pod "maas-api-e2e-inf-a-e051f3-7d699948b-spctr" (UID: "d5bbb584-5431-4d36-a98c-e9ed8be3114b") : secret "maas-api-e2e-inf-a-e051f3-serving-cert" not found
Nov 03 02:25:00.011224 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:00.011131    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bbb584-5431-4d36-a98c-e9ed8be3114b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e051f3-7d699948b-spctr\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:25:00.012997 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:00.012977    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-metrics-certs\") pod \"maas-api-e2e-inf-a-e051f3-7d699948b-spctr\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:25:00.105421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:00.101976    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c6r8\" (UniqueName: \"kubernetes.io/projected/d5bbb584-5431-4d36-a98c-e9ed8be3114b-kube-api-access-2c6r8\") pod \"maas-api-e2e-inf-a-e051f3-7d699948b-spctr\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:25:00.138601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:00.138554    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:25:00.139867 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:00.139835    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:25:00.140786 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:00.140762    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:25:00.533203 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:00.533165    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e051f3-7d699948b-spctr\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:25:00.535642 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:00.535622    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-maas-api-tls\") pod \"maas-api-e2e-inf-a-e051f3-7d699948b-spctr\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:25:00.758404 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:00.758361    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:25:01.141153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:01.141074    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"]
Nov 03 02:25:01.147625 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:25:01.147596    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5bbb584_5431_4d36_a98c_e9ed8be3114b.slice/crio-85b3032493854ebb44449bac861fcb99f4c9e72ab6bea6033ce973022c06094b WatchSource:0}: Error finding container 85b3032493854ebb44449bac861fcb99f4c9e72ab6bea6033ce973022c06094b: Status 404 returned error can't find the container with id 85b3032493854ebb44449bac861fcb99f4c9e72ab6bea6033ce973022c06094b
Nov 03 02:25:01.149167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:01.149145    5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:25:01.362444 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:01.362403    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr" event={"ID":"d5bbb584-5431-4d36-a98c-e9ed8be3114b","Type":"ContainerStarted","Data":"85b3032493854ebb44449bac861fcb99f4c9e72ab6bea6033ce973022c06094b"}
Nov 03 02:25:02.407684 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:02.407647    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr" event={"ID":"d5bbb584-5431-4d36-a98c-e9ed8be3114b","Type":"ContainerStarted","Data":"3df6793e03a5ef3a1d04b62acfebfe7db0b06ba552b93b31ac2029a393163080"}
Nov 03 02:25:02.410087 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:02.408725    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:25:02.523246 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:02.523186    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr" podStartSLOduration=2.595817746 podStartE2EDuration="3.523170948s" totalImagesPullingTime="927.353202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:02.519777531 +0000 UTC m=+3981.945384008" watchObservedRunningTime="2026-11-03 02:25:02.523170948 +0000 UTC m=+3981.948777467"
Nov 03 02:25:09.428365 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:09.428294    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:25:13.837029 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:25:13.836980    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:25:14.054377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:14.053589    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"]
Nov 03 02:25:14.138211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:14.138108    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"
Nov 03 02:25:14.143339 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:14.143302    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"]
Nov 03 02:25:14.219208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:14.219174    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8brf9\" (UniqueName: \"kubernetes.io/projected/22189dc0-97c0-4d3a-95bd-290d6e29d219-kube-api-access-8brf9\") pod \"payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2\" (UID: \"22189dc0-97c0-4d3a-95bd-290d6e29d219\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"
Nov 03 02:25:14.219423 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:14.219305    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22189dc0-97c0-4d3a-95bd-290d6e29d219-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2\" (UID: \"22189dc0-97c0-4d3a-95bd-290d6e29d219\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"
Nov 03 02:25:14.233288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:14.233260    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0b98c1-dockercfg-7dmx7\""
Nov 03 02:25:14.233474 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:14.233458    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0b98c1\""
Nov 03 02:25:14.320111 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:14.320077    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22189dc0-97c0-4d3a-95bd-290d6e29d219-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2\" (UID: \"22189dc0-97c0-4d3a-95bd-290d6e29d219\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"
Nov 03 02:25:14.320287 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:14.320220    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8brf9\" (UniqueName: \"kubernetes.io/projected/22189dc0-97c0-4d3a-95bd-290d6e29d219-kube-api-access-8brf9\") pod \"payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2\" (UID: \"22189dc0-97c0-4d3a-95bd-290d6e29d219\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"
Nov 03 02:25:14.320797 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:14.320778    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22189dc0-97c0-4d3a-95bd-290d6e29d219-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2\" (UID: \"22189dc0-97c0-4d3a-95bd-290d6e29d219\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"
Nov 03 02:25:14.378067 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:14.378032    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8brf9\" (UniqueName: \"kubernetes.io/projected/22189dc0-97c0-4d3a-95bd-290d6e29d219-kube-api-access-8brf9\") pod \"payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2\" (UID: \"22189dc0-97c0-4d3a-95bd-290d6e29d219\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"
Nov 03 02:25:14.459235 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:14.459182    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"
Nov 03 02:25:14.835243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:14.835151    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"]
Nov 03 02:25:14.841615 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:25:14.841581    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22189dc0_97c0_4d3a_95bd_290d6e29d219.slice/crio-ff71a9245747bc9e2f3a1892a24cc24001ba980304c03d4b406df888cf57c46e WatchSource:0}: Error finding container ff71a9245747bc9e2f3a1892a24cc24001ba980304c03d4b406df888cf57c46e: Status 404 returned error can't find the container with id ff71a9245747bc9e2f3a1892a24cc24001ba980304c03d4b406df888cf57c46e
Nov 03 02:25:15.512834 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:15.512797    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2" event={"ID":"22189dc0-97c0-4d3a-95bd-290d6e29d219","Type":"ContainerStarted","Data":"f8760d4872746fa4909a4a43dcbe8c09756c30eb9397bef1288d5d77dc243399"}
Nov 03 02:25:15.513045 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:15.512860    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"
Nov 03 02:25:15.513045 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:15.512878    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2" event={"ID":"22189dc0-97c0-4d3a-95bd-290d6e29d219","Type":"ContainerStarted","Data":"ff71a9245747bc9e2f3a1892a24cc24001ba980304c03d4b406df888cf57c46e"}
Nov 03 02:25:15.623135 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:15.623077    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2" podStartSLOduration=2.62306076 podStartE2EDuration="2.62306076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:15.620720509 +0000 UTC m=+3995.046326972" watchObservedRunningTime="2026-11-03 02:25:15.62306076 +0000 UTC m=+3995.048667224"
Nov 03 02:25:24.203691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:24.203641    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:25:26.523829 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:26.523795    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"
Nov 03 02:25:35.104646 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.104610    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"]
Nov 03 02:25:35.110234 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.110210    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.176676 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.176639    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"]
Nov 03 02:25:35.227524 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.227467    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b6c56b-openshift-default-dockercfg-dv6jv\""
Nov 03 02:25:35.259375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.255298    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-credential-socket\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.259375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.255393    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-envoy\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.259375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.255448    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-socket\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.259375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.255534    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-certs\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.259375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.255579    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-podinfo\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.259375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.255640    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-token\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.259375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.255665    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wljm4\" (UniqueName: \"kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-kube-api-access-wljm4\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.259375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.255756    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46dc7e6-e02b-4201-baad-c3c0764a0216-istiod-ca-cert\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.259375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.255823    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-data\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.378854 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.378761    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46dc7e6-e02b-4201-baad-c3c0764a0216-istiod-ca-cert\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.379150 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.379127    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-data\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.379380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.379363    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-credential-socket\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.379545 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.379530    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-envoy\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.379685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.379672    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-socket\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.379858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.379846    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-certs\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.379981 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.379967    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-podinfo\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.380143 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.380129    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-token\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.380250 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.380233    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wljm4\" (UniqueName: \"kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-kube-api-access-wljm4\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.381419 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.381393    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-data\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.381946 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.381918    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-credential-socket\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.382653 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.382571    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-socket\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.389715 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.386792    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-certs\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.390523 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.390489    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46dc7e6-e02b-4201-baad-c3c0764a0216-istiod-ca-cert\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.401552 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.399432    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-envoy\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.402755 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.402722    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-podinfo\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.437854 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.437816    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-token\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.439749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.439726    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wljm4\" (UniqueName: \"kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-kube-api-access-wljm4\") pod \"e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") " pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:35.724243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:35.724202    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:36.163390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:36.163340    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"]
Nov 03 02:25:36.168942 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:25:36.168907    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46dc7e6_e02b_4201_baad_c3c0764a0216.slice/crio-61c7f5a3c1c0498b4d198e7d4df8b4202e4e455ed054b72f29655f32aa066677 WatchSource:0}: Error finding container 61c7f5a3c1c0498b4d198e7d4df8b4202e4e455ed054b72f29655f32aa066677: Status 404 returned error can't find the container with id 61c7f5a3c1c0498b4d198e7d4df8b4202e4e455ed054b72f29655f32aa066677
Nov 03 02:25:36.171775 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:36.171734    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:25:36.171900 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:36.171805    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:25:36.171900 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:36.171842    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:25:36.816800 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:36.816762    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" event={"ID":"f46dc7e6-e02b-4201-baad-c3c0764a0216","Type":"ContainerStarted","Data":"5e4d80ce27f51c1ddb795eadb643e7c9269c9f3a0a5506341edcfbfc0f838d23"}
Nov 03 02:25:36.817002 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:36.816818    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" event={"ID":"f46dc7e6-e02b-4201-baad-c3c0764a0216","Type":"ContainerStarted","Data":"61c7f5a3c1c0498b4d198e7d4df8b4202e4e455ed054b72f29655f32aa066677"}
Nov 03 02:25:36.979880 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:36.979825    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" podStartSLOduration=1.9798090739999998 podStartE2EDuration="1.979809074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:25:36.978252834 +0000 UTC m=+4016.403859298" watchObservedRunningTime="2026-11-03 02:25:36.979809074 +0000 UTC m=+4016.405415539"
Nov 03 02:25:37.725006 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:37.724971    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:37.731337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:37.731310    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:37.824042 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:37.824008    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:37.825091 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:37.825065    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:25:43.812528 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:25:43.812482    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:25:56.724532 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:25:56.724474    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:26:08.818774 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:08.818698    5508 patch_prober.go:29] interesting pod/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 02:26:08.818774 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:08.818751    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 02:26:09.069737 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:09.069649    5508 generic.go:356] "Generic (PLEG): container finished" podID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerID="ce84032d70badf666dba728e6251c100de1a78080fb7b194f46a347c779c1c45" exitCode=0
Nov 03 02:26:09.069908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:09.069733    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" event={"ID":"eb908f8a-7347-4ef4-b66c-b43b313cf509","Type":"ContainerDied","Data":"ce84032d70badf666dba728e6251c100de1a78080fb7b194f46a347c779c1c45"}
Nov 03 02:26:09.070272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:09.070255    5508 scope.go:122] "RemoveContainer" containerID="ce84032d70badf666dba728e6251c100de1a78080fb7b194f46a347c779c1c45"
Nov 03 02:26:09.072114 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:09.072083    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:26:09.072250 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:09.072169    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:26:09.072250 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:09.072222    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:26:10.080781 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:10.080744    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" event={"ID":"eb908f8a-7347-4ef4-b66c-b43b313cf509","Type":"ContainerStarted","Data":"b843cecde40f1f9e5e38fc4ef6321ab4b3941909f5e0a5e45463556f909dc20e"}
Nov 03 02:26:10.514283 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:10.514242    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:26:10.514497 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:10.514450    5508 patch_prober.go:29] interesting pod/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 02:26:10.514550 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:10.514503    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 02:26:11.514446 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:11.514410    5508 patch_prober.go:29] interesting pod/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 02:26:11.514860 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:11.514465    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 02:26:12.514905 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:12.514869    5508 patch_prober.go:29] interesting pod/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 02:26:12.515443 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:12.514942    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 02:26:13.514605 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:13.514566    5508 patch_prober.go:29] interesting pod/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 02:26:13.514835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:13.514626    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 02:26:13.813626 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:26:13.813545    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:26:14.513839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:14.513803    5508 patch_prober.go:29] interesting pod/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 02:26:14.514072 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:14.513863    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 02:26:15.514140 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:15.514100    5508 patch_prober.go:29] interesting pod/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Nov 03 02:26:15.514615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:15.514162    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Nov 03 02:26:16.519245 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:16.519214    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:26:16.519720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:16.519679    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:26:16.520534 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:16.520506    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:26:37.974918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:37.974880    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"]
Nov 03 02:26:37.980092 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:37.980067    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.034653 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.034622    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0b98c1-serving-cert\""
Nov 03 02:26:38.081760 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.081722    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"]
Nov 03 02:26:38.116385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.116329    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-maas-api-tls\") pod \"maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.116594 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.116494    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-metrics-certs\") pod \"maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.116720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.116691    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljk2\" (UniqueName: \"kubernetes.io/projected/69b6d24e-83f9-4eab-aefb-2b340c12bc55-kube-api-access-sljk2\") pod \"maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.116869 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.116852    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b6d24e-83f9-4eab-aefb-2b340c12bc55-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.217990 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.217948    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-maas-api-tls\") pod \"maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.217990 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.217993    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-metrics-certs\") pod \"maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.218264 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.218068    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljk2\" (UniqueName: \"kubernetes.io/projected/69b6d24e-83f9-4eab-aefb-2b340c12bc55-kube-api-access-sljk2\") pod \"maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.218264 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.218159    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b6d24e-83f9-4eab-aefb-2b340c12bc55-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.218864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.218838    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b6d24e-83f9-4eab-aefb-2b340c12bc55-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.220469 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.220443    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-maas-api-tls\") pod \"maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.220588 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.220559    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-metrics-certs\") pod \"maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.276367 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.276262    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljk2\" (UniqueName: \"kubernetes.io/projected/69b6d24e-83f9-4eab-aefb-2b340c12bc55-kube-api-access-sljk2\") pod \"maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.295321 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.295289    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:38.672870 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:38.672834    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"]
Nov 03 02:26:38.679108 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:26:38.679081    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b6d24e_83f9_4eab_aefb_2b340c12bc55.slice/crio-b8ec48c5decdce51aeb31b0a49bb16f5cb7c2e8524e5b8e6d4442b9775a17bc3 WatchSource:0}: Error finding container b8ec48c5decdce51aeb31b0a49bb16f5cb7c2e8524e5b8e6d4442b9775a17bc3: Status 404 returned error can't find the container with id b8ec48c5decdce51aeb31b0a49bb16f5cb7c2e8524e5b8e6d4442b9775a17bc3
Nov 03 02:26:39.442869 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:39.442828    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk" event={"ID":"69b6d24e-83f9-4eab-aefb-2b340c12bc55","Type":"ContainerStarted","Data":"b8ec48c5decdce51aeb31b0a49bb16f5cb7c2e8524e5b8e6d4442b9775a17bc3"}
Nov 03 02:26:40.452248 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:40.452209    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk" event={"ID":"69b6d24e-83f9-4eab-aefb-2b340c12bc55","Type":"ContainerStarted","Data":"b72cb2cdf1831610fb4d56468afe7c47aba716d8063da7d3f397487a9a894dfb"}
Nov 03 02:26:40.452786 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:40.452442    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:40.563484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:40.563431    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk" podStartSLOduration=2.313373458 podStartE2EDuration="3.563414895s" totalImagesPullingTime="1.250041437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:26:40.562110491 +0000 UTC m=+4079.987716976" watchObservedRunningTime="2026-11-03 02:26:40.563414895 +0000 UTC m=+4079.989021360"
Nov 03 02:26:43.789157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:43.789121    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"]
Nov 03 02:26:43.793677 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:43.793655    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"
Nov 03 02:26:43.817881 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:26:43.817843    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:26:43.900842 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:43.900806    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e40047-dockercfg-j4qrg\""
Nov 03 02:26:43.901208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:43.901191    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e40047\""
Nov 03 02:26:43.934402 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:43.911299    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpkhc\" (UniqueName: \"kubernetes.io/projected/bbc91df0-c89c-45b8-a08e-2f37fd911639-kube-api-access-wpkhc\") pod \"payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl\" (UID: \"bbc91df0-c89c-45b8-a08e-2f37fd911639\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"
Nov 03 02:26:43.934402 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:43.911500    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc91df0-c89c-45b8-a08e-2f37fd911639-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl\" (UID: \"bbc91df0-c89c-45b8-a08e-2f37fd911639\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"
Nov 03 02:26:43.974030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:43.973405    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"]
Nov 03 02:26:44.017069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:44.013502    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpkhc\" (UniqueName: \"kubernetes.io/projected/bbc91df0-c89c-45b8-a08e-2f37fd911639-kube-api-access-wpkhc\") pod \"payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl\" (UID: \"bbc91df0-c89c-45b8-a08e-2f37fd911639\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"
Nov 03 02:26:44.017069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:44.013660    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc91df0-c89c-45b8-a08e-2f37fd911639-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl\" (UID: \"bbc91df0-c89c-45b8-a08e-2f37fd911639\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"
Nov 03 02:26:44.017069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:44.014613    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc91df0-c89c-45b8-a08e-2f37fd911639-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl\" (UID: \"bbc91df0-c89c-45b8-a08e-2f37fd911639\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"
Nov 03 02:26:44.072646 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:44.072558    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpkhc\" (UniqueName: \"kubernetes.io/projected/bbc91df0-c89c-45b8-a08e-2f37fd911639-kube-api-access-wpkhc\") pod \"payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl\" (UID: \"bbc91df0-c89c-45b8-a08e-2f37fd911639\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"
Nov 03 02:26:44.109579 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:44.109539    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"
Nov 03 02:26:44.494108 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:44.494074    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"]
Nov 03 02:26:44.500693 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:26:44.500660    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc91df0_c89c_45b8_a08e_2f37fd911639.slice/crio-f05ba175d141a37f511c64fc64914cb8ed0ba21480e6fd2ca340636b106e061a WatchSource:0}: Error finding container f05ba175d141a37f511c64fc64914cb8ed0ba21480e6fd2ca340636b106e061a: Status 404 returned error can't find the container with id f05ba175d141a37f511c64fc64914cb8ed0ba21480e6fd2ca340636b106e061a
Nov 03 02:26:45.489801 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:45.489763    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl" event={"ID":"bbc91df0-c89c-45b8-a08e-2f37fd911639","Type":"ContainerStarted","Data":"020d55b932e9bce4b009f8ce13fd1307de3d52c1cd898a4bec2aa2ff71a5ad8b"}
Nov 03 02:26:45.490303 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:45.489816    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl" event={"ID":"bbc91df0-c89c-45b8-a08e-2f37fd911639","Type":"ContainerStarted","Data":"f05ba175d141a37f511c64fc64914cb8ed0ba21480e6fd2ca340636b106e061a"}
Nov 03 02:26:45.490303 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:45.489853    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"
Nov 03 02:26:45.615689 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:45.615622    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl" podStartSLOduration=2.615599966 podStartE2EDuration="2.615599966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:26:45.602796763 +0000 UTC m=+4085.028403229" watchObservedRunningTime="2026-11-03 02:26:45.615599966 +0000 UTC m=+4085.041206432"
Nov 03 02:26:46.466645 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:46.466616    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:26:52.148303 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:52.148271    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"]
Nov 03 02:26:55.358654 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:55.358614    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:26:56.499542 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:56.499510    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"
Nov 03 02:26:57.148962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.148905    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr" podUID="de8b9a97-5095-4a9a-a64e-6053fdf408ca" containerName="payload-processing" containerID="cri-o://76a6f6a12d3fe9cb58a48d573949644daf976ae048da995f4345e162f035ba4b" gracePeriod=25
Nov 03 02:26:57.446874 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.446849    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"
Nov 03 02:26:57.505806 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.505703    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de8b9a97-5095-4a9a-a64e-6053fdf408ca-kube-api-access-jrpd2\" (UniqueName: \"kubernetes.io/projected/de8b9a97-5095-4a9a-a64e-6053fdf408ca-kube-api-access-jrpd2\") pod \"de8b9a97-5095-4a9a-a64e-6053fdf408ca\" (UID: \"de8b9a97-5095-4a9a-a64e-6053fdf408ca\") "
Nov 03 02:26:57.506211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.505845    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de8b9a97-5095-4a9a-a64e-6053fdf408ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de8b9a97-5095-4a9a-a64e-6053fdf408ca-plugins-config-volume\") pod \"de8b9a97-5095-4a9a-a64e-6053fdf408ca\" (UID: \"de8b9a97-5095-4a9a-a64e-6053fdf408ca\") "
Nov 03 02:26:57.506265 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.506238    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de8b9a97-5095-4a9a-a64e-6053fdf408ca-plugins-config-volume" pod "de8b9a97-5095-4a9a-a64e-6053fdf408ca" (UID: "de8b9a97-5095-4a9a-a64e-6053fdf408ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:26:57.506485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.506469    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de8b9a97-5095-4a9a-a64e-6053fdf408ca-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:57.507849 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.507818    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de8b9a97-5095-4a9a-a64e-6053fdf408ca-kube-api-access-jrpd2" pod "de8b9a97-5095-4a9a-a64e-6053fdf408ca" (UID: "de8b9a97-5095-4a9a-a64e-6053fdf408ca"). InnerVolumeSpecName "kube-api-access-jrpd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:26:57.607323 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.607294    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrpd2\" (UniqueName: \"kubernetes.io/projected/de8b9a97-5095-4a9a-a64e-6053fdf408ca-kube-api-access-jrpd2\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:26:57.751720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.751638    5508 generic.go:356] "Generic (PLEG): container finished" podID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerID="9e291a49da0ae12a84e63e1ae01e31e16275937e02525062c131efe8a3944e46" exitCode=0
Nov 03 02:26:57.751720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.751707    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx" event={"ID":"df719dd5-eb51-4427-9a9d-626b1c277cb5","Type":"ContainerDied","Data":"9e291a49da0ae12a84e63e1ae01e31e16275937e02525062c131efe8a3944e46"}
Nov 03 02:26:57.752247 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.752229    5508 scope.go:122] "RemoveContainer" containerID="9e291a49da0ae12a84e63e1ae01e31e16275937e02525062c131efe8a3944e46"
Nov 03 02:26:57.753361 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.753326    5508 generic.go:356] "Generic (PLEG): container finished" podID="de8b9a97-5095-4a9a-a64e-6053fdf408ca" containerID="76a6f6a12d3fe9cb58a48d573949644daf976ae048da995f4345e162f035ba4b" exitCode=0
Nov 03 02:26:57.753478 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.753394    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"
Nov 03 02:26:57.753478 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.753456    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr" event={"ID":"de8b9a97-5095-4a9a-a64e-6053fdf408ca","Type":"ContainerDied","Data":"76a6f6a12d3fe9cb58a48d573949644daf976ae048da995f4345e162f035ba4b"}
Nov 03 02:26:57.753576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.753497    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr" event={"ID":"de8b9a97-5095-4a9a-a64e-6053fdf408ca","Type":"ContainerDied","Data":"3220a3603676794d2d45e4c2aa75fb9479ddb21ec11f99f34ae830da3d52a9a3"}
Nov 03 02:26:57.753576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.753523    5508 scope.go:122] "RemoveContainer" containerID="76a6f6a12d3fe9cb58a48d573949644daf976ae048da995f4345e162f035ba4b"
Nov 03 02:26:57.754441 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.754396    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:26:57.754546 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.754488    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:26:57.754592 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.754541    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:26:57.765140 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.765124    5508 scope.go:122] "RemoveContainer" containerID="76a6f6a12d3fe9cb58a48d573949644daf976ae048da995f4345e162f035ba4b"
Nov 03 02:26:57.765443 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:26:57.765423    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a6f6a12d3fe9cb58a48d573949644daf976ae048da995f4345e162f035ba4b\": container with ID starting with 76a6f6a12d3fe9cb58a48d573949644daf976ae048da995f4345e162f035ba4b not found: ID does not exist" containerID="76a6f6a12d3fe9cb58a48d573949644daf976ae048da995f4345e162f035ba4b"
Nov 03 02:26:57.765506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.765450    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a6f6a12d3fe9cb58a48d573949644daf976ae048da995f4345e162f035ba4b"} err="failed to get container status \"76a6f6a12d3fe9cb58a48d573949644daf976ae048da995f4345e162f035ba4b\": rpc error: code = NotFound desc = could not find container \"76a6f6a12d3fe9cb58a48d573949644daf976ae048da995f4345e162f035ba4b\": container with ID starting with 76a6f6a12d3fe9cb58a48d573949644daf976ae048da995f4345e162f035ba4b not found: ID does not exist"
Nov 03 02:26:57.970504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.970470    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"]
Nov 03 02:26:57.980042 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:57.979982    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6fbd7-b8b47c779-dllbr"]
Nov 03 02:26:58.695035 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:58.694998    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de8b9a97-5095-4a9a-a64e-6053fdf408ca" path="/var/lib/kubelet/pods/de8b9a97-5095-4a9a-a64e-6053fdf408ca/volumes"
Nov 03 02:26:58.783689 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:58.783647    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx" event={"ID":"df719dd5-eb51-4427-9a9d-626b1c277cb5","Type":"ContainerStarted","Data":"018a6f8b96e9fc827589e86785c05c293474479b00d4212bf21d87e1afd68161"}
Nov 03 02:26:59.645535 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:59.645490    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:26:59.647124 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:59.647097    5508 patch_prober.go:29] interesting pod/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 03 02:26:59.647306 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:26:59.647150    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx" podUID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 03 02:27:00.646574 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:00.646538    5508 patch_prober.go:29] interesting pod/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 03 02:27:00.647017 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:00.646612    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx" podUID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 03 02:27:01.617062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:01.617026    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:27:01.646011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:01.645980    5508 patch_prober.go:29] interesting pod/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 03 02:27:01.646178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:01.646059    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx" podUID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 03 02:27:02.646730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:02.646689    5508 patch_prober.go:29] interesting pod/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 03 02:27:02.647168 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:02.646744    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx" podUID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 03 02:27:03.645821 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:03.645777    5508 patch_prober.go:29] interesting pod/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 03 02:27:03.646006 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:03.645849    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx" podUID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 03 02:27:04.652583 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:04.652552    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:27:04.653986 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:04.653961    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:27:13.813548 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:27:13.813506    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:27:15.618165 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:15.618122    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"]
Nov 03 02:27:15.717771 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:15.717734    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"]
Nov 03 02:27:20.619472 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:20.619421    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj" podUID="0a945f17-b9a9-4bf8-93a9-497c58ef528f" containerName="payload-processing" containerID="cri-o://4e854503415f4a0c5986b67eca564b281c250e66c746ff86d05ec42a7f6cba8b" gracePeriod=25
Nov 03 02:27:20.718837 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:20.718794    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4" podUID="8a750072-53df-42b4-94ea-0a30718c7207" containerName="payload-pre-processing" containerID="cri-o://505b37cd5d13ce758d0a75f6569337bf76fe52123dd07f15be71bab140946284" gracePeriod=25
Nov 03 02:27:20.919753 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:20.919721    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"
Nov 03 02:27:20.966165 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:20.966125    5508 generic.go:356] "Generic (PLEG): container finished" podID="0a945f17-b9a9-4bf8-93a9-497c58ef528f" containerID="4e854503415f4a0c5986b67eca564b281c250e66c746ff86d05ec42a7f6cba8b" exitCode=0
Nov 03 02:27:20.966360 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:20.966180    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"
Nov 03 02:27:20.966360 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:20.966247    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj" event={"ID":"0a945f17-b9a9-4bf8-93a9-497c58ef528f","Type":"ContainerDied","Data":"4e854503415f4a0c5986b67eca564b281c250e66c746ff86d05ec42a7f6cba8b"}
Nov 03 02:27:20.966360 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:20.966288    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj" event={"ID":"0a945f17-b9a9-4bf8-93a9-497c58ef528f","Type":"ContainerDied","Data":"6b262e8db75fda935819943c27e1e06cc3a082ae2b01e80f90d852ac63b6238a"}
Nov 03 02:27:20.966360 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:20.966320    5508 scope.go:122] "RemoveContainer" containerID="4e854503415f4a0c5986b67eca564b281c250e66c746ff86d05ec42a7f6cba8b"
Nov 03 02:27:20.968618 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:20.968575    5508 generic.go:356] "Generic (PLEG): container finished" podID="8a750072-53df-42b4-94ea-0a30718c7207" containerID="505b37cd5d13ce758d0a75f6569337bf76fe52123dd07f15be71bab140946284" exitCode=0
Nov 03 02:27:20.968798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:20.968643    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4" event={"ID":"8a750072-53df-42b4-94ea-0a30718c7207","Type":"ContainerDied","Data":"505b37cd5d13ce758d0a75f6569337bf76fe52123dd07f15be71bab140946284"}
Nov 03 02:27:20.979049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:20.979027    5508 scope.go:122] "RemoveContainer" containerID="4e854503415f4a0c5986b67eca564b281c250e66c746ff86d05ec42a7f6cba8b"
Nov 03 02:27:20.979459 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:27:20.979433    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e854503415f4a0c5986b67eca564b281c250e66c746ff86d05ec42a7f6cba8b\": container with ID starting with 4e854503415f4a0c5986b67eca564b281c250e66c746ff86d05ec42a7f6cba8b not found: ID does not exist" containerID="4e854503415f4a0c5986b67eca564b281c250e66c746ff86d05ec42a7f6cba8b"
Nov 03 02:27:20.979570 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:20.979465    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e854503415f4a0c5986b67eca564b281c250e66c746ff86d05ec42a7f6cba8b"} err="failed to get container status \"4e854503415f4a0c5986b67eca564b281c250e66c746ff86d05ec42a7f6cba8b\": rpc error: code = NotFound desc = could not find container \"4e854503415f4a0c5986b67eca564b281c250e66c746ff86d05ec42a7f6cba8b\": container with ID starting with 4e854503415f4a0c5986b67eca564b281c250e66c746ff86d05ec42a7f6cba8b not found: ID does not exist"
Nov 03 02:27:21.023877 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.023852    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"
Nov 03 02:27:21.024030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.023919    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a945f17-b9a9-4bf8-93a9-497c58ef528f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a945f17-b9a9-4bf8-93a9-497c58ef528f-plugins-config-volume\") pod \"0a945f17-b9a9-4bf8-93a9-497c58ef528f\" (UID: \"0a945f17-b9a9-4bf8-93a9-497c58ef528f\") "
Nov 03 02:27:21.024030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.023994    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a945f17-b9a9-4bf8-93a9-497c58ef528f-kube-api-access-x4s7k\" (UniqueName: \"kubernetes.io/projected/0a945f17-b9a9-4bf8-93a9-497c58ef528f-kube-api-access-x4s7k\") pod \"0a945f17-b9a9-4bf8-93a9-497c58ef528f\" (UID: \"0a945f17-b9a9-4bf8-93a9-497c58ef528f\") "
Nov 03 02:27:21.024267 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.024247    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a945f17-b9a9-4bf8-93a9-497c58ef528f-plugins-config-volume" pod "0a945f17-b9a9-4bf8-93a9-497c58ef528f" (UID: "0a945f17-b9a9-4bf8-93a9-497c58ef528f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:27:21.024464 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.024451    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a945f17-b9a9-4bf8-93a9-497c58ef528f-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:27:21.026059 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.026034    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a945f17-b9a9-4bf8-93a9-497c58ef528f-kube-api-access-x4s7k" pod "0a945f17-b9a9-4bf8-93a9-497c58ef528f" (UID: "0a945f17-b9a9-4bf8-93a9-497c58ef528f"). InnerVolumeSpecName "kube-api-access-x4s7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:27:21.125559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.125521    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a750072-53df-42b4-94ea-0a30718c7207-kube-api-access-zxrmh\" (UniqueName: \"kubernetes.io/projected/8a750072-53df-42b4-94ea-0a30718c7207-kube-api-access-zxrmh\") pod \"8a750072-53df-42b4-94ea-0a30718c7207\" (UID: \"8a750072-53df-42b4-94ea-0a30718c7207\") "
Nov 03 02:27:21.125785 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.125603    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a750072-53df-42b4-94ea-0a30718c7207-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a750072-53df-42b4-94ea-0a30718c7207-plugins-config-volume\") pod \"8a750072-53df-42b4-94ea-0a30718c7207\" (UID: \"8a750072-53df-42b4-94ea-0a30718c7207\") "
Nov 03 02:27:21.125985 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.125959    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4s7k\" (UniqueName: \"kubernetes.io/projected/0a945f17-b9a9-4bf8-93a9-497c58ef528f-kube-api-access-x4s7k\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:27:21.126106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.125984    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a750072-53df-42b4-94ea-0a30718c7207-plugins-config-volume" pod "8a750072-53df-42b4-94ea-0a30718c7207" (UID: "8a750072-53df-42b4-94ea-0a30718c7207"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:27:21.127676 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.127657    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a750072-53df-42b4-94ea-0a30718c7207-kube-api-access-zxrmh" pod "8a750072-53df-42b4-94ea-0a30718c7207" (UID: "8a750072-53df-42b4-94ea-0a30718c7207"). InnerVolumeSpecName "kube-api-access-zxrmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:27:21.227162 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.227126    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxrmh\" (UniqueName: \"kubernetes.io/projected/8a750072-53df-42b4-94ea-0a30718c7207-kube-api-access-zxrmh\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:27:21.227162 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.227158    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a750072-53df-42b4-94ea-0a30718c7207-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:27:21.381483 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.381446    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"]
Nov 03 02:27:21.385085 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.385051    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c56865-64db99d467-cbgmj"]
Nov 03 02:27:21.979161 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.979132    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"
Nov 03 02:27:21.979726 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.979134    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4" event={"ID":"8a750072-53df-42b4-94ea-0a30718c7207","Type":"ContainerDied","Data":"48fb3b56499f26545914a068eacc9258d7b83c2d1279b1e4d7aca1e91477842c"}
Nov 03 02:27:21.979726 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:21.979266    5508 scope.go:122] "RemoveContainer" containerID="505b37cd5d13ce758d0a75f6569337bf76fe52123dd07f15be71bab140946284"
Nov 03 02:27:22.133832 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:22.133791    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"]
Nov 03 02:27:22.180084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:22.180050    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c56865-674bf6f4c-kv6r4"]
Nov 03 02:27:22.694464 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:22.694432    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a945f17-b9a9-4bf8-93a9-497c58ef528f" path="/var/lib/kubelet/pods/0a945f17-b9a9-4bf8-93a9-497c58ef528f/volumes"
Nov 03 02:27:22.694908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:22.694892    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a750072-53df-42b4-94ea-0a30718c7207" path="/var/lib/kubelet/pods/8a750072-53df-42b4-94ea-0a30718c7207/volumes"
Nov 03 02:27:30.000603 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:30.000556    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:27:38.865564 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:38.865519    5508 patch_prober.go:29] interesting pod/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:27:38.866022 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:38.865614    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:27:41.175506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:41.175459    5508 generic.go:356] "Generic (PLEG): container finished" podID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerID="5e4d80ce27f51c1ddb795eadb643e7c9269c9f3a0a5506341edcfbfc0f838d23" exitCode=0
Nov 03 02:27:41.176078 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:41.175541    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" event={"ID":"f46dc7e6-e02b-4201-baad-c3c0764a0216","Type":"ContainerDied","Data":"5e4d80ce27f51c1ddb795eadb643e7c9269c9f3a0a5506341edcfbfc0f838d23"}
Nov 03 02:27:41.176231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:41.176213    5508 scope.go:122] "RemoveContainer" containerID="5e4d80ce27f51c1ddb795eadb643e7c9269c9f3a0a5506341edcfbfc0f838d23"
Nov 03 02:27:41.178489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:41.178454    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:27:41.178586 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:41.178550    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:27:41.178626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:41.178597    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:27:42.188930 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:42.188887    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" event={"ID":"f46dc7e6-e02b-4201-baad-c3c0764a0216","Type":"ContainerStarted","Data":"7e4698ad05328f185f80155390246e55c559ee68758ccbf9e33cf2f4addaf5f8"}
Nov 03 02:27:42.724913 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:42.724880    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:27:42.725103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:42.725087    5508 patch_prober.go:29] interesting pod/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 02:27:42.725165 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:42.725145    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 02:27:43.725562 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:43.725527    5508 patch_prober.go:29] interesting pod/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 02:27:43.725995 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:43.725589    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 02:27:43.812849 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:27:43.812815    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:27:44.725333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:44.725299    5508 patch_prober.go:29] interesting pod/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 02:27:44.725564 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:44.725375    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 02:27:45.725617 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:45.725585    5508 patch_prober.go:29] interesting pod/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 02:27:45.726053 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:45.725638    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 02:27:46.725383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:46.725337    5508 patch_prober.go:29] interesting pod/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Nov 03 02:27:46.725580 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:46.725408    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Nov 03 02:27:47.728949 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:47.728917    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:27:47.729373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:47.729211    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:27:47.729966 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:27:47.729949    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:28:12.955429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:12.955390    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"]
Nov 03 02:28:13.394643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.394556    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"]
Nov 03 02:28:13.394822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.394763    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk" podUID="69b6d24e-83f9-4eab-aefb-2b340c12bc55" containerName="maas-api" containerID="cri-o://b72cb2cdf1831610fb4d56468afe7c47aba716d8063da7d3f397487a9a894dfb" gracePeriod=30
Nov 03 02:28:13.700743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.700718    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:28:13.822381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.821735    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-metrics-certs\") pod \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") "
Nov 03 02:28:13.822381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.821988    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b6d24e-83f9-4eab-aefb-2b340c12bc55-kube-api-access-sljk2\" (UniqueName: \"kubernetes.io/projected/69b6d24e-83f9-4eab-aefb-2b340c12bc55-kube-api-access-sljk2\") pod \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") "
Nov 03 02:28:13.822381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.822085    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-maas-api-tls\") pod \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") "
Nov 03 02:28:13.822381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.822145    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69b6d24e-83f9-4eab-aefb-2b340c12bc55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b6d24e-83f9-4eab-aefb-2b340c12bc55-openshift-service-ca\") pod \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\" (UID: \"69b6d24e-83f9-4eab-aefb-2b340c12bc55\") "
Nov 03 02:28:13.834596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.834554    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b6d24e-83f9-4eab-aefb-2b340c12bc55-openshift-service-ca" pod "69b6d24e-83f9-4eab-aefb-2b340c12bc55" (UID: "69b6d24e-83f9-4eab-aefb-2b340c12bc55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:28:13.860672 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.860511    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b6d24e-83f9-4eab-aefb-2b340c12bc55-kube-api-access-sljk2" pod "69b6d24e-83f9-4eab-aefb-2b340c12bc55" (UID: "69b6d24e-83f9-4eab-aefb-2b340c12bc55"). InnerVolumeSpecName "kube-api-access-sljk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:28:13.869528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.869470    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-maas-api-tls" pod "69b6d24e-83f9-4eab-aefb-2b340c12bc55" (UID: "69b6d24e-83f9-4eab-aefb-2b340c12bc55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:28:13.882772 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.882735    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-metrics-certs" pod "69b6d24e-83f9-4eab-aefb-2b340c12bc55" (UID: "69b6d24e-83f9-4eab-aefb-2b340c12bc55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:28:13.897832 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:28:13.897793    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:28:13.924187 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.924112    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljk2\" (UniqueName: \"kubernetes.io/projected/69b6d24e-83f9-4eab-aefb-2b340c12bc55-kube-api-access-sljk2\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:13.924187 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.924151    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:13.924187 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.924169    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69b6d24e-83f9-4eab-aefb-2b340c12bc55-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:13.924187 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:13.924183    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69b6d24e-83f9-4eab-aefb-2b340c12bc55-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:14.483394 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.483331    5508 generic.go:356] "Generic (PLEG): container finished" podID="69b6d24e-83f9-4eab-aefb-2b340c12bc55" containerID="b72cb2cdf1831610fb4d56468afe7c47aba716d8063da7d3f397487a9a894dfb" exitCode=0
Nov 03 02:28:14.483858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.483409    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"
Nov 03 02:28:14.483858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.483419    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk" event={"ID":"69b6d24e-83f9-4eab-aefb-2b340c12bc55","Type":"ContainerDied","Data":"b72cb2cdf1831610fb4d56468afe7c47aba716d8063da7d3f397487a9a894dfb"}
Nov 03 02:28:14.483858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.483520    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk" event={"ID":"69b6d24e-83f9-4eab-aefb-2b340c12bc55","Type":"ContainerDied","Data":"b8ec48c5decdce51aeb31b0a49bb16f5cb7c2e8524e5b8e6d4442b9775a17bc3"}
Nov 03 02:28:14.483858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.483565    5508 scope.go:122] "RemoveContainer" containerID="b72cb2cdf1831610fb4d56468afe7c47aba716d8063da7d3f397487a9a894dfb"
Nov 03 02:28:14.495248 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.495225    5508 scope.go:122] "RemoveContainer" containerID="b72cb2cdf1831610fb4d56468afe7c47aba716d8063da7d3f397487a9a894dfb"
Nov 03 02:28:14.495600 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:28:14.495579    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b72cb2cdf1831610fb4d56468afe7c47aba716d8063da7d3f397487a9a894dfb\": container with ID starting with b72cb2cdf1831610fb4d56468afe7c47aba716d8063da7d3f397487a9a894dfb not found: ID does not exist" containerID="b72cb2cdf1831610fb4d56468afe7c47aba716d8063da7d3f397487a9a894dfb"
Nov 03 02:28:14.495654 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.495611    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b72cb2cdf1831610fb4d56468afe7c47aba716d8063da7d3f397487a9a894dfb"} err="failed to get container status \"b72cb2cdf1831610fb4d56468afe7c47aba716d8063da7d3f397487a9a894dfb\": rpc error: code = NotFound desc = could not find container \"b72cb2cdf1831610fb4d56468afe7c47aba716d8063da7d3f397487a9a894dfb\": container with ID starting with b72cb2cdf1831610fb4d56468afe7c47aba716d8063da7d3f397487a9a894dfb not found: ID does not exist"
Nov 03 02:28:14.638083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.638045    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"]
Nov 03 02:28:14.641846 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.641817    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0b98c1-7dfc6d6585-6grlk"]
Nov 03 02:28:14.694922 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.694846    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b6d24e-83f9-4eab-aefb-2b340c12bc55" path="/var/lib/kubelet/pods/69b6d24e-83f9-4eab-aefb-2b340c12bc55/volumes"
Nov 03 02:28:14.875620 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.875581    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"]
Nov 03 02:28:14.876436 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.876405    5508 cpu_manager.go:414] "containerMap: removing container" podUID="69b6d24e-83f9-4eab-aefb-2b340c12bc55" containerName="maas-api"
Nov 03 02:28:14.876436 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.876431    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b6d24e-83f9-4eab-aefb-2b340c12bc55" containerName="maas-api"
Nov 03 02:28:14.876626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.876452    5508 cpu_manager.go:414] "containerMap: removing container" podUID="de8b9a97-5095-4a9a-a64e-6053fdf408ca" containerName="payload-processing"
Nov 03 02:28:14.876626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.876460    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de8b9a97-5095-4a9a-a64e-6053fdf408ca" containerName="payload-processing"
Nov 03 02:28:14.876626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.876473    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8a750072-53df-42b4-94ea-0a30718c7207" containerName="payload-pre-processing"
Nov 03 02:28:14.876626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.876481    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a750072-53df-42b4-94ea-0a30718c7207" containerName="payload-pre-processing"
Nov 03 02:28:14.876626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.876512    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0a945f17-b9a9-4bf8-93a9-497c58ef528f" containerName="payload-processing"
Nov 03 02:28:14.876626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.876520    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a945f17-b9a9-4bf8-93a9-497c58ef528f" containerName="payload-processing"
Nov 03 02:28:14.876827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.876661    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a750072-53df-42b4-94ea-0a30718c7207" containerName="payload-pre-processing"
Nov 03 02:28:14.876827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.876675    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a945f17-b9a9-4bf8-93a9-497c58ef528f" containerName="payload-processing"
Nov 03 02:28:14.876827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.876690    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b6d24e-83f9-4eab-aefb-2b340c12bc55" containerName="maas-api"
Nov 03 02:28:14.876827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.876697    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="de8b9a97-5095-4a9a-a64e-6053fdf408ca" containerName="payload-processing"
Nov 03 02:28:14.881816 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.881797    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:14.930378 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.930333    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e40047-serving-cert\""
Nov 03 02:28:14.938732 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.938703    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3164d26-e210-4ca0-a32b-f169362543fa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e40047-8c5674889-mplcv\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:14.938881 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.938765    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98jb\" (UniqueName: \"kubernetes.io/projected/f3164d26-e210-4ca0-a32b-f169362543fa-kube-api-access-t98jb\") pod \"maas-api-e2e-inf-b-e40047-8c5674889-mplcv\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:14.939007 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.938979    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-metrics-certs\") pod \"maas-api-e2e-inf-b-e40047-8c5674889-mplcv\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:14.939086 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.939025    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-maas-api-tls\") pod \"maas-api-e2e-inf-b-e40047-8c5674889-mplcv\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:14.940504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:14.940477    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"]
Nov 03 02:28:15.040493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:15.040010    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98jb\" (UniqueName: \"kubernetes.io/projected/f3164d26-e210-4ca0-a32b-f169362543fa-kube-api-access-t98jb\") pod \"maas-api-e2e-inf-b-e40047-8c5674889-mplcv\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:15.040493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:15.040236    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-metrics-certs\") pod \"maas-api-e2e-inf-b-e40047-8c5674889-mplcv\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:15.040493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:15.040294    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-maas-api-tls\") pod \"maas-api-e2e-inf-b-e40047-8c5674889-mplcv\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:15.042549 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:15.040898    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3164d26-e210-4ca0-a32b-f169362543fa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e40047-8c5674889-mplcv\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:15.042549 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:15.041796    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3164d26-e210-4ca0-a32b-f169362543fa-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e40047-8c5674889-mplcv\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:15.043338 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:28:15.043319    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-serving-cert: secret "maas-api-e2e-inf-b-e40047-serving-cert" not found
Nov 03 02:28:15.043589 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:28:15.043566    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-maas-api-tls podName:f3164d26-e210-4ca0-a32b-f169362543fa nodeName:}" failed. No retries permitted until 2026-11-03 02:28:15.543543102 +0000 UTC m=+4174.969149568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-maas-api-tls") pod "maas-api-e2e-inf-b-e40047-8c5674889-mplcv" (UID: "f3164d26-e210-4ca0-a32b-f169362543fa") : secret "maas-api-e2e-inf-b-e40047-serving-cert" not found
Nov 03 02:28:15.046675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:15.046652    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-metrics-certs\") pod \"maas-api-e2e-inf-b-e40047-8c5674889-mplcv\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:15.101949 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:15.101917    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98jb\" (UniqueName: \"kubernetes.io/projected/f3164d26-e210-4ca0-a32b-f169362543fa-kube-api-access-t98jb\") pod \"maas-api-e2e-inf-b-e40047-8c5674889-mplcv\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:15.548052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:15.548013    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-maas-api-tls\") pod \"maas-api-e2e-inf-b-e40047-8c5674889-mplcv\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:15.550624 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:15.550596    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-maas-api-tls\") pod \"maas-api-e2e-inf-b-e40047-8c5674889-mplcv\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:15.796169 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:15.796133    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:16.180732 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:16.180698    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"]
Nov 03 02:28:16.187508 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:28:16.187479    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3164d26_e210_4ca0_a32b_f169362543fa.slice/crio-2edfae43364b091ebe548d393814d00103dbafcf117104e7d694a55af97b12ab WatchSource:0}: Error finding container 2edfae43364b091ebe548d393814d00103dbafcf117104e7d694a55af97b12ab: Status 404 returned error can't find the container with id 2edfae43364b091ebe548d393814d00103dbafcf117104e7d694a55af97b12ab
Nov 03 02:28:16.501951 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:16.501855    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv" event={"ID":"f3164d26-e210-4ca0-a32b-f169362543fa","Type":"ContainerStarted","Data":"2edfae43364b091ebe548d393814d00103dbafcf117104e7d694a55af97b12ab"}
Nov 03 02:28:17.743055 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.742972    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"]
Nov 03 02:28:17.747776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.747751    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"
Nov 03 02:28:17.797461 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.797415    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:28:17.859368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.851306    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:28:17.861661 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.860643    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"]
Nov 03 02:28:17.884563 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.884527    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr\" (UID: \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"
Nov 03 02:28:17.884716 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.884658    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr\" (UID: \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"
Nov 03 02:28:17.884778 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.884762    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5n89\" (UniqueName: \"kubernetes.io/projected/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-kube-api-access-t5n89\") pod \"maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr\" (UID: \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"
Nov 03 02:28:17.956398 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.956291    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2" podUID="22189dc0-97c0-4d3a-95bd-290d6e29d219" containerName="payload-processing" containerID="cri-o://f8760d4872746fa4909a4a43dcbe8c09756c30eb9397bef1288d5d77dc243399" gracePeriod=25
Nov 03 02:28:17.985963 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.985926    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr\" (UID: \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"
Nov 03 02:28:17.986135 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.986068    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5n89\" (UniqueName: \"kubernetes.io/projected/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-kube-api-access-t5n89\") pod \"maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr\" (UID: \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"
Nov 03 02:28:17.986263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.986244    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr\" (UID: \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"
Nov 03 02:28:17.986664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.986641    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr\" (UID: \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"
Nov 03 02:28:17.988938 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:17.988916    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr\" (UID: \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"
Nov 03 02:28:18.043522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.043450    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5n89\" (UniqueName: \"kubernetes.io/projected/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-kube-api-access-t5n89\") pod \"maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr\" (UID: \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"
Nov 03 02:28:18.073430 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.073399    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"
Nov 03 02:28:18.294730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.294667    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"
Nov 03 02:28:18.391276 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.391245    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22189dc0-97c0-4d3a-95bd-290d6e29d219-kube-api-access-8brf9\" (UniqueName: \"kubernetes.io/projected/22189dc0-97c0-4d3a-95bd-290d6e29d219-kube-api-access-8brf9\") pod \"22189dc0-97c0-4d3a-95bd-290d6e29d219\" (UID: \"22189dc0-97c0-4d3a-95bd-290d6e29d219\") "
Nov 03 02:28:18.391484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.391415    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22189dc0-97c0-4d3a-95bd-290d6e29d219-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22189dc0-97c0-4d3a-95bd-290d6e29d219-plugins-config-volume\") pod \"22189dc0-97c0-4d3a-95bd-290d6e29d219\" (UID: \"22189dc0-97c0-4d3a-95bd-290d6e29d219\") "
Nov 03 02:28:18.391843 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.391817    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22189dc0-97c0-4d3a-95bd-290d6e29d219-plugins-config-volume" pod "22189dc0-97c0-4d3a-95bd-290d6e29d219" (UID: "22189dc0-97c0-4d3a-95bd-290d6e29d219"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:28:18.393381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.393334    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22189dc0-97c0-4d3a-95bd-290d6e29d219-kube-api-access-8brf9" pod "22189dc0-97c0-4d3a-95bd-290d6e29d219" (UID: "22189dc0-97c0-4d3a-95bd-290d6e29d219"). InnerVolumeSpecName "kube-api-access-8brf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:28:18.466258 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.466223    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"]
Nov 03 02:28:18.471063 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:28:18.471028    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fdfb5a6_912d_4f32_ad57_aad20a3b9527.slice/crio-ebd0d70f83ed2ba101f9277c8857fb30faf46daf5fdc4636976bd8a95ea8bf1e WatchSource:0}: Error finding container ebd0d70f83ed2ba101f9277c8857fb30faf46daf5fdc4636976bd8a95ea8bf1e: Status 404 returned error can't find the container with id ebd0d70f83ed2ba101f9277c8857fb30faf46daf5fdc4636976bd8a95ea8bf1e
Nov 03 02:28:18.492861 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.492833    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22189dc0-97c0-4d3a-95bd-290d6e29d219-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:18.492861 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.492861    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8brf9\" (UniqueName: \"kubernetes.io/projected/22189dc0-97c0-4d3a-95bd-290d6e29d219-kube-api-access-8brf9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:18.518437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.518398    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr" event={"ID":"6fdfb5a6-912d-4f32-ad57-aad20a3b9527","Type":"ContainerStarted","Data":"ebd0d70f83ed2ba101f9277c8857fb30faf46daf5fdc4636976bd8a95ea8bf1e"}
Nov 03 02:28:18.520081 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.520050    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv" event={"ID":"f3164d26-e210-4ca0-a32b-f169362543fa","Type":"ContainerStarted","Data":"cf40420fd0121c24f0f1883942fe8a591042aaccd290c7c60f04ff3bed5ca182"}
Nov 03 02:28:18.520244 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.520134    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:18.521435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.521413    5508 generic.go:356] "Generic (PLEG): container finished" podID="22189dc0-97c0-4d3a-95bd-290d6e29d219" containerID="f8760d4872746fa4909a4a43dcbe8c09756c30eb9397bef1288d5d77dc243399" exitCode=0
Nov 03 02:28:18.521547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.521451    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"
Nov 03 02:28:18.521547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.521459    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2" event={"ID":"22189dc0-97c0-4d3a-95bd-290d6e29d219","Type":"ContainerDied","Data":"f8760d4872746fa4909a4a43dcbe8c09756c30eb9397bef1288d5d77dc243399"}
Nov 03 02:28:18.521547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.521506    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2" event={"ID":"22189dc0-97c0-4d3a-95bd-290d6e29d219","Type":"ContainerDied","Data":"ff71a9245747bc9e2f3a1892a24cc24001ba980304c03d4b406df888cf57c46e"}
Nov 03 02:28:18.521547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.521537    5508 scope.go:122] "RemoveContainer" containerID="f8760d4872746fa4909a4a43dcbe8c09756c30eb9397bef1288d5d77dc243399"
Nov 03 02:28:18.534214 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.534185    5508 scope.go:122] "RemoveContainer" containerID="f8760d4872746fa4909a4a43dcbe8c09756c30eb9397bef1288d5d77dc243399"
Nov 03 02:28:18.534586 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:28:18.534564    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8760d4872746fa4909a4a43dcbe8c09756c30eb9397bef1288d5d77dc243399\": container with ID starting with f8760d4872746fa4909a4a43dcbe8c09756c30eb9397bef1288d5d77dc243399 not found: ID does not exist" containerID="f8760d4872746fa4909a4a43dcbe8c09756c30eb9397bef1288d5d77dc243399"
Nov 03 02:28:18.534656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.534595    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8760d4872746fa4909a4a43dcbe8c09756c30eb9397bef1288d5d77dc243399"} err="failed to get container status \"f8760d4872746fa4909a4a43dcbe8c09756c30eb9397bef1288d5d77dc243399\": rpc error: code = NotFound desc = could not find container \"f8760d4872746fa4909a4a43dcbe8c09756c30eb9397bef1288d5d77dc243399\": container with ID starting with f8760d4872746fa4909a4a43dcbe8c09756c30eb9397bef1288d5d77dc243399 not found: ID does not exist"
Nov 03 02:28:18.629752 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.629635    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv" podStartSLOduration=3.352769776 podStartE2EDuration="4.62961702s" totalImagesPullingTime="1.276847244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:28:18.629387798 +0000 UTC m=+4178.054994288" watchObservedRunningTime="2026-11-03 02:28:18.62961702 +0000 UTC m=+4178.055223483"
Nov 03 02:28:18.783374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.780402    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"]
Nov 03 02:28:18.805371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:18.802062    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0b98c1-f85999947-slmx2"]
Nov 03 02:28:19.531912 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:19.531881    5508 generic.go:356] "Generic (PLEG): container finished" podID="6fdfb5a6-912d-4f32-ad57-aad20a3b9527" containerID="e5cbe36f26d5a6b47dc120a8c33c3816c277b14734f2d25f5f76b72afe423e8c" exitCode=0
Nov 03 02:28:19.532117 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:19.531960    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr" event={"ID":"6fdfb5a6-912d-4f32-ad57-aad20a3b9527","Type":"ContainerDied","Data":"e5cbe36f26d5a6b47dc120a8c33c3816c277b14734f2d25f5f76b72afe423e8c"}
Nov 03 02:28:20.703158 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:20.703130    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"
Nov 03 02:28:20.703847 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:20.703826    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22189dc0-97c0-4d3a-95bd-290d6e29d219" path="/var/lib/kubelet/pods/22189dc0-97c0-4d3a-95bd-290d6e29d219/volumes"
Nov 03 02:28:20.828400 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:20.828312    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-openshift-ca\") pod \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\" (UID: \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\") "
Nov 03 02:28:20.828577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:20.828547    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-kube-api-access-t5n89\" (UniqueName: \"kubernetes.io/projected/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-kube-api-access-t5n89\") pod \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\" (UID: \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\") "
Nov 03 02:28:20.828718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:20.828700    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-xks-ca\") pod \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\" (UID: \"6fdfb5a6-912d-4f32-ad57-aad20a3b9527\") "
Nov 03 02:28:20.828798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:20.828708    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-openshift-ca" pod "6fdfb5a6-912d-4f32-ad57-aad20a3b9527" (UID: "6fdfb5a6-912d-4f32-ad57-aad20a3b9527"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:28:20.829328 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:20.829291    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:20.830966 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:20.830940    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-kube-api-access-t5n89" pod "6fdfb5a6-912d-4f32-ad57-aad20a3b9527" (UID: "6fdfb5a6-912d-4f32-ad57-aad20a3b9527"). InnerVolumeSpecName "kube-api-access-t5n89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:28:20.831220 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:20.831192    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-xks-ca" pod "6fdfb5a6-912d-4f32-ad57-aad20a3b9527" (UID: "6fdfb5a6-912d-4f32-ad57-aad20a3b9527"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:28:20.930749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:20.930713    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5n89\" (UniqueName: \"kubernetes.io/projected/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-kube-api-access-t5n89\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:20.930749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:20.930749    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdfb5a6-912d-4f32-ad57-aad20a3b9527-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:21.546692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:21.546662    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"
Nov 03 02:28:21.546880 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:21.546693    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr" event={"ID":"6fdfb5a6-912d-4f32-ad57-aad20a3b9527","Type":"ContainerDied","Data":"ebd0d70f83ed2ba101f9277c8857fb30faf46daf5fdc4636976bd8a95ea8bf1e"}
Nov 03 02:28:21.546880 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:21.546739    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd0d70f83ed2ba101f9277c8857fb30faf46daf5fdc4636976bd8a95ea8bf1e"
Nov 03 02:28:24.537325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:24.537296    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:28:29.839689 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:29.839649    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:28:34.066527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:34.066230    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"]
Nov 03 02:28:34.070201 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:34.070171    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96423399-82sn4"]
Nov 03 02:28:34.693098 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:34.693062    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbc61062-0645-4d53-8c11-eb88238a0d1c" path="/var/lib/kubelet/pods/dbc61062-0645-4d53-8c11-eb88238a0d1c/volumes"
Nov 03 02:28:41.518516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:41.518486    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:28:41.637914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:41.637891    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:28:42.180756 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:42.180712    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"]
Nov 03 02:28:42.180957 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:42.180937    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy" containerID="cri-o://b843cecde40f1f9e5e38fc4ef6321ab4b3941909f5e0a5e45463556f909dc20e" gracePeriod=30
Nov 03 02:28:43.814760 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:28:43.814721    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:28:46.522676 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:46.522587    5508 patch_prober.go:29] interesting pod/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:28:46.522676 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:46.522662    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:28:47.485335 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.485306    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:28:47.540300 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.540266    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-credential-socket\") pod \"eb908f8a-7347-4ef4-b66c-b43b313cf509\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") "
Nov 03 02:28:47.540788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.540335    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-socket\") pod \"eb908f8a-7347-4ef4-b66c-b43b313cf509\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") "
Nov 03 02:28:47.540788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.540393    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-certs\") pod \"eb908f8a-7347-4ef4-b66c-b43b313cf509\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") "
Nov 03 02:28:47.540788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.540470    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-data\") pod \"eb908f8a-7347-4ef4-b66c-b43b313cf509\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") "
Nov 03 02:28:47.540788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.540513    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-envoy\") pod \"eb908f8a-7347-4ef4-b66c-b43b313cf509\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") "
Nov 03 02:28:47.540788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.540649    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-token\" (UniqueName: \"kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-token\") pod \"eb908f8a-7347-4ef4-b66c-b43b313cf509\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") "
Nov 03 02:28:47.540788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.540651    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-credential-socket" pod "eb908f8a-7347-4ef4-b66c-b43b313cf509" (UID: "eb908f8a-7347-4ef4-b66c-b43b313cf509"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:28:47.541176 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.540848    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb908f8a-7347-4ef4-b66c-b43b313cf509-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb908f8a-7347-4ef4-b66c-b43b313cf509-istiod-ca-cert\") pod \"eb908f8a-7347-4ef4-b66c-b43b313cf509\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") "
Nov 03 02:28:47.541176 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.540871    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-certs" pod "eb908f8a-7347-4ef4-b66c-b43b313cf509" (UID: "eb908f8a-7347-4ef4-b66c-b43b313cf509"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:28:47.541176 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.540894    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-data" pod "eb908f8a-7347-4ef4-b66c-b43b313cf509" (UID: "eb908f8a-7347-4ef4-b66c-b43b313cf509"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:28:47.541176 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.540939    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-kube-api-access-kz84c\" (UniqueName: \"kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-kube-api-access-kz84c\") pod \"eb908f8a-7347-4ef4-b66c-b43b313cf509\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") "
Nov 03 02:28:47.541176 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.541062    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-podinfo\") pod \"eb908f8a-7347-4ef4-b66c-b43b313cf509\" (UID: \"eb908f8a-7347-4ef4-b66c-b43b313cf509\") "
Nov 03 02:28:47.541498 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.541256    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb908f8a-7347-4ef4-b66c-b43b313cf509-istiod-ca-cert" pod "eb908f8a-7347-4ef4-b66c-b43b313cf509" (UID: "eb908f8a-7347-4ef4-b66c-b43b313cf509"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:28:47.541740 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.541716    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-socket" pod "eb908f8a-7347-4ef4-b66c-b43b313cf509" (UID: "eb908f8a-7347-4ef4-b66c-b43b313cf509"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:28:47.541918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.541777    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:47.542061 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.541929    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:47.542061 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.541960    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:47.542061 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.541977    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb908f8a-7347-4ef4-b66c-b43b313cf509-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:47.543065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.543041    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-envoy" pod "eb908f8a-7347-4ef4-b66c-b43b313cf509" (UID: "eb908f8a-7347-4ef4-b66c-b43b313cf509"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:28:47.543179 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.543124    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-token" pod "eb908f8a-7347-4ef4-b66c-b43b313cf509" (UID: "eb908f8a-7347-4ef4-b66c-b43b313cf509"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:28:47.543368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.543332    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-podinfo" pod "eb908f8a-7347-4ef4-b66c-b43b313cf509" (UID: "eb908f8a-7347-4ef4-b66c-b43b313cf509"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:28:47.543596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.543578    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-kube-api-access-kz84c" pod "eb908f8a-7347-4ef4-b66c-b43b313cf509" (UID: "eb908f8a-7347-4ef4-b66c-b43b313cf509"). InnerVolumeSpecName "kube-api-access-kz84c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:28:47.643157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.643118    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz84c\" (UniqueName: \"kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-kube-api-access-kz84c\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:47.643157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.643157    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:47.643410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.643175    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:47.643410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.643190    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:47.643410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.643203    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb908f8a-7347-4ef4-b66c-b43b313cf509-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:28:47.929363 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.929304    5508 generic.go:356] "Generic (PLEG): container finished" podID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerID="b843cecde40f1f9e5e38fc4ef6321ab4b3941909f5e0a5e45463556f909dc20e" exitCode=0
Nov 03 02:28:47.929547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.929377    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"
Nov 03 02:28:47.929547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.929398    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" event={"ID":"eb908f8a-7347-4ef4-b66c-b43b313cf509","Type":"ContainerDied","Data":"b843cecde40f1f9e5e38fc4ef6321ab4b3941909f5e0a5e45463556f909dc20e"}
Nov 03 02:28:47.929547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.929462    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs" event={"ID":"eb908f8a-7347-4ef4-b66c-b43b313cf509","Type":"ContainerDied","Data":"2af3ceddf290ff3c1250da9c948c6ddad3b31f78ad2bc4811be633cb6b59b6be"}
Nov 03 02:28:47.929547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.929489    5508 scope.go:122] "RemoveContainer" containerID="b843cecde40f1f9e5e38fc4ef6321ab4b3941909f5e0a5e45463556f909dc20e"
Nov 03 02:28:47.943452 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.943432    5508 scope.go:122] "RemoveContainer" containerID="ce84032d70badf666dba728e6251c100de1a78080fb7b194f46a347c779c1c45"
Nov 03 02:28:47.954760 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.954741    5508 scope.go:122] "RemoveContainer" containerID="b843cecde40f1f9e5e38fc4ef6321ab4b3941909f5e0a5e45463556f909dc20e"
Nov 03 02:28:47.955092 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:28:47.955071    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b843cecde40f1f9e5e38fc4ef6321ab4b3941909f5e0a5e45463556f909dc20e\": container with ID starting with b843cecde40f1f9e5e38fc4ef6321ab4b3941909f5e0a5e45463556f909dc20e not found: ID does not exist" containerID="b843cecde40f1f9e5e38fc4ef6321ab4b3941909f5e0a5e45463556f909dc20e"
Nov 03 02:28:47.955153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.955103    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b843cecde40f1f9e5e38fc4ef6321ab4b3941909f5e0a5e45463556f909dc20e"} err="failed to get container status \"b843cecde40f1f9e5e38fc4ef6321ab4b3941909f5e0a5e45463556f909dc20e\": rpc error: code = NotFound desc = could not find container \"b843cecde40f1f9e5e38fc4ef6321ab4b3941909f5e0a5e45463556f909dc20e\": container with ID starting with b843cecde40f1f9e5e38fc4ef6321ab4b3941909f5e0a5e45463556f909dc20e not found: ID does not exist"
Nov 03 02:28:47.955153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.955122    5508 scope.go:122] "RemoveContainer" containerID="ce84032d70badf666dba728e6251c100de1a78080fb7b194f46a347c779c1c45"
Nov 03 02:28:47.955443 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:28:47.955420    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce84032d70badf666dba728e6251c100de1a78080fb7b194f46a347c779c1c45\": container with ID starting with ce84032d70badf666dba728e6251c100de1a78080fb7b194f46a347c779c1c45 not found: ID does not exist" containerID="ce84032d70badf666dba728e6251c100de1a78080fb7b194f46a347c779c1c45"
Nov 03 02:28:47.955509 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:47.955453    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce84032d70badf666dba728e6251c100de1a78080fb7b194f46a347c779c1c45"} err="failed to get container status \"ce84032d70badf666dba728e6251c100de1a78080fb7b194f46a347c779c1c45\": rpc error: code = NotFound desc = could not find container \"ce84032d70badf666dba728e6251c100de1a78080fb7b194f46a347c779c1c45\": container with ID starting with ce84032d70badf666dba728e6251c100de1a78080fb7b194f46a347c779c1c45 not found: ID does not exist"
Nov 03 02:28:48.085622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:48.085584    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"]
Nov 03 02:28:48.131578 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:48.131539    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c56865-openshift-default-7c8f666469-rdrbs"]
Nov 03 02:28:48.693272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:28:48.693225    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" path="/var/lib/kubelet/pods/eb908f8a-7347-4ef4-b66c-b43b313cf509/volumes"
Nov 03 02:29:04.972534 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:04.972497    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:29:13.813198 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:29:13.813151    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:29:38.904596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.904555    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"]
Nov 03 02:29:38.905198 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.905181    5508 cpu_manager.go:414] "containerMap: removing container" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy"
Nov 03 02:29:38.905256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.905199    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy"
Nov 03 02:29:38.905256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.905224    5508 cpu_manager.go:414] "containerMap: removing container" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy"
Nov 03 02:29:38.905256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.905230    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy"
Nov 03 02:29:38.905256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.905247    5508 cpu_manager.go:414] "containerMap: removing container" podUID="22189dc0-97c0-4d3a-95bd-290d6e29d219" containerName="payload-processing"
Nov 03 02:29:38.905256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.905253    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22189dc0-97c0-4d3a-95bd-290d6e29d219" containerName="payload-processing"
Nov 03 02:29:38.905502 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.905266    5508 cpu_manager.go:414] "containerMap: removing container" podUID="6fdfb5a6-912d-4f32-ad57-aad20a3b9527" containerName="revoke-keys"
Nov 03 02:29:38.905502 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.905273    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fdfb5a6-912d-4f32-ad57-aad20a3b9527" containerName="revoke-keys"
Nov 03 02:29:38.905585 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.905515    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="22189dc0-97c0-4d3a-95bd-290d6e29d219" containerName="payload-processing"
Nov 03 02:29:38.905585 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.905532    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fdfb5a6-912d-4f32-ad57-aad20a3b9527" containerName="revoke-keys"
Nov 03 02:29:38.905585 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.905546    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy"
Nov 03 02:29:38.905585 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.905556    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb908f8a-7347-4ef4-b66c-b43b313cf509" containerName="istio-proxy"
Nov 03 02:29:38.909483 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.909465    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"
Nov 03 02:29:38.945030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.944998    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a83ac7-7qrqs\" (UID: \"76169ad3-96ee-48f1-824c-9defe8f83540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"
Nov 03 02:29:38.945188 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.945145    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a83ac7-7qrqs\" (UID: \"76169ad3-96ee-48f1-824c-9defe8f83540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"
Nov 03 02:29:38.945271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.945258    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45kxs\" (UniqueName: \"kubernetes.io/projected/76169ad3-96ee-48f1-824c-9defe8f83540-kube-api-access-45kxs\") pod \"maas-api-revoke-sub-13a83ac7-7qrqs\" (UID: \"76169ad3-96ee-48f1-824c-9defe8f83540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"
Nov 03 02:29:38.958263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:38.958232    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:29:39.004493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:39.004447    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:29:39.008241 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:39.008209    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"]
Nov 03 02:29:39.046407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:39.046376    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a83ac7-7qrqs\" (UID: \"76169ad3-96ee-48f1-824c-9defe8f83540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"
Nov 03 02:29:39.046576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:39.046477    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45kxs\" (UniqueName: \"kubernetes.io/projected/76169ad3-96ee-48f1-824c-9defe8f83540-kube-api-access-45kxs\") pod \"maas-api-revoke-sub-13a83ac7-7qrqs\" (UID: \"76169ad3-96ee-48f1-824c-9defe8f83540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"
Nov 03 02:29:39.046576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:39.046569    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a83ac7-7qrqs\" (UID: \"76169ad3-96ee-48f1-824c-9defe8f83540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"
Nov 03 02:29:39.047061 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:39.047022    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a83ac7-7qrqs\" (UID: \"76169ad3-96ee-48f1-824c-9defe8f83540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"
Nov 03 02:29:39.049074 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:39.049041    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a83ac7-7qrqs\" (UID: \"76169ad3-96ee-48f1-824c-9defe8f83540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"
Nov 03 02:29:39.103739 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:39.103706    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45kxs\" (UniqueName: \"kubernetes.io/projected/76169ad3-96ee-48f1-824c-9defe8f83540-kube-api-access-45kxs\") pod \"maas-api-revoke-sub-13a83ac7-7qrqs\" (UID: \"76169ad3-96ee-48f1-824c-9defe8f83540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"
Nov 03 02:29:39.223593 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:39.223507    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"
Nov 03 02:29:39.607176 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:39.607091    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"]
Nov 03 02:29:39.613719 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:29:39.613684    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76169ad3_96ee_48f1_824c_9defe8f83540.slice/crio-cc9bfa493bff5a6c502e647f2d8f537d6ce77d52e511133c925857eb16250bb0 WatchSource:0}: Error finding container cc9bfa493bff5a6c502e647f2d8f537d6ce77d52e511133c925857eb16250bb0: Status 404 returned error can't find the container with id cc9bfa493bff5a6c502e647f2d8f537d6ce77d52e511133c925857eb16250bb0
Nov 03 02:29:40.354859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:40.354768    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs" event={"ID":"76169ad3-96ee-48f1-824c-9defe8f83540","Type":"ContainerStarted","Data":"9748169732ceb4db756f230c0ef44dba47069ee0e0ade337e74cc217e99174ea"}
Nov 03 02:29:40.354859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:40.354833    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs" event={"ID":"76169ad3-96ee-48f1-824c-9defe8f83540","Type":"ContainerStarted","Data":"cc9bfa493bff5a6c502e647f2d8f537d6ce77d52e511133c925857eb16250bb0"}
Nov 03 02:29:40.464603 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:40.464543    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs" podStartSLOduration=2.464528229 podStartE2EDuration="2.464528229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:29:40.462404098 +0000 UTC m=+4259.888010579" watchObservedRunningTime="2026-11-03 02:29:40.464528229 +0000 UTC m=+4259.890134694"
Nov 03 02:29:41.363503 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:41.363466    5508 generic.go:356] "Generic (PLEG): container finished" podID="76169ad3-96ee-48f1-824c-9defe8f83540" containerID="9748169732ceb4db756f230c0ef44dba47069ee0e0ade337e74cc217e99174ea" exitCode=0
Nov 03 02:29:41.363949 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:41.363546    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs" event={"ID":"76169ad3-96ee-48f1-824c-9defe8f83540","Type":"ContainerDied","Data":"9748169732ceb4db756f230c0ef44dba47069ee0e0ade337e74cc217e99174ea"}
Nov 03 02:29:42.528492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:42.528469    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"
Nov 03 02:29:42.592427 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:42.592393    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76169ad3-96ee-48f1-824c-9defe8f83540-kube-api-access-45kxs\" (UniqueName: \"kubernetes.io/projected/76169ad3-96ee-48f1-824c-9defe8f83540-kube-api-access-45kxs\") pod \"76169ad3-96ee-48f1-824c-9defe8f83540\" (UID: \"76169ad3-96ee-48f1-824c-9defe8f83540\") "
Nov 03 02:29:42.592649 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:42.592486    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-xks-ca\") pod \"76169ad3-96ee-48f1-824c-9defe8f83540\" (UID: \"76169ad3-96ee-48f1-824c-9defe8f83540\") "
Nov 03 02:29:42.592649 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:42.592633    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-openshift-ca\") pod \"76169ad3-96ee-48f1-824c-9defe8f83540\" (UID: \"76169ad3-96ee-48f1-824c-9defe8f83540\") "
Nov 03 02:29:42.593029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:42.592997    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-openshift-ca" pod "76169ad3-96ee-48f1-824c-9defe8f83540" (UID: "76169ad3-96ee-48f1-824c-9defe8f83540"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:29:42.593233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:42.593193    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:42.594699 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:42.594674    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-xks-ca" pod "76169ad3-96ee-48f1-824c-9defe8f83540" (UID: "76169ad3-96ee-48f1-824c-9defe8f83540"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:29:42.594699 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:42.594683    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76169ad3-96ee-48f1-824c-9defe8f83540-kube-api-access-45kxs" pod "76169ad3-96ee-48f1-824c-9defe8f83540" (UID: "76169ad3-96ee-48f1-824c-9defe8f83540"). InnerVolumeSpecName "kube-api-access-45kxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:29:42.694785 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:42.694753    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45kxs\" (UniqueName: \"kubernetes.io/projected/76169ad3-96ee-48f1-824c-9defe8f83540-kube-api-access-45kxs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:42.694785 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:42.694779    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76169ad3-96ee-48f1-824c-9defe8f83540-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:43.384362 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:43.384326    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"
Nov 03 02:29:43.384548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:43.384319    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs" event={"ID":"76169ad3-96ee-48f1-824c-9defe8f83540","Type":"ContainerDied","Data":"cc9bfa493bff5a6c502e647f2d8f537d6ce77d52e511133c925857eb16250bb0"}
Nov 03 02:29:43.384548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:43.384461    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc9bfa493bff5a6c502e647f2d8f537d6ce77d52e511133c925857eb16250bb0"
Nov 03 02:29:43.812998 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:29:43.812955    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:29:44.742765 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:44.742689    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"]
Nov 03 02:29:44.984678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:44.984641    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"]
Nov 03 02:29:46.213262 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:46.213226    5508 scope.go:122] "RemoveContainer" containerID="9f14fd652f7b9a2c31508fb1449deb15003613a2beff9c2529ef8cc87b43d20d"
Nov 03 02:29:49.746965 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:49.746917    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g" podUID="e9983db5-4103-41af-98ac-4d3847ade519" containerName="payload-processing" containerID="cri-o://9ceebcb1779117c5adb9f38ccca26a31df4418d83c76c1d641a7bf4bb8f77667" gracePeriod=25
Nov 03 02:29:49.985678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:49.985621    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74" podUID="845bcb6d-d2e0-4806-be2a-5feed5fb7796" containerName="payload-pre-processing" containerID="cri-o://3a039fa6a28bd34aa30357499740be0d6f6a8c6b938d7b29c2db214426212043" gracePeriod=25
Nov 03 02:29:50.118852 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.118823    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"
Nov 03 02:29:50.155435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.155399    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9983db5-4103-41af-98ac-4d3847ade519-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9983db5-4103-41af-98ac-4d3847ade519-plugins-config-volume\") pod \"e9983db5-4103-41af-98ac-4d3847ade519\" (UID: \"e9983db5-4103-41af-98ac-4d3847ade519\") "
Nov 03 02:29:50.155644 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.155542    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9983db5-4103-41af-98ac-4d3847ade519-kube-api-access-fjrqd\" (UniqueName: \"kubernetes.io/projected/e9983db5-4103-41af-98ac-4d3847ade519-kube-api-access-fjrqd\") pod \"e9983db5-4103-41af-98ac-4d3847ade519\" (UID: \"e9983db5-4103-41af-98ac-4d3847ade519\") "
Nov 03 02:29:50.155778 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.155753    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9983db5-4103-41af-98ac-4d3847ade519-plugins-config-volume" pod "e9983db5-4103-41af-98ac-4d3847ade519" (UID: "e9983db5-4103-41af-98ac-4d3847ade519"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:29:50.155889 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.155872    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9983db5-4103-41af-98ac-4d3847ade519-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:50.157598 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.157574    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9983db5-4103-41af-98ac-4d3847ade519-kube-api-access-fjrqd" pod "e9983db5-4103-41af-98ac-4d3847ade519" (UID: "e9983db5-4103-41af-98ac-4d3847ade519"). InnerVolumeSpecName "kube-api-access-fjrqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:29:50.244564 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.244536    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"
Nov 03 02:29:50.256801 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.256728    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845bcb6d-d2e0-4806-be2a-5feed5fb7796-kube-api-access-tf9nl\" (UniqueName: \"kubernetes.io/projected/845bcb6d-d2e0-4806-be2a-5feed5fb7796-kube-api-access-tf9nl\") pod \"845bcb6d-d2e0-4806-be2a-5feed5fb7796\" (UID: \"845bcb6d-d2e0-4806-be2a-5feed5fb7796\") "
Nov 03 02:29:50.256965 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.256856    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845bcb6d-d2e0-4806-be2a-5feed5fb7796-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845bcb6d-d2e0-4806-be2a-5feed5fb7796-plugins-config-volume\") pod \"845bcb6d-d2e0-4806-be2a-5feed5fb7796\" (UID: \"845bcb6d-d2e0-4806-be2a-5feed5fb7796\") "
Nov 03 02:29:50.257358 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.257316    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845bcb6d-d2e0-4806-be2a-5feed5fb7796-plugins-config-volume" pod "845bcb6d-d2e0-4806-be2a-5feed5fb7796" (UID: "845bcb6d-d2e0-4806-be2a-5feed5fb7796"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:29:50.257589 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.257565    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjrqd\" (UniqueName: \"kubernetes.io/projected/e9983db5-4103-41af-98ac-4d3847ade519-kube-api-access-fjrqd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:50.257665 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.257596    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845bcb6d-d2e0-4806-be2a-5feed5fb7796-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:50.259082 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.259058    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845bcb6d-d2e0-4806-be2a-5feed5fb7796-kube-api-access-tf9nl" pod "845bcb6d-d2e0-4806-be2a-5feed5fb7796" (UID: "845bcb6d-d2e0-4806-be2a-5feed5fb7796"). InnerVolumeSpecName "kube-api-access-tf9nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:29:50.358663 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.358625    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf9nl\" (UniqueName: \"kubernetes.io/projected/845bcb6d-d2e0-4806-be2a-5feed5fb7796-kube-api-access-tf9nl\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:50.453643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.453609    5508 generic.go:356] "Generic (PLEG): container finished" podID="e9983db5-4103-41af-98ac-4d3847ade519" containerID="9ceebcb1779117c5adb9f38ccca26a31df4418d83c76c1d641a7bf4bb8f77667" exitCode=0
Nov 03 02:29:50.453853 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.453663    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"
Nov 03 02:29:50.453853 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.453694    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g" event={"ID":"e9983db5-4103-41af-98ac-4d3847ade519","Type":"ContainerDied","Data":"9ceebcb1779117c5adb9f38ccca26a31df4418d83c76c1d641a7bf4bb8f77667"}
Nov 03 02:29:50.453853 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.453755    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g" event={"ID":"e9983db5-4103-41af-98ac-4d3847ade519","Type":"ContainerDied","Data":"f1d03b533a833a4d09e7578e7aa3e96f1071471426018059ca0010bbf2b9ecd3"}
Nov 03 02:29:50.453853 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.453786    5508 scope.go:122] "RemoveContainer" containerID="9ceebcb1779117c5adb9f38ccca26a31df4418d83c76c1d641a7bf4bb8f77667"
Nov 03 02:29:50.455432 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.455407    5508 generic.go:356] "Generic (PLEG): container finished" podID="845bcb6d-d2e0-4806-be2a-5feed5fb7796" containerID="3a039fa6a28bd34aa30357499740be0d6f6a8c6b938d7b29c2db214426212043" exitCode=0
Nov 03 02:29:50.455594 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.455465    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74" event={"ID":"845bcb6d-d2e0-4806-be2a-5feed5fb7796","Type":"ContainerDied","Data":"3a039fa6a28bd34aa30357499740be0d6f6a8c6b938d7b29c2db214426212043"}
Nov 03 02:29:50.455594 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.455505    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74" event={"ID":"845bcb6d-d2e0-4806-be2a-5feed5fb7796","Type":"ContainerDied","Data":"395a7cb1281c7f0e217cdd82a6e44736a392da1b0d39affe681871eb81020f9a"}
Nov 03 02:29:50.455594 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.455473    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"
Nov 03 02:29:50.465095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.465074    5508 scope.go:122] "RemoveContainer" containerID="9ceebcb1779117c5adb9f38ccca26a31df4418d83c76c1d641a7bf4bb8f77667"
Nov 03 02:29:50.465462 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:29:50.465429    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ceebcb1779117c5adb9f38ccca26a31df4418d83c76c1d641a7bf4bb8f77667\": container with ID starting with 9ceebcb1779117c5adb9f38ccca26a31df4418d83c76c1d641a7bf4bb8f77667 not found: ID does not exist" containerID="9ceebcb1779117c5adb9f38ccca26a31df4418d83c76c1d641a7bf4bb8f77667"
Nov 03 02:29:50.465565 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.465471    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ceebcb1779117c5adb9f38ccca26a31df4418d83c76c1d641a7bf4bb8f77667"} err="failed to get container status \"9ceebcb1779117c5adb9f38ccca26a31df4418d83c76c1d641a7bf4bb8f77667\": rpc error: code = NotFound desc = could not find container \"9ceebcb1779117c5adb9f38ccca26a31df4418d83c76c1d641a7bf4bb8f77667\": container with ID starting with 9ceebcb1779117c5adb9f38ccca26a31df4418d83c76c1d641a7bf4bb8f77667 not found: ID does not exist"
Nov 03 02:29:50.465565 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.465491    5508 scope.go:122] "RemoveContainer" containerID="3a039fa6a28bd34aa30357499740be0d6f6a8c6b938d7b29c2db214426212043"
Nov 03 02:29:50.475192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.475139    5508 scope.go:122] "RemoveContainer" containerID="3a039fa6a28bd34aa30357499740be0d6f6a8c6b938d7b29c2db214426212043"
Nov 03 02:29:50.475461 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:29:50.475439    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a039fa6a28bd34aa30357499740be0d6f6a8c6b938d7b29c2db214426212043\": container with ID starting with 3a039fa6a28bd34aa30357499740be0d6f6a8c6b938d7b29c2db214426212043 not found: ID does not exist" containerID="3a039fa6a28bd34aa30357499740be0d6f6a8c6b938d7b29c2db214426212043"
Nov 03 02:29:50.475573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.475466    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a039fa6a28bd34aa30357499740be0d6f6a8c6b938d7b29c2db214426212043"} err="failed to get container status \"3a039fa6a28bd34aa30357499740be0d6f6a8c6b938d7b29c2db214426212043\": rpc error: code = NotFound desc = could not find container \"3a039fa6a28bd34aa30357499740be0d6f6a8c6b938d7b29c2db214426212043\": container with ID starting with 3a039fa6a28bd34aa30357499740be0d6f6a8c6b938d7b29c2db214426212043 not found: ID does not exist"
Nov 03 02:29:50.618448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.618234    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"]
Nov 03 02:29:50.624096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.624050    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e051f3-6df44b4567-m2c74"]
Nov 03 02:29:50.693727 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.693694    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845bcb6d-d2e0-4806-be2a-5feed5fb7796" path="/var/lib/kubelet/pods/845bcb6d-d2e0-4806-be2a-5feed5fb7796/volumes"
Nov 03 02:29:50.719961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.719928    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"]
Nov 03 02:29:50.727520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.727462    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e051f3-588c5f57-gth4g"]
Nov 03 02:29:50.783651 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.783625    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"]
Nov 03 02:29:50.790066 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:50.790042    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d6fbd7-s4zhr"]
Nov 03 02:29:51.016164 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:51.016128    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"]
Nov 03 02:29:51.016418 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:51.016363    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx" podUID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerName="istio-proxy" containerID="cri-o://018a6f8b96e9fc827589e86785c05c293474479b00d4212bf21d87e1afd68161" gracePeriod=30
Nov 03 02:29:52.693455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:52.693421    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fdfb5a6-912d-4f32-ad57-aad20a3b9527" path="/var/lib/kubelet/pods/6fdfb5a6-912d-4f32-ad57-aad20a3b9527/volumes"
Nov 03 02:29:52.693839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:52.693801    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9983db5-4103-41af-98ac-4d3847ade519" path="/var/lib/kubelet/pods/e9983db5-4103-41af-98ac-4d3847ade519/volumes"
Nov 03 02:29:56.296809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.296746    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:29:56.336976 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.336940    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-certs\") pod \"df719dd5-eb51-4427-9a9d-626b1c277cb5\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") "
Nov 03 02:29:56.337149 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.336988    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-socket\") pod \"df719dd5-eb51-4427-9a9d-626b1c277cb5\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") "
Nov 03 02:29:56.337149 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.337068    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-data\") pod \"df719dd5-eb51-4427-9a9d-626b1c277cb5\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") "
Nov 03 02:29:56.337262 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.337199    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-certs" pod "df719dd5-eb51-4427-9a9d-626b1c277cb5" (UID: "df719dd5-eb51-4427-9a9d-626b1c277cb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:29:56.337262 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.337195    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df719dd5-eb51-4427-9a9d-626b1c277cb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df719dd5-eb51-4427-9a9d-626b1c277cb5-istiod-ca-cert\") pod \"df719dd5-eb51-4427-9a9d-626b1c277cb5\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") "
Nov 03 02:29:56.337358 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.337271    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-credential-socket\") pod \"df719dd5-eb51-4427-9a9d-626b1c277cb5\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") "
Nov 03 02:29:56.337429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.337383    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-envoy\") pod \"df719dd5-eb51-4427-9a9d-626b1c277cb5\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") "
Nov 03 02:29:56.337429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.337409    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-socket" pod "df719dd5-eb51-4427-9a9d-626b1c277cb5" (UID: "df719dd5-eb51-4427-9a9d-626b1c277cb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:29:56.337551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.337436    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-podinfo\") pod \"df719dd5-eb51-4427-9a9d-626b1c277cb5\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") "
Nov 03 02:29:56.337551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.337444    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-data" pod "df719dd5-eb51-4427-9a9d-626b1c277cb5" (UID: "df719dd5-eb51-4427-9a9d-626b1c277cb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:29:56.337551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.337476    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-token\" (UniqueName: \"kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-token\") pod \"df719dd5-eb51-4427-9a9d-626b1c277cb5\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") "
Nov 03 02:29:56.337715 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.337555    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-credential-socket" pod "df719dd5-eb51-4427-9a9d-626b1c277cb5" (UID: "df719dd5-eb51-4427-9a9d-626b1c277cb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:29:56.337715 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.337571    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-kube-api-access-bhvj7\" (UniqueName: \"kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-kube-api-access-bhvj7\") pod \"df719dd5-eb51-4427-9a9d-626b1c277cb5\" (UID: \"df719dd5-eb51-4427-9a9d-626b1c277cb5\") "
Nov 03 02:29:56.337715 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.337562    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df719dd5-eb51-4427-9a9d-626b1c277cb5-istiod-ca-cert" pod "df719dd5-eb51-4427-9a9d-626b1c277cb5" (UID: "df719dd5-eb51-4427-9a9d-626b1c277cb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:29:56.338153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.338109    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:56.338153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.338134    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:56.338153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.338144    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:56.338153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.338152    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df719dd5-eb51-4427-9a9d-626b1c277cb5-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:56.338781 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.338161    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:56.339615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.339589    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-envoy" pod "df719dd5-eb51-4427-9a9d-626b1c277cb5" (UID: "df719dd5-eb51-4427-9a9d-626b1c277cb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:29:56.340069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.340043    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-podinfo" pod "df719dd5-eb51-4427-9a9d-626b1c277cb5" (UID: "df719dd5-eb51-4427-9a9d-626b1c277cb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:29:56.340186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.340102    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-token" pod "df719dd5-eb51-4427-9a9d-626b1c277cb5" (UID: "df719dd5-eb51-4427-9a9d-626b1c277cb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:29:56.340474 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.340458    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-kube-api-access-bhvj7" pod "df719dd5-eb51-4427-9a9d-626b1c277cb5" (UID: "df719dd5-eb51-4427-9a9d-626b1c277cb5"). InnerVolumeSpecName "kube-api-access-bhvj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:29:56.439342 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.439312    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:56.439342 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.439338    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:56.439564 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.439378    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhvj7\" (UniqueName: \"kubernetes.io/projected/df719dd5-eb51-4427-9a9d-626b1c277cb5-kube-api-access-bhvj7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:56.439564 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.439391    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df719dd5-eb51-4427-9a9d-626b1c277cb5-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:29:56.500094 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.500063    5508 generic.go:356] "Generic (PLEG): container finished" podID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerID="018a6f8b96e9fc827589e86785c05c293474479b00d4212bf21d87e1afd68161" exitCode=0
Nov 03 02:29:56.500295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.500108    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"
Nov 03 02:29:56.500295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.500152    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx" event={"ID":"df719dd5-eb51-4427-9a9d-626b1c277cb5","Type":"ContainerDied","Data":"018a6f8b96e9fc827589e86785c05c293474479b00d4212bf21d87e1afd68161"}
Nov 03 02:29:56.500295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.500208    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx" event={"ID":"df719dd5-eb51-4427-9a9d-626b1c277cb5","Type":"ContainerDied","Data":"ba728e2f4709a9154e5f0b7ef37c4738594c2a806d14548fc9fa9892133eb906"}
Nov 03 02:29:56.500295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.500233    5508 scope.go:122] "RemoveContainer" containerID="018a6f8b96e9fc827589e86785c05c293474479b00d4212bf21d87e1afd68161"
Nov 03 02:29:56.511453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.511435    5508 scope.go:122] "RemoveContainer" containerID="9e291a49da0ae12a84e63e1ae01e31e16275937e02525062c131efe8a3944e46"
Nov 03 02:29:56.521938 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.521920    5508 scope.go:122] "RemoveContainer" containerID="018a6f8b96e9fc827589e86785c05c293474479b00d4212bf21d87e1afd68161"
Nov 03 02:29:56.522208 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:29:56.522188    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018a6f8b96e9fc827589e86785c05c293474479b00d4212bf21d87e1afd68161\": container with ID starting with 018a6f8b96e9fc827589e86785c05c293474479b00d4212bf21d87e1afd68161 not found: ID does not exist" containerID="018a6f8b96e9fc827589e86785c05c293474479b00d4212bf21d87e1afd68161"
Nov 03 02:29:56.522261 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.522218    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018a6f8b96e9fc827589e86785c05c293474479b00d4212bf21d87e1afd68161"} err="failed to get container status \"018a6f8b96e9fc827589e86785c05c293474479b00d4212bf21d87e1afd68161\": rpc error: code = NotFound desc = could not find container \"018a6f8b96e9fc827589e86785c05c293474479b00d4212bf21d87e1afd68161\": container with ID starting with 018a6f8b96e9fc827589e86785c05c293474479b00d4212bf21d87e1afd68161 not found: ID does not exist"
Nov 03 02:29:56.522261 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.522236    5508 scope.go:122] "RemoveContainer" containerID="9e291a49da0ae12a84e63e1ae01e31e16275937e02525062c131efe8a3944e46"
Nov 03 02:29:56.522526 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:29:56.522508    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e291a49da0ae12a84e63e1ae01e31e16275937e02525062c131efe8a3944e46\": container with ID starting with 9e291a49da0ae12a84e63e1ae01e31e16275937e02525062c131efe8a3944e46 not found: ID does not exist" containerID="9e291a49da0ae12a84e63e1ae01e31e16275937e02525062c131efe8a3944e46"
Nov 03 02:29:56.522595 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.522534    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e291a49da0ae12a84e63e1ae01e31e16275937e02525062c131efe8a3944e46"} err="failed to get container status \"9e291a49da0ae12a84e63e1ae01e31e16275937e02525062c131efe8a3944e46\": rpc error: code = NotFound desc = could not find container \"9e291a49da0ae12a84e63e1ae01e31e16275937e02525062c131efe8a3944e46\": container with ID starting with 9e291a49da0ae12a84e63e1ae01e31e16275937e02525062c131efe8a3944e46 not found: ID does not exist"
Nov 03 02:29:56.655407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.655372    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"]
Nov 03 02:29:56.658735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.658706    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d6fbd7-openshift-default-5cb6c8b44f-qlzbx"]
Nov 03 02:29:56.694099 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.694070    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df719dd5-eb51-4427-9a9d-626b1c277cb5" path="/var/lib/kubelet/pods/df719dd5-eb51-4427-9a9d-626b1c277cb5/volumes"
Nov 03 02:29:56.895165 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895131    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"]
Nov 03 02:29:56.895737 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895723    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e9983db5-4103-41af-98ac-4d3847ade519" containerName="payload-processing"
Nov 03 02:29:56.895803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895739    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9983db5-4103-41af-98ac-4d3847ade519" containerName="payload-processing"
Nov 03 02:29:56.895803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895769    5508 cpu_manager.go:414] "containerMap: removing container" podUID="76169ad3-96ee-48f1-824c-9defe8f83540" containerName="revoke-keys"
Nov 03 02:29:56.895803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895776    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76169ad3-96ee-48f1-824c-9defe8f83540" containerName="revoke-keys"
Nov 03 02:29:56.895803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895793    5508 cpu_manager.go:414] "containerMap: removing container" podUID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerName="istio-proxy"
Nov 03 02:29:56.895803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895799    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerName="istio-proxy"
Nov 03 02:29:56.896001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895811    5508 cpu_manager.go:414] "containerMap: removing container" podUID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerName="istio-proxy"
Nov 03 02:29:56.896001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895817    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerName="istio-proxy"
Nov 03 02:29:56.896001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895831    5508 cpu_manager.go:414] "containerMap: removing container" podUID="845bcb6d-d2e0-4806-be2a-5feed5fb7796" containerName="payload-pre-processing"
Nov 03 02:29:56.896001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895836    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845bcb6d-d2e0-4806-be2a-5feed5fb7796" containerName="payload-pre-processing"
Nov 03 02:29:56.896001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895916    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9983db5-4103-41af-98ac-4d3847ade519" containerName="payload-processing"
Nov 03 02:29:56.896001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895929    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="76169ad3-96ee-48f1-824c-9defe8f83540" containerName="revoke-keys"
Nov 03 02:29:56.896001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895938    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerName="istio-proxy"
Nov 03 02:29:56.896001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895945    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="df719dd5-eb51-4427-9a9d-626b1c277cb5" containerName="istio-proxy"
Nov 03 02:29:56.896001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.895952    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="845bcb6d-d2e0-4806-be2a-5feed5fb7796" containerName="payload-pre-processing"
Nov 03 02:29:56.901385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.901366    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"
Nov 03 02:29:56.945379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.945332    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d956463-h47zf\" (UID: \"501b9d64-d4d3-403f-a321-17935b78eaf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"
Nov 03 02:29:56.945507 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.945473    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d956463-h47zf\" (UID: \"501b9d64-d4d3-403f-a321-17935b78eaf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"
Nov 03 02:29:56.945561 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.945529    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pngcg\" (UniqueName: \"kubernetes.io/projected/501b9d64-d4d3-403f-a321-17935b78eaf2-kube-api-access-pngcg\") pod \"maas-api-revoke-sub-5d956463-h47zf\" (UID: \"501b9d64-d4d3-403f-a321-17935b78eaf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"
Nov 03 02:29:56.949953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.949931    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:29:56.950216 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.950201    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:29:56.956550 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:56.956529    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"]
Nov 03 02:29:57.046431 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:57.046397    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d956463-h47zf\" (UID: \"501b9d64-d4d3-403f-a321-17935b78eaf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"
Nov 03 02:29:57.046621 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:57.046473    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d956463-h47zf\" (UID: \"501b9d64-d4d3-403f-a321-17935b78eaf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"
Nov 03 02:29:57.046690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:57.046632    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pngcg\" (UniqueName: \"kubernetes.io/projected/501b9d64-d4d3-403f-a321-17935b78eaf2-kube-api-access-pngcg\") pod \"maas-api-revoke-sub-5d956463-h47zf\" (UID: \"501b9d64-d4d3-403f-a321-17935b78eaf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"
Nov 03 02:29:57.047103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:57.047086    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d956463-h47zf\" (UID: \"501b9d64-d4d3-403f-a321-17935b78eaf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"
Nov 03 02:29:57.048836 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:57.048812    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d956463-h47zf\" (UID: \"501b9d64-d4d3-403f-a321-17935b78eaf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"
Nov 03 02:29:57.103559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:57.103530    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pngcg\" (UniqueName: \"kubernetes.io/projected/501b9d64-d4d3-403f-a321-17935b78eaf2-kube-api-access-pngcg\") pod \"maas-api-revoke-sub-5d956463-h47zf\" (UID: \"501b9d64-d4d3-403f-a321-17935b78eaf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"
Nov 03 02:29:57.214501 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:57.214425    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"
Nov 03 02:29:57.593697 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:57.593247    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"]
Nov 03 02:29:57.607118 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:29:57.607086    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501b9d64_d4d3_403f_a321_17935b78eaf2.slice/crio-144a3afcd0e95ace3a13d560036ee29cc0d36e902e65fe96b237c85ec5f388e4 WatchSource:0}: Error finding container 144a3afcd0e95ace3a13d560036ee29cc0d36e902e65fe96b237c85ec5f388e4: Status 404 returned error can't find the container with id 144a3afcd0e95ace3a13d560036ee29cc0d36e902e65fe96b237c85ec5f388e4
Nov 03 02:29:58.516857 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:58.516811    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf" event={"ID":"501b9d64-d4d3-403f-a321-17935b78eaf2","Type":"ContainerStarted","Data":"2c9f518788ae595031d41f377b6926c99d06af3cbc216d72a68eff4d1a7ae9a8"}
Nov 03 02:29:58.517054 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:58.516879    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf" event={"ID":"501b9d64-d4d3-403f-a321-17935b78eaf2","Type":"ContainerStarted","Data":"144a3afcd0e95ace3a13d560036ee29cc0d36e902e65fe96b237c85ec5f388e4"}
Nov 03 02:29:58.627057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:58.627002    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf" podStartSLOduration=2.6269879080000003 podStartE2EDuration="2.626987908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:29:58.624262075 +0000 UTC m=+4278.049868560" watchObservedRunningTime="2026-11-03 02:29:58.626987908 +0000 UTC m=+4278.052594370"
Nov 03 02:29:59.525798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:59.525763    5508 generic.go:356] "Generic (PLEG): container finished" podID="501b9d64-d4d3-403f-a321-17935b78eaf2" containerID="2c9f518788ae595031d41f377b6926c99d06af3cbc216d72a68eff4d1a7ae9a8" exitCode=0
Nov 03 02:29:59.526065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:29:59.525859    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf" event={"ID":"501b9d64-d4d3-403f-a321-17935b78eaf2","Type":"ContainerDied","Data":"2c9f518788ae595031d41f377b6926c99d06af3cbc216d72a68eff4d1a7ae9a8"}
Nov 03 02:30:00.264971 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.264936    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"]
Nov 03 02:30:00.269510 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.269481    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"
Nov 03 02:30:00.271066 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.271039    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"]
Nov 03 02:30:00.275986 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.275962    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"
Nov 03 02:30:00.313643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.313609    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"]
Nov 03 02:30:00.318899 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.318874    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"]
Nov 03 02:30:00.319145 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.319008    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"
Nov 03 02:30:00.324561 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.324533    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"
Nov 03 02:30:00.334442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.334413    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"]
Nov 03 02:30:00.338913 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.338889    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"]
Nov 03 02:30:00.352711 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.352544    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"]
Nov 03 02:30:00.352711 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.352661    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"
Nov 03 02:30:00.359984 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.359777    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"]
Nov 03 02:30:00.366017 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.365875    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"
Nov 03 02:30:00.383415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.378411    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"]
Nov 03 02:30:00.383415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.381826    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"]
Nov 03 02:30:00.395871 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.395839    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m44w9\" (UniqueName: \"kubernetes.io/projected/f1e29727-c92f-4f44-8244-a2af9f83a3ac-kube-api-access-m44w9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4\" (UID: \"f1e29727-c92f-4f44-8244-a2af9f83a3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"
Nov 03 02:30:00.396020 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.395972    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e29727-c92f-4f44-8244-a2af9f83a3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4\" (UID: \"f1e29727-c92f-4f44-8244-a2af9f83a3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"
Nov 03 02:30:00.396020 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.396004    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwqt\" (UniqueName: \"kubernetes.io/projected/25d517d9-1e6d-4106-bf57-4cf3415707b6-kube-api-access-xhwqt\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq\" (UID: \"25d517d9-1e6d-4106-bf57-4cf3415707b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"
Nov 03 02:30:00.396282 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.396263    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbqms\" (UniqueName: \"kubernetes.io/projected/1d218587-4638-4bf3-b319-5cbafcbb7933-kube-api-access-jbqms\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6\" (UID: \"1d218587-4638-4bf3-b319-5cbafcbb7933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"
Nov 03 02:30:00.396329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.396308    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d218587-4638-4bf3-b319-5cbafcbb7933-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6\" (UID: \"1d218587-4638-4bf3-b319-5cbafcbb7933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"
Nov 03 02:30:00.396450 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.396431    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed674d66-2642-4b30-a4b3-9873263a1cfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6\" (UID: \"ed674d66-2642-4b30-a4b3-9873263a1cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"
Nov 03 02:30:00.396498 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.396464    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwbk\" (UniqueName: \"kubernetes.io/projected/ed674d66-2642-4b30-a4b3-9873263a1cfc-kube-api-access-nwwbk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6\" (UID: \"ed674d66-2642-4b30-a4b3-9873263a1cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"
Nov 03 02:30:00.396554 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.396506    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d517d9-1e6d-4106-bf57-4cf3415707b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq\" (UID: \"25d517d9-1e6d-4106-bf57-4cf3415707b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"
Nov 03 02:30:00.403270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.403235    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"]
Nov 03 02:30:00.442408 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.442381    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"]
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.513175    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbqms\" (UniqueName: \"kubernetes.io/projected/1d218587-4638-4bf3-b319-5cbafcbb7933-kube-api-access-jbqms\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6\" (UID: \"1d218587-4638-4bf3-b319-5cbafcbb7933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.513319    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d218587-4638-4bf3-b319-5cbafcbb7933-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6\" (UID: \"1d218587-4638-4bf3-b319-5cbafcbb7933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.513384    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed674d66-2642-4b30-a4b3-9873263a1cfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6\" (UID: \"ed674d66-2642-4b30-a4b3-9873263a1cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.513412    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwbk\" (UniqueName: \"kubernetes.io/projected/ed674d66-2642-4b30-a4b3-9873263a1cfc-kube-api-access-nwwbk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6\" (UID: \"ed674d66-2642-4b30-a4b3-9873263a1cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.513455    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d517d9-1e6d-4106-bf57-4cf3415707b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq\" (UID: \"25d517d9-1e6d-4106-bf57-4cf3415707b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.513517    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m44w9\" (UniqueName: \"kubernetes.io/projected/f1e29727-c92f-4f44-8244-a2af9f83a3ac-kube-api-access-m44w9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4\" (UID: \"f1e29727-c92f-4f44-8244-a2af9f83a3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.513575    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e29727-c92f-4f44-8244-a2af9f83a3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4\" (UID: \"f1e29727-c92f-4f44-8244-a2af9f83a3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.513601    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwqt\" (UniqueName: \"kubernetes.io/projected/25d517d9-1e6d-4106-bf57-4cf3415707b6-kube-api-access-xhwqt\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq\" (UID: \"25d517d9-1e6d-4106-bf57-4cf3415707b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.513692    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt2gs\" (UniqueName: \"kubernetes.io/projected/d260f09c-3f76-4662-b580-14e75503724a-kube-api-access-qt2gs\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w\" (UID: \"d260f09c-3f76-4662-b580-14e75503724a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.513725    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d260f09c-3f76-4662-b580-14e75503724a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w\" (UID: \"d260f09c-3f76-4662-b580-14e75503724a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.513776    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl8vq\" (UniqueName: \"kubernetes.io/projected/b265ea0a-729a-4d1c-9421-9d43f4d44aba-kube-api-access-cl8vq\") pod \"maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw\" (UID: \"b265ea0a-729a-4d1c-9421-9d43f4d44aba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.513853    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b265ea0a-729a-4d1c-9421-9d43f4d44aba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw\" (UID: \"b265ea0a-729a-4d1c-9421-9d43f4d44aba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.515336    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d218587-4638-4bf3-b319-5cbafcbb7933-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6\" (UID: \"1d218587-4638-4bf3-b319-5cbafcbb7933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.515971    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed674d66-2642-4b30-a4b3-9873263a1cfc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6\" (UID: \"ed674d66-2642-4b30-a4b3-9873263a1cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.517174    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d517d9-1e6d-4106-bf57-4cf3415707b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq\" (UID: \"25d517d9-1e6d-4106-bf57-4cf3415707b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"
Nov 03 02:30:00.518420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.518031    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e29727-c92f-4f44-8244-a2af9f83a3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4\" (UID: \"f1e29727-c92f-4f44-8244-a2af9f83a3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"
Nov 03 02:30:00.574446 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.573715    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwbk\" (UniqueName: \"kubernetes.io/projected/ed674d66-2642-4b30-a4b3-9873263a1cfc-kube-api-access-nwwbk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6\" (UID: \"ed674d66-2642-4b30-a4b3-9873263a1cfc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"
Nov 03 02:30:00.574698 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.574648    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbqms\" (UniqueName: \"kubernetes.io/projected/1d218587-4638-4bf3-b319-5cbafcbb7933-kube-api-access-jbqms\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6\" (UID: \"1d218587-4638-4bf3-b319-5cbafcbb7933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"
Nov 03 02:30:00.576047 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.576021    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m44w9\" (UniqueName: \"kubernetes.io/projected/f1e29727-c92f-4f44-8244-a2af9f83a3ac-kube-api-access-m44w9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4\" (UID: \"f1e29727-c92f-4f44-8244-a2af9f83a3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"
Nov 03 02:30:00.576881 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.576832    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwqt\" (UniqueName: \"kubernetes.io/projected/25d517d9-1e6d-4106-bf57-4cf3415707b6-kube-api-access-xhwqt\") pod \"maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq\" (UID: \"25d517d9-1e6d-4106-bf57-4cf3415707b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"
Nov 03 02:30:00.584147 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.584116    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"
Nov 03 02:30:00.594742 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.594718    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"
Nov 03 02:30:00.615698 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.615657    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt2gs\" (UniqueName: \"kubernetes.io/projected/d260f09c-3f76-4662-b580-14e75503724a-kube-api-access-qt2gs\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w\" (UID: \"d260f09c-3f76-4662-b580-14e75503724a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"
Nov 03 02:30:00.615877 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.615852    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d260f09c-3f76-4662-b580-14e75503724a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w\" (UID: \"d260f09c-3f76-4662-b580-14e75503724a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"
Nov 03 02:30:00.616331 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.615949    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl8vq\" (UniqueName: \"kubernetes.io/projected/b265ea0a-729a-4d1c-9421-9d43f4d44aba-kube-api-access-cl8vq\") pod \"maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw\" (UID: \"b265ea0a-729a-4d1c-9421-9d43f4d44aba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"
Nov 03 02:30:00.616331 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.616148    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b265ea0a-729a-4d1c-9421-9d43f4d44aba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw\" (UID: \"b265ea0a-729a-4d1c-9421-9d43f4d44aba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"
Nov 03 02:30:00.616652 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.616628    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d260f09c-3f76-4662-b580-14e75503724a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w\" (UID: \"d260f09c-3f76-4662-b580-14e75503724a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"
Nov 03 02:30:00.616959 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.616938    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b265ea0a-729a-4d1c-9421-9d43f4d44aba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw\" (UID: \"b265ea0a-729a-4d1c-9421-9d43f4d44aba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"
Nov 03 02:30:00.643832 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.643575    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"
Nov 03 02:30:00.667602 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.667569    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"
Nov 03 02:30:00.722133 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.722086    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl8vq\" (UniqueName: \"kubernetes.io/projected/b265ea0a-729a-4d1c-9421-9d43f4d44aba-kube-api-access-cl8vq\") pod \"maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw\" (UID: \"b265ea0a-729a-4d1c-9421-9d43f4d44aba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"
Nov 03 02:30:00.722791 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.722644    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt2gs\" (UniqueName: \"kubernetes.io/projected/d260f09c-3f76-4662-b580-14e75503724a-kube-api-access-qt2gs\") pod \"maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w\" (UID: \"d260f09c-3f76-4662-b580-14e75503724a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"
Nov 03 02:30:00.762577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.762429    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"
Nov 03 02:30:00.968371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.962899    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-xks-ca\") pod \"501b9d64-d4d3-403f-a321-17935b78eaf2\" (UID: \"501b9d64-d4d3-403f-a321-17935b78eaf2\") "
Nov 03 02:30:00.968371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.963053    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-openshift-ca\") pod \"501b9d64-d4d3-403f-a321-17935b78eaf2\" (UID: \"501b9d64-d4d3-403f-a321-17935b78eaf2\") "
Nov 03 02:30:00.968371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.963106    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501b9d64-d4d3-403f-a321-17935b78eaf2-kube-api-access-pngcg\" (UniqueName: \"kubernetes.io/projected/501b9d64-d4d3-403f-a321-17935b78eaf2-kube-api-access-pngcg\") pod \"501b9d64-d4d3-403f-a321-17935b78eaf2\" (UID: \"501b9d64-d4d3-403f-a321-17935b78eaf2\") "
Nov 03 02:30:00.991744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.990769    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-openshift-ca" pod "501b9d64-d4d3-403f-a321-17935b78eaf2" (UID: "501b9d64-d4d3-403f-a321-17935b78eaf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:30:00.991744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.991318    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501b9d64-d4d3-403f-a321-17935b78eaf2-kube-api-access-pngcg" pod "501b9d64-d4d3-403f-a321-17935b78eaf2" (UID: "501b9d64-d4d3-403f-a321-17935b78eaf2"). InnerVolumeSpecName "kube-api-access-pngcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:30:00.991744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:00.991509    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"
Nov 03 02:30:01.005623 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.000995    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"
Nov 03 02:30:01.021578 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.021529    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-xks-ca" pod "501b9d64-d4d3-403f-a321-17935b78eaf2" (UID: "501b9d64-d4d3-403f-a321-17935b78eaf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:30:01.041170 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.041095    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"]
Nov 03 02:30:01.049322 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:30:01.049280    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d218587_4638_4bf3_b319_5cbafcbb7933.slice/crio-f9473252eeb2a80007822068b5de4c96c42b2cded3943670f89a46ca51c8c57b WatchSource:0}: Error finding container f9473252eeb2a80007822068b5de4c96c42b2cded3943670f89a46ca51c8c57b: Status 404 returned error can't find the container with id f9473252eeb2a80007822068b5de4c96c42b2cded3943670f89a46ca51c8c57b
Nov 03 02:30:01.064499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.064430    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:01.064499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.064458    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501b9d64-d4d3-403f-a321-17935b78eaf2-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:01.064499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.064476    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pngcg\" (UniqueName: \"kubernetes.io/projected/501b9d64-d4d3-403f-a321-17935b78eaf2-kube-api-access-pngcg\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:01.135394 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:30:01.135362    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e29727_c92f_4f44_8244_a2af9f83a3ac.slice/crio-afd5e130d80768fc210b380f7693f9c9a9d91f9659112b4b7116b0953e4532eb WatchSource:0}: Error finding container afd5e130d80768fc210b380f7693f9c9a9d91f9659112b4b7116b0953e4532eb: Status 404 returned error can't find the container with id afd5e130d80768fc210b380f7693f9c9a9d91f9659112b4b7116b0953e4532eb
Nov 03 02:30:01.136635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.136602    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"]
Nov 03 02:30:01.160663 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.160628    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"]
Nov 03 02:30:01.166018 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.165990    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"]
Nov 03 02:30:01.251704 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:30:01.251665    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded674d66_2642_4b30_a4b3_9873263a1cfc.slice/crio-0434b519092d486da3879213317a710ad1e851e5ed7128fa377830de3e0a9184 WatchSource:0}: Error finding container 0434b519092d486da3879213317a710ad1e851e5ed7128fa377830de3e0a9184: Status 404 returned error can't find the container with id 0434b519092d486da3879213317a710ad1e851e5ed7128fa377830de3e0a9184
Nov 03 02:30:01.255301 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:30:01.255266    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d517d9_1e6d_4106_bf57_4cf3415707b6.slice/crio-1058969b38e31f95646a1638624c55eac0287610e26cbe9396e7629ad4dae702 WatchSource:0}: Error finding container 1058969b38e31f95646a1638624c55eac0287610e26cbe9396e7629ad4dae702: Status 404 returned error can't find the container with id 1058969b38e31f95646a1638624c55eac0287610e26cbe9396e7629ad4dae702
Nov 03 02:30:01.416778 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.416742    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"]
Nov 03 02:30:01.420673 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.420647    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"]
Nov 03 02:30:01.454852 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:30:01.454468    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd260f09c_3f76_4662_b580_14e75503724a.slice/crio-be508976d5dee88458a4fbba084fb5c02943aa518206865b3084e78d7096164c WatchSource:0}: Error finding container be508976d5dee88458a4fbba084fb5c02943aa518206865b3084e78d7096164c: Status 404 returned error can't find the container with id be508976d5dee88458a4fbba084fb5c02943aa518206865b3084e78d7096164c
Nov 03 02:30:01.455192 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:30:01.455170    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb265ea0a_729a_4d1c_9421_9d43f4d44aba.slice/crio-371c1bc187ea3e1f5e51585511cdbb3a7967d3c7103cec50fa9d8c143bda9d55 WatchSource:0}: Error finding container 371c1bc187ea3e1f5e51585511cdbb3a7967d3c7103cec50fa9d8c143bda9d55: Status 404 returned error can't find the container with id 371c1bc187ea3e1f5e51585511cdbb3a7967d3c7103cec50fa9d8c143bda9d55
Nov 03 02:30:01.566778 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.566728    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq" event={"ID":"25d517d9-1e6d-4106-bf57-4cf3415707b6","Type":"ContainerStarted","Data":"1058969b38e31f95646a1638624c55eac0287610e26cbe9396e7629ad4dae702"}
Nov 03 02:30:01.568326 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.568296    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4" event={"ID":"f1e29727-c92f-4f44-8244-a2af9f83a3ac","Type":"ContainerStarted","Data":"afd5e130d80768fc210b380f7693f9c9a9d91f9659112b4b7116b0953e4532eb"}
Nov 03 02:30:01.569797 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.569766    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw" event={"ID":"b265ea0a-729a-4d1c-9421-9d43f4d44aba","Type":"ContainerStarted","Data":"371c1bc187ea3e1f5e51585511cdbb3a7967d3c7103cec50fa9d8c143bda9d55"}
Nov 03 02:30:01.571812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.571787    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6" event={"ID":"1d218587-4638-4bf3-b319-5cbafcbb7933","Type":"ContainerStarted","Data":"2586ad948cdac7581d0c4faf5c2f0b802644c60d1385171944c008d4c789cea3"}
Nov 03 02:30:01.571952 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.571835    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6" event={"ID":"1d218587-4638-4bf3-b319-5cbafcbb7933","Type":"ContainerStarted","Data":"f9473252eeb2a80007822068b5de4c96c42b2cded3943670f89a46ca51c8c57b"}
Nov 03 02:30:01.585612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.573725    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6" event={"ID":"ed674d66-2642-4b30-a4b3-9873263a1cfc","Type":"ContainerStarted","Data":"0434b519092d486da3879213317a710ad1e851e5ed7128fa377830de3e0a9184"}
Nov 03 02:30:01.585612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.575197    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"
Nov 03 02:30:01.585612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.575219    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf" event={"ID":"501b9d64-d4d3-403f-a321-17935b78eaf2","Type":"ContainerDied","Data":"144a3afcd0e95ace3a13d560036ee29cc0d36e902e65fe96b237c85ec5f388e4"}
Nov 03 02:30:01.585612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.575260    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="144a3afcd0e95ace3a13d560036ee29cc0d36e902e65fe96b237c85ec5f388e4"
Nov 03 02:30:01.585612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.579097    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w" event={"ID":"d260f09c-3f76-4662-b580-14e75503724a","Type":"ContainerStarted","Data":"be508976d5dee88458a4fbba084fb5c02943aa518206865b3084e78d7096164c"}
Nov 03 02:30:01.682913 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:01.682814    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6" podStartSLOduration=1.682800856 podStartE2EDuration="1.682800856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:01.680290079 +0000 UTC m=+4281.105896573" watchObservedRunningTime="2026-11-03 02:30:01.682800856 +0000 UTC m=+4281.108407320"
Nov 03 02:30:02.590516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.590484    5508 generic.go:356] "Generic (PLEG): container finished" podID="25d517d9-1e6d-4106-bf57-4cf3415707b6" containerID="a0ef41375dc38d766fe8e8570bb1fdfe71770e40856577a517788d5b8e469acc" exitCode=60
Nov 03 02:30:02.590910 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.590565    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq" event={"ID":"25d517d9-1e6d-4106-bf57-4cf3415707b6","Type":"ContainerDied","Data":"a0ef41375dc38d766fe8e8570bb1fdfe71770e40856577a517788d5b8e469acc"}
Nov 03 02:30:02.590910 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.590893    5508 scope.go:122] "RemoveContainer" containerID="a0ef41375dc38d766fe8e8570bb1fdfe71770e40856577a517788d5b8e469acc"
Nov 03 02:30:02.593025 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.592497    5508 generic.go:356] "Generic (PLEG): container finished" podID="f1e29727-c92f-4f44-8244-a2af9f83a3ac" containerID="7fa219531c4b53d6dd4d30044a790a27c0ccb775f27a507ebb676c8be6fa0747" exitCode=60
Nov 03 02:30:02.593025 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.592540    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4" event={"ID":"f1e29727-c92f-4f44-8244-a2af9f83a3ac","Type":"ContainerDied","Data":"7fa219531c4b53d6dd4d30044a790a27c0ccb775f27a507ebb676c8be6fa0747"}
Nov 03 02:30:02.593025 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.592762    5508 scope.go:122] "RemoveContainer" containerID="7fa219531c4b53d6dd4d30044a790a27c0ccb775f27a507ebb676c8be6fa0747"
Nov 03 02:30:02.595711 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.595683    5508 generic.go:356] "Generic (PLEG): container finished" podID="b265ea0a-729a-4d1c-9421-9d43f4d44aba" containerID="bcae4be4ad1aeb8d77ef832934d2f0eca7c7a9699079137dc275b4a6525d6466" exitCode=60
Nov 03 02:30:02.595817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.595712    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw" event={"ID":"b265ea0a-729a-4d1c-9421-9d43f4d44aba","Type":"ContainerDied","Data":"bcae4be4ad1aeb8d77ef832934d2f0eca7c7a9699079137dc275b4a6525d6466"}
Nov 03 02:30:02.596078 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.596051    5508 scope.go:122] "RemoveContainer" containerID="bcae4be4ad1aeb8d77ef832934d2f0eca7c7a9699079137dc275b4a6525d6466"
Nov 03 02:30:02.602805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.602746    5508 generic.go:356] "Generic (PLEG): container finished" podID="1d218587-4638-4bf3-b319-5cbafcbb7933" containerID="2586ad948cdac7581d0c4faf5c2f0b802644c60d1385171944c008d4c789cea3" exitCode=60
Nov 03 02:30:02.602805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.602790    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6" event={"ID":"1d218587-4638-4bf3-b319-5cbafcbb7933","Type":"ContainerDied","Data":"2586ad948cdac7581d0c4faf5c2f0b802644c60d1385171944c008d4c789cea3"}
Nov 03 02:30:02.603062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.603047    5508 scope.go:122] "RemoveContainer" containerID="2586ad948cdac7581d0c4faf5c2f0b802644c60d1385171944c008d4c789cea3"
Nov 03 02:30:02.605307 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.605286    5508 generic.go:356] "Generic (PLEG): container finished" podID="ed674d66-2642-4b30-a4b3-9873263a1cfc" containerID="3f1dc1a43ca2177fe9588948ff8154e37e9c852f31d080e44caa1712ad3e5a70" exitCode=60
Nov 03 02:30:02.605434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.605418    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6" event={"ID":"ed674d66-2642-4b30-a4b3-9873263a1cfc","Type":"ContainerDied","Data":"3f1dc1a43ca2177fe9588948ff8154e37e9c852f31d080e44caa1712ad3e5a70"}
Nov 03 02:30:02.605724 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.605707    5508 scope.go:122] "RemoveContainer" containerID="3f1dc1a43ca2177fe9588948ff8154e37e9c852f31d080e44caa1712ad3e5a70"
Nov 03 02:30:02.608337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.607784    5508 generic.go:356] "Generic (PLEG): container finished" podID="d260f09c-3f76-4662-b580-14e75503724a" containerID="f3d6f48ef25f762dca90c937f0c161f1d4f3d971a3a4dc75d8a3795b5424f3ed" exitCode=60
Nov 03 02:30:02.608337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.607837    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w" event={"ID":"d260f09c-3f76-4662-b580-14e75503724a","Type":"ContainerDied","Data":"f3d6f48ef25f762dca90c937f0c161f1d4f3d971a3a4dc75d8a3795b5424f3ed"}
Nov 03 02:30:02.608337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:02.608054    5508 scope.go:122] "RemoveContainer" containerID="f3d6f48ef25f762dca90c937f0c161f1d4f3d971a3a4dc75d8a3795b5424f3ed"
Nov 03 02:30:03.628730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:03.628624    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6" event={"ID":"1d218587-4638-4bf3-b319-5cbafcbb7933","Type":"ContainerStarted","Data":"f1dc911dae279e25791ec6a02fd495232fc384705d62769c98fb933bca0ffa9e"}
Nov 03 02:30:03.630908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:03.630880    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6" event={"ID":"ed674d66-2642-4b30-a4b3-9873263a1cfc","Type":"ContainerStarted","Data":"f5b8ff59ff5cd05b6728ab7181ea9172569e2763127a74a5c8ba7e825fcbcc47"}
Nov 03 02:30:03.633224 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:03.633190    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w" event={"ID":"d260f09c-3f76-4662-b580-14e75503724a","Type":"ContainerStarted","Data":"565c35fb52403e6e122e15a36e7b824a8850083134a847532f80fec787155c3a"}
Nov 03 02:30:03.635174 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:03.635144    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq" event={"ID":"25d517d9-1e6d-4106-bf57-4cf3415707b6","Type":"ContainerStarted","Data":"2959f353e425b922364ace91a4e68beaf4ac626e7d9a53d74639895b8dce5024"}
Nov 03 02:30:03.637067 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:03.637043    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4" event={"ID":"f1e29727-c92f-4f44-8244-a2af9f83a3ac","Type":"ContainerStarted","Data":"3a751e4c47f6cc3ae3219233b2df32284c5aa66868a870b269b2cd5ea552c968"}
Nov 03 02:30:03.638914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:03.638896    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw" event={"ID":"b265ea0a-729a-4d1c-9421-9d43f4d44aba","Type":"ContainerStarted","Data":"cfa271e30ab97084bda294426043efa6cb2fd5ad207f2f0a3b09f28734218615"}
Nov 03 02:30:03.845660 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:03.845601    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6" podStartSLOduration=3.845578963 podStartE2EDuration="3.845578963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:03.844858636 +0000 UTC m=+4283.270465100" watchObservedRunningTime="2026-11-03 02:30:03.845578963 +0000 UTC m=+4283.271185454"
Nov 03 02:30:03.956032 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:03.955978    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw" podStartSLOduration=3.955964124 podStartE2EDuration="3.955964124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:03.952471249 +0000 UTC m=+4283.378077716" watchObservedRunningTime="2026-11-03 02:30:03.955964124 +0000 UTC m=+4283.381570588"
Nov 03 02:30:04.064221 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.064158    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4" podStartSLOduration=4.064138981 podStartE2EDuration="4.064138981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:04.059290429 +0000 UTC m=+4283.484896895" watchObservedRunningTime="2026-11-03 02:30:04.064138981 +0000 UTC m=+4283.489745446"
Nov 03 02:30:04.170298 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.170237    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w" podStartSLOduration=4.17022299 podStartE2EDuration="4.17022299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:04.166629669 +0000 UTC m=+4283.592236135" watchObservedRunningTime="2026-11-03 02:30:04.17022299 +0000 UTC m=+4283.595829454"
Nov 03 02:30:04.275663 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.275604    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq" podStartSLOduration=4.275588383 podStartE2EDuration="4.275588383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:30:04.273695288 +0000 UTC m=+4283.699301775" watchObservedRunningTime="2026-11-03 02:30:04.275588383 +0000 UTC m=+4283.701194848"
Nov 03 02:30:04.651152 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.651108    5508 generic.go:356] "Generic (PLEG): container finished" podID="25d517d9-1e6d-4106-bf57-4cf3415707b6" containerID="2959f353e425b922364ace91a4e68beaf4ac626e7d9a53d74639895b8dce5024" exitCode=60
Nov 03 02:30:04.651646 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.651168    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq" event={"ID":"25d517d9-1e6d-4106-bf57-4cf3415707b6","Type":"ContainerDied","Data":"2959f353e425b922364ace91a4e68beaf4ac626e7d9a53d74639895b8dce5024"}
Nov 03 02:30:04.651646 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.651253    5508 scope.go:122] "RemoveContainer" containerID="a0ef41375dc38d766fe8e8570bb1fdfe71770e40856577a517788d5b8e469acc"
Nov 03 02:30:04.651646 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.651520    5508 scope.go:122] "RemoveContainer" containerID="2959f353e425b922364ace91a4e68beaf4ac626e7d9a53d74639895b8dce5024"
Nov 03 02:30:04.651861 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:04.651840    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq_odh-ai-gateway-infra(25d517d9-1e6d-4106-bf57-4cf3415707b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq" podUID="25d517d9-1e6d-4106-bf57-4cf3415707b6"
Nov 03 02:30:04.654250 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.654225    5508 generic.go:356] "Generic (PLEG): container finished" podID="f1e29727-c92f-4f44-8244-a2af9f83a3ac" containerID="3a751e4c47f6cc3ae3219233b2df32284c5aa66868a870b269b2cd5ea552c968" exitCode=60
Nov 03 02:30:04.654377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.654301    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4" event={"ID":"f1e29727-c92f-4f44-8244-a2af9f83a3ac","Type":"ContainerDied","Data":"3a751e4c47f6cc3ae3219233b2df32284c5aa66868a870b269b2cd5ea552c968"}
Nov 03 02:30:04.654607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.654593    5508 scope.go:122] "RemoveContainer" containerID="3a751e4c47f6cc3ae3219233b2df32284c5aa66868a870b269b2cd5ea552c968"
Nov 03 02:30:04.655040 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:04.654925    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4_odh-ai-gateway-infra(f1e29727-c92f-4f44-8244-a2af9f83a3ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4" podUID="f1e29727-c92f-4f44-8244-a2af9f83a3ac"
Nov 03 02:30:04.656541 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.656517    5508 generic.go:356] "Generic (PLEG): container finished" podID="b265ea0a-729a-4d1c-9421-9d43f4d44aba" containerID="cfa271e30ab97084bda294426043efa6cb2fd5ad207f2f0a3b09f28734218615" exitCode=60
Nov 03 02:30:04.656646 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.656598    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw" event={"ID":"b265ea0a-729a-4d1c-9421-9d43f4d44aba","Type":"ContainerDied","Data":"cfa271e30ab97084bda294426043efa6cb2fd5ad207f2f0a3b09f28734218615"}
Nov 03 02:30:04.656921 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.656901    5508 scope.go:122] "RemoveContainer" containerID="cfa271e30ab97084bda294426043efa6cb2fd5ad207f2f0a3b09f28734218615"
Nov 03 02:30:04.657241 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:04.657211    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw_odh-ai-gateway-infra(b265ea0a-729a-4d1c-9421-9d43f4d44aba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw" podUID="b265ea0a-729a-4d1c-9421-9d43f4d44aba"
Nov 03 02:30:04.658892 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.658866    5508 generic.go:356] "Generic (PLEG): container finished" podID="1d218587-4638-4bf3-b319-5cbafcbb7933" containerID="f1dc911dae279e25791ec6a02fd495232fc384705d62769c98fb933bca0ffa9e" exitCode=60
Nov 03 02:30:04.658999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.658947    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6" event={"ID":"1d218587-4638-4bf3-b319-5cbafcbb7933","Type":"ContainerDied","Data":"f1dc911dae279e25791ec6a02fd495232fc384705d62769c98fb933bca0ffa9e"}
Nov 03 02:30:04.659260 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.659243    5508 scope.go:122] "RemoveContainer" containerID="f1dc911dae279e25791ec6a02fd495232fc384705d62769c98fb933bca0ffa9e"
Nov 03 02:30:04.659600 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:04.659577    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6_odh-ai-gateway-infra(1d218587-4638-4bf3-b319-5cbafcbb7933)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6" podUID="1d218587-4638-4bf3-b319-5cbafcbb7933"
Nov 03 02:30:04.661118 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.661098    5508 generic.go:356] "Generic (PLEG): container finished" podID="ed674d66-2642-4b30-a4b3-9873263a1cfc" containerID="f5b8ff59ff5cd05b6728ab7181ea9172569e2763127a74a5c8ba7e825fcbcc47" exitCode=60
Nov 03 02:30:04.661207 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.661179    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6" event={"ID":"ed674d66-2642-4b30-a4b3-9873263a1cfc","Type":"ContainerDied","Data":"f5b8ff59ff5cd05b6728ab7181ea9172569e2763127a74a5c8ba7e825fcbcc47"}
Nov 03 02:30:04.661456 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.661439    5508 scope.go:122] "RemoveContainer" containerID="f5b8ff59ff5cd05b6728ab7181ea9172569e2763127a74a5c8ba7e825fcbcc47"
Nov 03 02:30:04.661751 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:04.661726    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6_odh-ai-gateway-infra(ed674d66-2642-4b30-a4b3-9873263a1cfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6" podUID="ed674d66-2642-4b30-a4b3-9873263a1cfc"
Nov 03 02:30:04.663381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.663331    5508 generic.go:356] "Generic (PLEG): container finished" podID="d260f09c-3f76-4662-b580-14e75503724a" containerID="565c35fb52403e6e122e15a36e7b824a8850083134a847532f80fec787155c3a" exitCode=60
Nov 03 02:30:04.663496 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.663382    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w" event={"ID":"d260f09c-3f76-4662-b580-14e75503724a","Type":"ContainerDied","Data":"565c35fb52403e6e122e15a36e7b824a8850083134a847532f80fec787155c3a"}
Nov 03 02:30:04.663662 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.663646    5508 scope.go:122] "RemoveContainer" containerID="565c35fb52403e6e122e15a36e7b824a8850083134a847532f80fec787155c3a"
Nov 03 02:30:04.663912 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:04.663892    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w_odh-ai-gateway-infra(d260f09c-3f76-4662-b580-14e75503724a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w" podUID="d260f09c-3f76-4662-b580-14e75503724a"
Nov 03 02:30:04.666343 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.666323    5508 scope.go:122] "RemoveContainer" containerID="7fa219531c4b53d6dd4d30044a790a27c0ccb775f27a507ebb676c8be6fa0747"
Nov 03 02:30:04.681785 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.681760    5508 scope.go:122] "RemoveContainer" containerID="bcae4be4ad1aeb8d77ef832934d2f0eca7c7a9699079137dc275b4a6525d6466"
Nov 03 02:30:04.696673 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.696649    5508 scope.go:122] "RemoveContainer" containerID="2586ad948cdac7581d0c4faf5c2f0b802644c60d1385171944c008d4c789cea3"
Nov 03 02:30:04.709340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.709315    5508 scope.go:122] "RemoveContainer" containerID="3f1dc1a43ca2177fe9588948ff8154e37e9c852f31d080e44caa1712ad3e5a70"
Nov 03 02:30:04.721875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:04.721856    5508 scope.go:122] "RemoveContainer" containerID="f3d6f48ef25f762dca90c937f0c161f1d4f3d971a3a4dc75d8a3795b5424f3ed"
Nov 03 02:30:05.672384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:05.672336    5508 scope.go:122] "RemoveContainer" containerID="3a751e4c47f6cc3ae3219233b2df32284c5aa66868a870b269b2cd5ea552c968"
Nov 03 02:30:05.672865 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:05.672794    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4_odh-ai-gateway-infra(f1e29727-c92f-4f44-8244-a2af9f83a3ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4" podUID="f1e29727-c92f-4f44-8244-a2af9f83a3ac"
Nov 03 02:30:05.674314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:05.674292    5508 scope.go:122] "RemoveContainer" containerID="cfa271e30ab97084bda294426043efa6cb2fd5ad207f2f0a3b09f28734218615"
Nov 03 02:30:05.674665 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:05.674631    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw_odh-ai-gateway-infra(b265ea0a-729a-4d1c-9421-9d43f4d44aba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw" podUID="b265ea0a-729a-4d1c-9421-9d43f4d44aba"
Nov 03 02:30:05.676050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:05.676030    5508 scope.go:122] "RemoveContainer" containerID="f1dc911dae279e25791ec6a02fd495232fc384705d62769c98fb933bca0ffa9e"
Nov 03 02:30:05.676333 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:05.676310    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6_odh-ai-gateway-infra(1d218587-4638-4bf3-b319-5cbafcbb7933)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6" podUID="1d218587-4638-4bf3-b319-5cbafcbb7933"
Nov 03 02:30:05.677719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:05.677702    5508 scope.go:122] "RemoveContainer" containerID="f5b8ff59ff5cd05b6728ab7181ea9172569e2763127a74a5c8ba7e825fcbcc47"
Nov 03 02:30:05.678012 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:05.677977    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6_odh-ai-gateway-infra(ed674d66-2642-4b30-a4b3-9873263a1cfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6" podUID="ed674d66-2642-4b30-a4b3-9873263a1cfc"
Nov 03 02:30:05.679431 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:05.679412    5508 scope.go:122] "RemoveContainer" containerID="565c35fb52403e6e122e15a36e7b824a8850083134a847532f80fec787155c3a"
Nov 03 02:30:05.679668 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:05.679642    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w_odh-ai-gateway-infra(d260f09c-3f76-4662-b580-14e75503724a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w" podUID="d260f09c-3f76-4662-b580-14e75503724a"
Nov 03 02:30:05.681296 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:05.681276    5508 scope.go:122] "RemoveContainer" containerID="2959f353e425b922364ace91a4e68beaf4ac626e7d9a53d74639895b8dce5024"
Nov 03 02:30:05.681584 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:05.681561    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq_odh-ai-gateway-infra(25d517d9-1e6d-4106-bf57-4cf3415707b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq" podUID="25d517d9-1e6d-4106-bf57-4cf3415707b6"
Nov 03 02:30:09.148284 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:09.148234    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:30:13.812399 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:13.812334    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:30:14.685516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:14.685482    5508 scope.go:122] "RemoveContainer" containerID="f5b8ff59ff5cd05b6728ab7181ea9172569e2763127a74a5c8ba7e825fcbcc47"
Nov 03 02:30:14.686689 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:14.686245    5508 scope.go:122] "RemoveContainer" containerID="565c35fb52403e6e122e15a36e7b824a8850083134a847532f80fec787155c3a"
Nov 03 02:30:15.684782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:15.684754    5508 scope.go:122] "RemoveContainer" containerID="3a751e4c47f6cc3ae3219233b2df32284c5aa66868a870b269b2cd5ea552c968"
Nov 03 02:30:15.791929 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:15.791889    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6" event={"ID":"ed674d66-2642-4b30-a4b3-9873263a1cfc","Type":"ContainerStarted","Data":"6f06a6fce67bde74a8a39276fab92c92a23279419805b6762d8ab537cd6970ee"}
Nov 03 02:30:15.794102 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:15.794077    5508 generic.go:356] "Generic (PLEG): container finished" podID="d260f09c-3f76-4662-b580-14e75503724a" containerID="8380adc62e8f7b0edeabe432695a5e09113a7a26f5b9cc8067861253930db17e" exitCode=60
Nov 03 02:30:15.794246 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:15.794147    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w" event={"ID":"d260f09c-3f76-4662-b580-14e75503724a","Type":"ContainerDied","Data":"8380adc62e8f7b0edeabe432695a5e09113a7a26f5b9cc8067861253930db17e"}
Nov 03 02:30:15.794246 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:15.794213    5508 scope.go:122] "RemoveContainer" containerID="565c35fb52403e6e122e15a36e7b824a8850083134a847532f80fec787155c3a"
Nov 03 02:30:15.794647 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:15.794617    5508 scope.go:122] "RemoveContainer" containerID="8380adc62e8f7b0edeabe432695a5e09113a7a26f5b9cc8067861253930db17e"
Nov 03 02:30:15.794974 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:15.794953    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w_odh-ai-gateway-infra(d260f09c-3f76-4662-b580-14e75503724a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w" podUID="d260f09c-3f76-4662-b580-14e75503724a"
Nov 03 02:30:16.812808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:16.812724    5508 generic.go:356] "Generic (PLEG): container finished" podID="f1e29727-c92f-4f44-8244-a2af9f83a3ac" containerID="821890f16f30769fa600245f87f74643a65f9ee821dbd6a203bae87e9ff8b922" exitCode=60
Nov 03 02:30:16.812808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:16.812771    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4" event={"ID":"f1e29727-c92f-4f44-8244-a2af9f83a3ac","Type":"ContainerDied","Data":"821890f16f30769fa600245f87f74643a65f9ee821dbd6a203bae87e9ff8b922"}
Nov 03 02:30:16.813389 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:16.812860    5508 scope.go:122] "RemoveContainer" containerID="3a751e4c47f6cc3ae3219233b2df32284c5aa66868a870b269b2cd5ea552c968"
Nov 03 02:30:16.813389 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:16.813338    5508 scope.go:122] "RemoveContainer" containerID="821890f16f30769fa600245f87f74643a65f9ee821dbd6a203bae87e9ff8b922"
Nov 03 02:30:16.813740 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:16.813717    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4_odh-ai-gateway-infra(f1e29727-c92f-4f44-8244-a2af9f83a3ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4" podUID="f1e29727-c92f-4f44-8244-a2af9f83a3ac"
Nov 03 02:30:16.817038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:16.817012    5508 generic.go:356] "Generic (PLEG): container finished" podID="ed674d66-2642-4b30-a4b3-9873263a1cfc" containerID="6f06a6fce67bde74a8a39276fab92c92a23279419805b6762d8ab537cd6970ee" exitCode=60
Nov 03 02:30:16.817172 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:16.817085    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6" event={"ID":"ed674d66-2642-4b30-a4b3-9873263a1cfc","Type":"ContainerDied","Data":"6f06a6fce67bde74a8a39276fab92c92a23279419805b6762d8ab537cd6970ee"}
Nov 03 02:30:16.817587 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:16.817565    5508 scope.go:122] "RemoveContainer" containerID="6f06a6fce67bde74a8a39276fab92c92a23279419805b6762d8ab537cd6970ee"
Nov 03 02:30:16.818027 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:16.818003    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6_odh-ai-gateway-infra(ed674d66-2642-4b30-a4b3-9873263a1cfc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6" podUID="ed674d66-2642-4b30-a4b3-9873263a1cfc"
Nov 03 02:30:16.828095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:16.828077    5508 scope.go:122] "RemoveContainer" containerID="f5b8ff59ff5cd05b6728ab7181ea9172569e2763127a74a5c8ba7e825fcbcc47"
Nov 03 02:30:16.938916 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:16.938805    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"]
Nov 03 02:30:17.021827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.021789    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"]
Nov 03 02:30:17.200595 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.200563    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"
Nov 03 02:30:17.323012 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.322977    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d260f09c-3f76-4662-b580-14e75503724a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d260f09c-3f76-4662-b580-14e75503724a-maas-api-service-ca\") pod \"d260f09c-3f76-4662-b580-14e75503724a\" (UID: \"d260f09c-3f76-4662-b580-14e75503724a\") "
Nov 03 02:30:17.323225 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.323205    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d260f09c-3f76-4662-b580-14e75503724a-kube-api-access-qt2gs\" (UniqueName: \"kubernetes.io/projected/d260f09c-3f76-4662-b580-14e75503724a-kube-api-access-qt2gs\") pod \"d260f09c-3f76-4662-b580-14e75503724a\" (UID: \"d260f09c-3f76-4662-b580-14e75503724a\") "
Nov 03 02:30:17.323432 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.323405    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d260f09c-3f76-4662-b580-14e75503724a-maas-api-service-ca" pod "d260f09c-3f76-4662-b580-14e75503724a" (UID: "d260f09c-3f76-4662-b580-14e75503724a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:30:17.323786 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.323765    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d260f09c-3f76-4662-b580-14e75503724a-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:17.326097 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.326070    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d260f09c-3f76-4662-b580-14e75503724a-kube-api-access-qt2gs" pod "d260f09c-3f76-4662-b580-14e75503724a" (UID: "d260f09c-3f76-4662-b580-14e75503724a"). InnerVolumeSpecName "kube-api-access-qt2gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:30:17.425561 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.425532    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt2gs\" (UniqueName: \"kubernetes.io/projected/d260f09c-3f76-4662-b580-14e75503724a-kube-api-access-qt2gs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:17.684874 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.684782    5508 scope.go:122] "RemoveContainer" containerID="cfa271e30ab97084bda294426043efa6cb2fd5ad207f2f0a3b09f28734218615"
Nov 03 02:30:17.685051 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.684934    5508 scope.go:122] "RemoveContainer" containerID="f1dc911dae279e25791ec6a02fd495232fc384705d62769c98fb933bca0ffa9e"
Nov 03 02:30:17.685297 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.685274    5508 scope.go:122] "RemoveContainer" containerID="2959f353e425b922364ace91a4e68beaf4ac626e7d9a53d74639895b8dce5024"
Nov 03 02:30:17.866645 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.866273    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"
Nov 03 02:30:17.871377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.869597    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w" event={"ID":"d260f09c-3f76-4662-b580-14e75503724a","Type":"ContainerDied","Data":"be508976d5dee88458a4fbba084fb5c02943aa518206865b3084e78d7096164c"}
Nov 03 02:30:17.871377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.869674    5508 scope.go:122] "RemoveContainer" containerID="8380adc62e8f7b0edeabe432695a5e09113a7a26f5b9cc8067861253930db17e"
Nov 03 02:30:17.935390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:17.935283    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"]
Nov 03 02:30:18.056413 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.054456    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"]
Nov 03 02:30:18.065374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.063267    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e1793d-29894550-k4d4w"]
Nov 03 02:30:18.305585 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.305556    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"
Nov 03 02:30:18.343409 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.343382    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"
Nov 03 02:30:18.478463 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.478375    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1e29727-c92f-4f44-8244-a2af9f83a3ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e29727-c92f-4f44-8244-a2af9f83a3ac-maas-api-service-ca\") pod \"f1e29727-c92f-4f44-8244-a2af9f83a3ac\" (UID: \"f1e29727-c92f-4f44-8244-a2af9f83a3ac\") "
Nov 03 02:30:18.478611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.478548    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e29727-c92f-4f44-8244-a2af9f83a3ac-kube-api-access-m44w9\" (UniqueName: \"kubernetes.io/projected/f1e29727-c92f-4f44-8244-a2af9f83a3ac-kube-api-access-m44w9\") pod \"f1e29727-c92f-4f44-8244-a2af9f83a3ac\" (UID: \"f1e29727-c92f-4f44-8244-a2af9f83a3ac\") "
Nov 03 02:30:18.478722 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.478703    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1e29727-c92f-4f44-8244-a2af9f83a3ac-maas-api-service-ca" pod "f1e29727-c92f-4f44-8244-a2af9f83a3ac" (UID: "f1e29727-c92f-4f44-8244-a2af9f83a3ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:30:18.478776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.478709    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed674d66-2642-4b30-a4b3-9873263a1cfc-kube-api-access-nwwbk\" (UniqueName: \"kubernetes.io/projected/ed674d66-2642-4b30-a4b3-9873263a1cfc-kube-api-access-nwwbk\") pod \"ed674d66-2642-4b30-a4b3-9873263a1cfc\" (UID: \"ed674d66-2642-4b30-a4b3-9873263a1cfc\") "
Nov 03 02:30:18.478814 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.478788    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed674d66-2642-4b30-a4b3-9873263a1cfc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed674d66-2642-4b30-a4b3-9873263a1cfc-maas-api-service-ca\") pod \"ed674d66-2642-4b30-a4b3-9873263a1cfc\" (UID: \"ed674d66-2642-4b30-a4b3-9873263a1cfc\") "
Nov 03 02:30:18.479217 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.479182    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed674d66-2642-4b30-a4b3-9873263a1cfc-maas-api-service-ca" pod "ed674d66-2642-4b30-a4b3-9873263a1cfc" (UID: "ed674d66-2642-4b30-a4b3-9873263a1cfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:30:18.479483 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.479461    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1e29727-c92f-4f44-8244-a2af9f83a3ac-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:18.479572 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.479491    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed674d66-2642-4b30-a4b3-9873263a1cfc-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:18.481264 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.481241    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e29727-c92f-4f44-8244-a2af9f83a3ac-kube-api-access-m44w9" pod "f1e29727-c92f-4f44-8244-a2af9f83a3ac" (UID: "f1e29727-c92f-4f44-8244-a2af9f83a3ac"). InnerVolumeSpecName "kube-api-access-m44w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:30:18.481404 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.481303    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed674d66-2642-4b30-a4b3-9873263a1cfc-kube-api-access-nwwbk" pod "ed674d66-2642-4b30-a4b3-9873263a1cfc" (UID: "ed674d66-2642-4b30-a4b3-9873263a1cfc"). InnerVolumeSpecName "kube-api-access-nwwbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:30:18.580994 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.580951    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m44w9\" (UniqueName: \"kubernetes.io/projected/f1e29727-c92f-4f44-8244-a2af9f83a3ac-kube-api-access-m44w9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:18.580994 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.580996    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwwbk\" (UniqueName: \"kubernetes.io/projected/ed674d66-2642-4b30-a4b3-9873263a1cfc-kube-api-access-nwwbk\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:18.694648 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.694615    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d260f09c-3f76-4662-b580-14e75503724a" path="/var/lib/kubelet/pods/d260f09c-3f76-4662-b580-14e75503724a/volumes"
Nov 03 02:30:18.901284 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.901227    5508 generic.go:356] "Generic (PLEG): container finished" podID="25d517d9-1e6d-4106-bf57-4cf3415707b6" containerID="1195de038df49c8d53613664bf6c839f7231fe609bee0fecb8454b5ceba618c7" exitCode=60
Nov 03 02:30:18.901284 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.901269    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq" event={"ID":"25d517d9-1e6d-4106-bf57-4cf3415707b6","Type":"ContainerDied","Data":"1195de038df49c8d53613664bf6c839f7231fe609bee0fecb8454b5ceba618c7"}
Nov 03 02:30:18.901791 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.901344    5508 scope.go:122] "RemoveContainer" containerID="2959f353e425b922364ace91a4e68beaf4ac626e7d9a53d74639895b8dce5024"
Nov 03 02:30:18.901958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.901825    5508 scope.go:122] "RemoveContainer" containerID="1195de038df49c8d53613664bf6c839f7231fe609bee0fecb8454b5ceba618c7"
Nov 03 02:30:18.902189 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:18.902168    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq_odh-ai-gateway-infra(25d517d9-1e6d-4106-bf57-4cf3415707b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq" podUID="25d517d9-1e6d-4106-bf57-4cf3415707b6"
Nov 03 02:30:18.903061 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.903040    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4" event={"ID":"f1e29727-c92f-4f44-8244-a2af9f83a3ac","Type":"ContainerDied","Data":"afd5e130d80768fc210b380f7693f9c9a9d91f9659112b4b7116b0953e4532eb"}
Nov 03 02:30:18.903167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.903124    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"
Nov 03 02:30:18.905206 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.905186    5508 generic.go:356] "Generic (PLEG): container finished" podID="b265ea0a-729a-4d1c-9421-9d43f4d44aba" containerID="7d4bdce4e8cb793b100b28856502080a9a59bdae4e69c2abced0dd37dabb53d5" exitCode=60
Nov 03 02:30:18.905324 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.905257    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw" event={"ID":"b265ea0a-729a-4d1c-9421-9d43f4d44aba","Type":"ContainerDied","Data":"7d4bdce4e8cb793b100b28856502080a9a59bdae4e69c2abced0dd37dabb53d5"}
Nov 03 02:30:18.905650 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.905635    5508 scope.go:122] "RemoveContainer" containerID="7d4bdce4e8cb793b100b28856502080a9a59bdae4e69c2abced0dd37dabb53d5"
Nov 03 02:30:18.905922 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:18.905899    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw_odh-ai-gateway-infra(b265ea0a-729a-4d1c-9421-9d43f4d44aba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw" podUID="b265ea0a-729a-4d1c-9421-9d43f4d44aba"
Nov 03 02:30:18.907193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.907173    5508 generic.go:356] "Generic (PLEG): container finished" podID="1d218587-4638-4bf3-b319-5cbafcbb7933" containerID="34fcea86f0e041b0371cbdd007194bee7be673723be69e1f92eac0fe98d617d2" exitCode=60
Nov 03 02:30:18.907278 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.907244    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6" event={"ID":"1d218587-4638-4bf3-b319-5cbafcbb7933","Type":"ContainerDied","Data":"34fcea86f0e041b0371cbdd007194bee7be673723be69e1f92eac0fe98d617d2"}
Nov 03 02:30:18.907699 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.907673    5508 scope.go:122] "RemoveContainer" containerID="34fcea86f0e041b0371cbdd007194bee7be673723be69e1f92eac0fe98d617d2"
Nov 03 02:30:18.907962 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:18.907939    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6_odh-ai-gateway-infra(1d218587-4638-4bf3-b319-5cbafcbb7933)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6" podUID="1d218587-4638-4bf3-b319-5cbafcbb7933"
Nov 03 02:30:18.908678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.908663    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"
Nov 03 02:30:18.908761 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.908705    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6" event={"ID":"ed674d66-2642-4b30-a4b3-9873263a1cfc","Type":"ContainerDied","Data":"0434b519092d486da3879213317a710ad1e851e5ed7128fa377830de3e0a9184"}
Nov 03 02:30:18.957539 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.957518    5508 scope.go:122] "RemoveContainer" containerID="821890f16f30769fa600245f87f74643a65f9ee821dbd6a203bae87e9ff8b922"
Nov 03 02:30:18.967384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.967339    5508 scope.go:122] "RemoveContainer" containerID="cfa271e30ab97084bda294426043efa6cb2fd5ad207f2f0a3b09f28734218615"
Nov 03 02:30:18.980879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.980859    5508 scope.go:122] "RemoveContainer" containerID="f1dc911dae279e25791ec6a02fd495232fc384705d62769c98fb933bca0ffa9e"
Nov 03 02:30:18.994317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:18.994294    5508 scope.go:122] "RemoveContainer" containerID="6f06a6fce67bde74a8a39276fab92c92a23279419805b6762d8ab537cd6970ee"
Nov 03 02:30:19.117097 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:19.117025    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"]
Nov 03 02:30:19.124103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:19.124076    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-059b23-29894550-zg7x6"]
Nov 03 02:30:19.445191 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:19.445158    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"]
Nov 03 02:30:19.453231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:19.453194    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9576a-29894550-c89k4"]
Nov 03 02:30:20.022156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.022123    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"]
Nov 03 02:30:20.190819 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.190792    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"
Nov 03 02:30:20.239057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.239011    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"]
Nov 03 02:30:20.304878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.304825    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d517d9-1e6d-4106-bf57-4cf3415707b6-kube-api-access-xhwqt\" (UniqueName: \"kubernetes.io/projected/25d517d9-1e6d-4106-bf57-4cf3415707b6-kube-api-access-xhwqt\") pod \"25d517d9-1e6d-4106-bf57-4cf3415707b6\" (UID: \"25d517d9-1e6d-4106-bf57-4cf3415707b6\") "
Nov 03 02:30:20.305127 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.305104    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d517d9-1e6d-4106-bf57-4cf3415707b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d517d9-1e6d-4106-bf57-4cf3415707b6-maas-api-service-ca\") pod \"25d517d9-1e6d-4106-bf57-4cf3415707b6\" (UID: \"25d517d9-1e6d-4106-bf57-4cf3415707b6\") "
Nov 03 02:30:20.308994 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.308969    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d517d9-1e6d-4106-bf57-4cf3415707b6-maas-api-service-ca" pod "25d517d9-1e6d-4106-bf57-4cf3415707b6" (UID: "25d517d9-1e6d-4106-bf57-4cf3415707b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:30:20.312603 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.312576    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d517d9-1e6d-4106-bf57-4cf3415707b6-kube-api-access-xhwqt" pod "25d517d9-1e6d-4106-bf57-4cf3415707b6" (UID: "25d517d9-1e6d-4106-bf57-4cf3415707b6"). InnerVolumeSpecName "kube-api-access-xhwqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:30:20.348957 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.348926    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"]
Nov 03 02:30:20.410137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.410105    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d517d9-1e6d-4106-bf57-4cf3415707b6-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:20.410137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.410132    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhwqt\" (UniqueName: \"kubernetes.io/projected/25d517d9-1e6d-4106-bf57-4cf3415707b6-kube-api-access-xhwqt\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:20.422762 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.422733    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"
Nov 03 02:30:20.511048 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.511015    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b265ea0a-729a-4d1c-9421-9d43f4d44aba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b265ea0a-729a-4d1c-9421-9d43f4d44aba-maas-api-service-ca\") pod \"b265ea0a-729a-4d1c-9421-9d43f4d44aba\" (UID: \"b265ea0a-729a-4d1c-9421-9d43f4d44aba\") "
Nov 03 02:30:20.511385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.511269    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b265ea0a-729a-4d1c-9421-9d43f4d44aba-kube-api-access-cl8vq\" (UniqueName: \"kubernetes.io/projected/b265ea0a-729a-4d1c-9421-9d43f4d44aba-kube-api-access-cl8vq\") pod \"b265ea0a-729a-4d1c-9421-9d43f4d44aba\" (UID: \"b265ea0a-729a-4d1c-9421-9d43f4d44aba\") "
Nov 03 02:30:20.511471 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.511417    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b265ea0a-729a-4d1c-9421-9d43f4d44aba-maas-api-service-ca" pod "b265ea0a-729a-4d1c-9421-9d43f4d44aba" (UID: "b265ea0a-729a-4d1c-9421-9d43f4d44aba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:30:20.511823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.511799    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b265ea0a-729a-4d1c-9421-9d43f4d44aba-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:20.513762 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.513738    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b265ea0a-729a-4d1c-9421-9d43f4d44aba-kube-api-access-cl8vq" pod "b265ea0a-729a-4d1c-9421-9d43f4d44aba" (UID: "b265ea0a-729a-4d1c-9421-9d43f4d44aba"). InnerVolumeSpecName "kube-api-access-cl8vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:30:20.537358 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.537321    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"
Nov 03 02:30:20.612280 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.612196    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d218587-4638-4bf3-b319-5cbafcbb7933-kube-api-access-jbqms\" (UniqueName: \"kubernetes.io/projected/1d218587-4638-4bf3-b319-5cbafcbb7933-kube-api-access-jbqms\") pod \"1d218587-4638-4bf3-b319-5cbafcbb7933\" (UID: \"1d218587-4638-4bf3-b319-5cbafcbb7933\") "
Nov 03 02:30:20.612495 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.612313    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d218587-4638-4bf3-b319-5cbafcbb7933-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d218587-4638-4bf3-b319-5cbafcbb7933-maas-api-service-ca\") pod \"1d218587-4638-4bf3-b319-5cbafcbb7933\" (UID: \"1d218587-4638-4bf3-b319-5cbafcbb7933\") "
Nov 03 02:30:20.612690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.612665    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d218587-4638-4bf3-b319-5cbafcbb7933-maas-api-service-ca" pod "1d218587-4638-4bf3-b319-5cbafcbb7933" (UID: "1d218587-4638-4bf3-b319-5cbafcbb7933"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:30:20.612757 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.612742    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl8vq\" (UniqueName: \"kubernetes.io/projected/b265ea0a-729a-4d1c-9421-9d43f4d44aba-kube-api-access-cl8vq\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:20.614322 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.614302    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d218587-4638-4bf3-b319-5cbafcbb7933-kube-api-access-jbqms" pod "1d218587-4638-4bf3-b319-5cbafcbb7933" (UID: "1d218587-4638-4bf3-b319-5cbafcbb7933"). InnerVolumeSpecName "kube-api-access-jbqms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:30:20.694714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.694675    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed674d66-2642-4b30-a4b3-9873263a1cfc" path="/var/lib/kubelet/pods/ed674d66-2642-4b30-a4b3-9873263a1cfc/volumes"
Nov 03 02:30:20.695392 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.695370    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e29727-c92f-4f44-8244-a2af9f83a3ac" path="/var/lib/kubelet/pods/f1e29727-c92f-4f44-8244-a2af9f83a3ac/volumes"
Nov 03 02:30:20.715668 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.715639    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbqms\" (UniqueName: \"kubernetes.io/projected/1d218587-4638-4bf3-b319-5cbafcbb7933-kube-api-access-jbqms\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:20.715788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.715671    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d218587-4638-4bf3-b319-5cbafcbb7933-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:30:20.932724 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.932692    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"
Nov 03 02:30:20.932724 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.932715    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq" event={"ID":"25d517d9-1e6d-4106-bf57-4cf3415707b6","Type":"ContainerDied","Data":"1058969b38e31f95646a1638624c55eac0287610e26cbe9396e7629ad4dae702"}
Nov 03 02:30:20.933010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.932814    5508 scope.go:122] "RemoveContainer" containerID="1195de038df49c8d53613664bf6c839f7231fe609bee0fecb8454b5ceba618c7"
Nov 03 02:30:20.934425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.934382    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw" event={"ID":"b265ea0a-729a-4d1c-9421-9d43f4d44aba","Type":"ContainerDied","Data":"371c1bc187ea3e1f5e51585511cdbb3a7967d3c7103cec50fa9d8c143bda9d55"}
Nov 03 02:30:20.934548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.934440    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"
Nov 03 02:30:20.936189 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.936167    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"
Nov 03 02:30:20.936303 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.936167    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6" event={"ID":"1d218587-4638-4bf3-b319-5cbafcbb7933","Type":"ContainerDied","Data":"f9473252eeb2a80007822068b5de4c96c42b2cded3943670f89a46ca51c8c57b"}
Nov 03 02:30:20.944540 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.944516    5508 scope.go:122] "RemoveContainer" containerID="7d4bdce4e8cb793b100b28856502080a9a59bdae4e69c2abced0dd37dabb53d5"
Nov 03 02:30:20.955374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:20.955340    5508 scope.go:122] "RemoveContainer" containerID="34fcea86f0e041b0371cbdd007194bee7be673723be69e1f92eac0fe98d617d2"
Nov 03 02:30:21.086826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:21.086792    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"]
Nov 03 02:30:21.094368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:21.094315    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-c34879-29894550-xgtl6"]
Nov 03 02:30:21.193577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:21.193505    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"]
Nov 03 02:30:21.199196 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:21.199168    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-9da431-29894550-f8wdq"]
Nov 03 02:30:21.300723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:21.300695    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"]
Nov 03 02:30:21.304151 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:21.304122    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e40047-29894550-d6lmw"]
Nov 03 02:30:22.693314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:22.693283    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d218587-4638-4bf3-b319-5cbafcbb7933" path="/var/lib/kubelet/pods/1d218587-4638-4bf3-b319-5cbafcbb7933/volumes"
Nov 03 02:30:22.693766 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:22.693751    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d517d9-1e6d-4106-bf57-4cf3415707b6" path="/var/lib/kubelet/pods/25d517d9-1e6d-4106-bf57-4cf3415707b6/volumes"
Nov 03 02:30:22.694128 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:22.694115    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b265ea0a-729a-4d1c-9421-9d43f4d44aba" path="/var/lib/kubelet/pods/b265ea0a-729a-4d1c-9421-9d43f4d44aba/volumes"
Nov 03 02:30:43.897973 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:30:43.897844    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:30:58.547452 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:58.547415    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:30:59.584636 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:59.584556    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"]
Nov 03 02:30:59.585150 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:59.584760    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr" podUID="d5bbb584-5431-4d36-a98c-e9ed8be3114b" containerName="maas-api" containerID="cri-o://3df6793e03a5ef3a1d04b62acfebfe7db0b06ba552b93b31ac2029a393163080" gracePeriod=30
Nov 03 02:30:59.913932 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:30:59.913909    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:31:00.084962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.084925    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-metrics-certs\") pod \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") "
Nov 03 02:31:00.085171 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.085153    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5bbb584-5431-4d36-a98c-e9ed8be3114b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bbb584-5431-4d36-a98c-e9ed8be3114b-openshift-service-ca\") pod \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") "
Nov 03 02:31:00.085239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.085219    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5bbb584-5431-4d36-a98c-e9ed8be3114b-kube-api-access-2c6r8\" (UniqueName: \"kubernetes.io/projected/d5bbb584-5431-4d36-a98c-e9ed8be3114b-kube-api-access-2c6r8\") pod \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") "
Nov 03 02:31:00.085301 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.085290    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-maas-api-tls\") pod \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\" (UID: \"d5bbb584-5431-4d36-a98c-e9ed8be3114b\") "
Nov 03 02:31:00.085632 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.085599    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5bbb584-5431-4d36-a98c-e9ed8be3114b-openshift-service-ca" pod "d5bbb584-5431-4d36-a98c-e9ed8be3114b" (UID: "d5bbb584-5431-4d36-a98c-e9ed8be3114b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:31:00.085851 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.085831    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bbb584-5431-4d36-a98c-e9ed8be3114b-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:00.087521 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.087496    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-maas-api-tls" pod "d5bbb584-5431-4d36-a98c-e9ed8be3114b" (UID: "d5bbb584-5431-4d36-a98c-e9ed8be3114b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:31:00.087726 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.087701    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5bbb584-5431-4d36-a98c-e9ed8be3114b-kube-api-access-2c6r8" pod "d5bbb584-5431-4d36-a98c-e9ed8be3114b" (UID: "d5bbb584-5431-4d36-a98c-e9ed8be3114b"). InnerVolumeSpecName "kube-api-access-2c6r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:31:00.088445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.088428    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-metrics-certs" pod "d5bbb584-5431-4d36-a98c-e9ed8be3114b" (UID: "d5bbb584-5431-4d36-a98c-e9ed8be3114b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:31:00.186610 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.186576    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c6r8\" (UniqueName: \"kubernetes.io/projected/d5bbb584-5431-4d36-a98c-e9ed8be3114b-kube-api-access-2c6r8\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:00.186610 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.186603    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:00.186610 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.186615    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5bbb584-5431-4d36-a98c-e9ed8be3114b-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:00.252780 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.252743    5508 generic.go:356] "Generic (PLEG): container finished" podID="d5bbb584-5431-4d36-a98c-e9ed8be3114b" containerID="3df6793e03a5ef3a1d04b62acfebfe7db0b06ba552b93b31ac2029a393163080" exitCode=0
Nov 03 02:31:00.252949 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.252803    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"
Nov 03 02:31:00.252949 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.252824    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr" event={"ID":"d5bbb584-5431-4d36-a98c-e9ed8be3114b","Type":"ContainerDied","Data":"3df6793e03a5ef3a1d04b62acfebfe7db0b06ba552b93b31ac2029a393163080"}
Nov 03 02:31:00.252949 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.252882    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr" event={"ID":"d5bbb584-5431-4d36-a98c-e9ed8be3114b","Type":"ContainerDied","Data":"85b3032493854ebb44449bac861fcb99f4c9e72ab6bea6033ce973022c06094b"}
Nov 03 02:31:00.252949 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.252904    5508 scope.go:122] "RemoveContainer" containerID="3df6793e03a5ef3a1d04b62acfebfe7db0b06ba552b93b31ac2029a393163080"
Nov 03 02:31:00.264282 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.264261    5508 scope.go:122] "RemoveContainer" containerID="3df6793e03a5ef3a1d04b62acfebfe7db0b06ba552b93b31ac2029a393163080"
Nov 03 02:31:00.264579 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:31:00.264560    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df6793e03a5ef3a1d04b62acfebfe7db0b06ba552b93b31ac2029a393163080\": container with ID starting with 3df6793e03a5ef3a1d04b62acfebfe7db0b06ba552b93b31ac2029a393163080 not found: ID does not exist" containerID="3df6793e03a5ef3a1d04b62acfebfe7db0b06ba552b93b31ac2029a393163080"
Nov 03 02:31:00.264635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.264591    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df6793e03a5ef3a1d04b62acfebfe7db0b06ba552b93b31ac2029a393163080"} err="failed to get container status \"3df6793e03a5ef3a1d04b62acfebfe7db0b06ba552b93b31ac2029a393163080\": rpc error: code = NotFound desc = could not find container \"3df6793e03a5ef3a1d04b62acfebfe7db0b06ba552b93b31ac2029a393163080\": container with ID starting with 3df6793e03a5ef3a1d04b62acfebfe7db0b06ba552b93b31ac2029a393163080 not found: ID does not exist"
Nov 03 02:31:00.406727 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.406691    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"]
Nov 03 02:31:00.410998 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.410968    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e051f3-7d699948b-spctr"]
Nov 03 02:31:00.695491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:00.695457    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5bbb584-5431-4d36-a98c-e9ed8be3114b" path="/var/lib/kubelet/pods/d5bbb584-5431-4d36-a98c-e9ed8be3114b/volumes"
Nov 03 02:31:13.812844 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:31:13.812810    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:31:18.487608 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:18.487572    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"]
Nov 03 02:31:23.488419 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:23.488378    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl" podUID="bbc91df0-c89c-45b8-a08e-2f37fd911639" containerName="payload-pre-processing" containerID="cri-o://020d55b932e9bce4b009f8ce13fd1307de3d52c1cd898a4bec2aa2ff71a5ad8b" gracePeriod=25
Nov 03 02:31:23.758377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:23.758305    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"
Nov 03 02:31:23.836514 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:23.836480    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc91df0-c89c-45b8-a08e-2f37fd911639-kube-api-access-wpkhc\" (UniqueName: \"kubernetes.io/projected/bbc91df0-c89c-45b8-a08e-2f37fd911639-kube-api-access-wpkhc\") pod \"bbc91df0-c89c-45b8-a08e-2f37fd911639\" (UID: \"bbc91df0-c89c-45b8-a08e-2f37fd911639\") "
Nov 03 02:31:23.836710 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:23.836679    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc91df0-c89c-45b8-a08e-2f37fd911639-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc91df0-c89c-45b8-a08e-2f37fd911639-plugins-config-volume\") pod \"bbc91df0-c89c-45b8-a08e-2f37fd911639\" (UID: \"bbc91df0-c89c-45b8-a08e-2f37fd911639\") "
Nov 03 02:31:23.837069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:23.837043    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc91df0-c89c-45b8-a08e-2f37fd911639-plugins-config-volume" pod "bbc91df0-c89c-45b8-a08e-2f37fd911639" (UID: "bbc91df0-c89c-45b8-a08e-2f37fd911639"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:31:23.838664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:23.838640    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc91df0-c89c-45b8-a08e-2f37fd911639-kube-api-access-wpkhc" pod "bbc91df0-c89c-45b8-a08e-2f37fd911639" (UID: "bbc91df0-c89c-45b8-a08e-2f37fd911639"). InnerVolumeSpecName "kube-api-access-wpkhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:31:23.938750 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:23.938707    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc91df0-c89c-45b8-a08e-2f37fd911639-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:23.938750 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:23.938743    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpkhc\" (UniqueName: \"kubernetes.io/projected/bbc91df0-c89c-45b8-a08e-2f37fd911639-kube-api-access-wpkhc\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:24.497293 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:24.497253    5508 generic.go:356] "Generic (PLEG): container finished" podID="bbc91df0-c89c-45b8-a08e-2f37fd911639" containerID="020d55b932e9bce4b009f8ce13fd1307de3d52c1cd898a4bec2aa2ff71a5ad8b" exitCode=0
Nov 03 02:31:24.497293 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:24.497300    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"
Nov 03 02:31:24.497859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:24.497339    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl" event={"ID":"bbc91df0-c89c-45b8-a08e-2f37fd911639","Type":"ContainerDied","Data":"020d55b932e9bce4b009f8ce13fd1307de3d52c1cd898a4bec2aa2ff71a5ad8b"}
Nov 03 02:31:24.497859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:24.497463    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl" event={"ID":"bbc91df0-c89c-45b8-a08e-2f37fd911639","Type":"ContainerDied","Data":"f05ba175d141a37f511c64fc64914cb8ed0ba21480e6fd2ca340636b106e061a"}
Nov 03 02:31:24.497859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:24.497492    5508 scope.go:122] "RemoveContainer" containerID="020d55b932e9bce4b009f8ce13fd1307de3d52c1cd898a4bec2aa2ff71a5ad8b"
Nov 03 02:31:24.509277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:24.509259    5508 scope.go:122] "RemoveContainer" containerID="020d55b932e9bce4b009f8ce13fd1307de3d52c1cd898a4bec2aa2ff71a5ad8b"
Nov 03 02:31:24.509600 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:31:24.509581    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020d55b932e9bce4b009f8ce13fd1307de3d52c1cd898a4bec2aa2ff71a5ad8b\": container with ID starting with 020d55b932e9bce4b009f8ce13fd1307de3d52c1cd898a4bec2aa2ff71a5ad8b not found: ID does not exist" containerID="020d55b932e9bce4b009f8ce13fd1307de3d52c1cd898a4bec2aa2ff71a5ad8b"
Nov 03 02:31:24.509689 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:24.509609    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020d55b932e9bce4b009f8ce13fd1307de3d52c1cd898a4bec2aa2ff71a5ad8b"} err="failed to get container status \"020d55b932e9bce4b009f8ce13fd1307de3d52c1cd898a4bec2aa2ff71a5ad8b\": rpc error: code = NotFound desc = could not find container \"020d55b932e9bce4b009f8ce13fd1307de3d52c1cd898a4bec2aa2ff71a5ad8b\": container with ID starting with 020d55b932e9bce4b009f8ce13fd1307de3d52c1cd898a4bec2aa2ff71a5ad8b not found: ID does not exist"
Nov 03 02:31:24.652817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:24.652781    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"]
Nov 03 02:31:24.699035 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:24.699000    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e40047-6f46994c75-xqbpl"]
Nov 03 02:31:26.693735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:26.693699    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc91df0-c89c-45b8-a08e-2f37fd911639" path="/var/lib/kubelet/pods/bbc91df0-c89c-45b8-a08e-2f37fd911639/volumes"
Nov 03 02:31:35.441835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:35.441798    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"]
Nov 03 02:31:35.442255 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:35.442066    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerName="istio-proxy" containerID="cri-o://7e4698ad05328f185f80155390246e55c559ee68758ccbf9e33cf2f4addaf5f8" gracePeriod=30
Nov 03 02:31:38.033901 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.033864    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:31:38.491174 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491143    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"]
Nov 03 02:31:38.491736 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491722    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d5bbb584-5431-4d36-a98c-e9ed8be3114b" containerName="maas-api"
Nov 03 02:31:38.491825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491737    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5bbb584-5431-4d36-a98c-e9ed8be3114b" containerName="maas-api"
Nov 03 02:31:38.491825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491751    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ed674d66-2642-4b30-a4b3-9873263a1cfc" containerName="cleanup"
Nov 03 02:31:38.491825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491756    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed674d66-2642-4b30-a4b3-9873263a1cfc" containerName="cleanup"
Nov 03 02:31:38.491825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491766    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d260f09c-3f76-4662-b580-14e75503724a" containerName="cleanup"
Nov 03 02:31:38.491825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491771    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d260f09c-3f76-4662-b580-14e75503724a" containerName="cleanup"
Nov 03 02:31:38.491825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491778    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b265ea0a-729a-4d1c-9421-9d43f4d44aba" containerName="cleanup"
Nov 03 02:31:38.491825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491783    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b265ea0a-729a-4d1c-9421-9d43f4d44aba" containerName="cleanup"
Nov 03 02:31:38.491825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491793    5508 cpu_manager.go:414] "containerMap: removing container" podUID="25d517d9-1e6d-4106-bf57-4cf3415707b6" containerName="cleanup"
Nov 03 02:31:38.491825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491798    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d517d9-1e6d-4106-bf57-4cf3415707b6" containerName="cleanup"
Nov 03 02:31:38.491825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491809    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f1e29727-c92f-4f44-8244-a2af9f83a3ac" containerName="cleanup"
Nov 03 02:31:38.491825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491816    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e29727-c92f-4f44-8244-a2af9f83a3ac" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491846    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f1e29727-c92f-4f44-8244-a2af9f83a3ac" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491854    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e29727-c92f-4f44-8244-a2af9f83a3ac" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491865    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b265ea0a-729a-4d1c-9421-9d43f4d44aba" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491872    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b265ea0a-729a-4d1c-9421-9d43f4d44aba" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491882    5508 cpu_manager.go:414] "containerMap: removing container" podUID="bbc91df0-c89c-45b8-a08e-2f37fd911639" containerName="payload-pre-processing"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491887    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc91df0-c89c-45b8-a08e-2f37fd911639" containerName="payload-pre-processing"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491898    5508 cpu_manager.go:414] "containerMap: removing container" podUID="1d218587-4638-4bf3-b319-5cbafcbb7933" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491903    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d218587-4638-4bf3-b319-5cbafcbb7933" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491916    5508 cpu_manager.go:414] "containerMap: removing container" podUID="1d218587-4638-4bf3-b319-5cbafcbb7933" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491923    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d218587-4638-4bf3-b319-5cbafcbb7933" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491935    5508 cpu_manager.go:414] "containerMap: removing container" podUID="25d517d9-1e6d-4106-bf57-4cf3415707b6" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491941    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d517d9-1e6d-4106-bf57-4cf3415707b6" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491955    5508 cpu_manager.go:414] "containerMap: removing container" podUID="1d218587-4638-4bf3-b319-5cbafcbb7933" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491964    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d218587-4638-4bf3-b319-5cbafcbb7933" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491975    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ed674d66-2642-4b30-a4b3-9873263a1cfc" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491981    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed674d66-2642-4b30-a4b3-9873263a1cfc" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.491996    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d260f09c-3f76-4662-b580-14e75503724a" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492005    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d260f09c-3f76-4662-b580-14e75503724a" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492018    5508 cpu_manager.go:414] "containerMap: removing container" podUID="501b9d64-d4d3-403f-a321-17935b78eaf2" containerName="revoke-keys"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492025    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501b9d64-d4d3-403f-a321-17935b78eaf2" containerName="revoke-keys"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492182    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e29727-c92f-4f44-8244-a2af9f83a3ac" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492197    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d218587-4638-4bf3-b319-5cbafcbb7933" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492206    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d517d9-1e6d-4106-bf57-4cf3415707b6" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492215    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d218587-4638-4bf3-b319-5cbafcbb7933" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492225    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed674d66-2642-4b30-a4b3-9873263a1cfc" containerName="cleanup"
Nov 03 02:31:38.492270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492233    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d260f09c-3f76-4662-b580-14e75503724a" containerName="cleanup"
Nov 03 02:31:38.493329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492379    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b265ea0a-729a-4d1c-9421-9d43f4d44aba" containerName="cleanup"
Nov 03 02:31:38.493329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492393    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d517d9-1e6d-4106-bf57-4cf3415707b6" containerName="cleanup"
Nov 03 02:31:38.493329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492400    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="501b9d64-d4d3-403f-a321-17935b78eaf2" containerName="revoke-keys"
Nov 03 02:31:38.493329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492406    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d260f09c-3f76-4662-b580-14e75503724a" containerName="cleanup"
Nov 03 02:31:38.493329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492425    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5bbb584-5431-4d36-a98c-e9ed8be3114b" containerName="maas-api"
Nov 03 02:31:38.493329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492435    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed674d66-2642-4b30-a4b3-9873263a1cfc" containerName="cleanup"
Nov 03 02:31:38.493329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492443    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b265ea0a-729a-4d1c-9421-9d43f4d44aba" containerName="cleanup"
Nov 03 02:31:38.493329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492450    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d517d9-1e6d-4106-bf57-4cf3415707b6" containerName="cleanup"
Nov 03 02:31:38.493329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492462    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e29727-c92f-4f44-8244-a2af9f83a3ac" containerName="cleanup"
Nov 03 02:31:38.493329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.492467    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc91df0-c89c-45b8-a08e-2f37fd911639" containerName="payload-pre-processing"
Nov 03 02:31:38.497573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.497554    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"
Nov 03 02:31:38.549471 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.549436    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:31:38.595405 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.595369    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:31:38.599415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.599386    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"]
Nov 03 02:31:38.650906 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.650867    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0be01c8e-lw255\" (UID: \"98fda50c-5d63-472f-a465-faca6a3b95fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"
Nov 03 02:31:38.651098 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.650933    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0be01c8e-lw255\" (UID: \"98fda50c-5d63-472f-a465-faca6a3b95fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"
Nov 03 02:31:38.651098 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.651026    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bwjn\" (UniqueName: \"kubernetes.io/projected/98fda50c-5d63-472f-a465-faca6a3b95fb-kube-api-access-7bwjn\") pod \"maas-api-revoke-sub-0be01c8e-lw255\" (UID: \"98fda50c-5d63-472f-a465-faca6a3b95fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"
Nov 03 02:31:38.751868 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.751772    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0be01c8e-lw255\" (UID: \"98fda50c-5d63-472f-a465-faca6a3b95fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"
Nov 03 02:31:38.751868 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.751820    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bwjn\" (UniqueName: \"kubernetes.io/projected/98fda50c-5d63-472f-a465-faca6a3b95fb-kube-api-access-7bwjn\") pod \"maas-api-revoke-sub-0be01c8e-lw255\" (UID: \"98fda50c-5d63-472f-a465-faca6a3b95fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"
Nov 03 02:31:38.752116 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.752052    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0be01c8e-lw255\" (UID: \"98fda50c-5d63-472f-a465-faca6a3b95fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"
Nov 03 02:31:38.752519 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.752489    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0be01c8e-lw255\" (UID: \"98fda50c-5d63-472f-a465-faca6a3b95fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"
Nov 03 02:31:38.754617 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.754595    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0be01c8e-lw255\" (UID: \"98fda50c-5d63-472f-a465-faca6a3b95fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"
Nov 03 02:31:38.809318 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.809278    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bwjn\" (UniqueName: \"kubernetes.io/projected/98fda50c-5d63-472f-a465-faca6a3b95fb-kube-api-access-7bwjn\") pod \"maas-api-revoke-sub-0be01c8e-lw255\" (UID: \"98fda50c-5d63-472f-a465-faca6a3b95fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"
Nov 03 02:31:38.811986 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:38.811964    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"
Nov 03 02:31:39.192161 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:39.192128    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"]
Nov 03 02:31:39.198812 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:31:39.198780    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fda50c_5d63_472f_a465_faca6a3b95fb.slice/crio-7f8c717c0843f66cc4d2c20e32edb147aac8caf2779619926e9f25586284c409 WatchSource:0}: Error finding container 7f8c717c0843f66cc4d2c20e32edb147aac8caf2779619926e9f25586284c409: Status 404 returned error can't find the container with id 7f8c717c0843f66cc4d2c20e32edb147aac8caf2779619926e9f25586284c409
Nov 03 02:31:39.637046 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:39.637010    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255" event={"ID":"98fda50c-5d63-472f-a465-faca6a3b95fb","Type":"ContainerStarted","Data":"2e22bb0083b83ca727e0a1bef9f01b0c73edbfab123cae65699a370634d04130"}
Nov 03 02:31:39.637235 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:39.637067    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255" event={"ID":"98fda50c-5d63-472f-a465-faca6a3b95fb","Type":"ContainerStarted","Data":"7f8c717c0843f66cc4d2c20e32edb147aac8caf2779619926e9f25586284c409"}
Nov 03 02:31:39.745531 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:39.745402    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255" podStartSLOduration=1.745385014 podStartE2EDuration="1.745385014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:31:39.744889149 +0000 UTC m=+4379.170495632" watchObservedRunningTime="2026-11-03 02:31:39.745385014 +0000 UTC m=+4379.170991484"
Nov 03 02:31:40.648294 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.648262    5508 generic.go:356] "Generic (PLEG): container finished" podID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerID="7e4698ad05328f185f80155390246e55c559ee68758ccbf9e33cf2f4addaf5f8" exitCode=0
Nov 03 02:31:40.648769 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.648334    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" event={"ID":"f46dc7e6-e02b-4201-baad-c3c0764a0216","Type":"ContainerDied","Data":"7e4698ad05328f185f80155390246e55c559ee68758ccbf9e33cf2f4addaf5f8"}
Nov 03 02:31:40.648769 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.648465    5508 scope.go:122] "RemoveContainer" containerID="5e4d80ce27f51c1ddb795eadb643e7c9269c9f3a0a5506341edcfbfc0f838d23"
Nov 03 02:31:40.651764 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.651733    5508 generic.go:356] "Generic (PLEG): container finished" podID="98fda50c-5d63-472f-a465-faca6a3b95fb" containerID="2e22bb0083b83ca727e0a1bef9f01b0c73edbfab123cae65699a370634d04130" exitCode=0
Nov 03 02:31:40.651909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.651822    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255" event={"ID":"98fda50c-5d63-472f-a465-faca6a3b95fb","Type":"ContainerDied","Data":"2e22bb0083b83ca727e0a1bef9f01b0c73edbfab123cae65699a370634d04130"}
Nov 03 02:31:40.731923 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.731895    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:31:40.778492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.778456    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-envoy\") pod \"f46dc7e6-e02b-4201-baad-c3c0764a0216\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") "
Nov 03 02:31:40.778682 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.778524    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-podinfo\") pod \"f46dc7e6-e02b-4201-baad-c3c0764a0216\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") "
Nov 03 02:31:40.778682 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.778577    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-certs\") pod \"f46dc7e6-e02b-4201-baad-c3c0764a0216\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") "
Nov 03 02:31:40.778682 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.778620    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-socket\") pod \"f46dc7e6-e02b-4201-baad-c3c0764a0216\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") "
Nov 03 02:31:40.778878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.778767    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46dc7e6-e02b-4201-baad-c3c0764a0216-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46dc7e6-e02b-4201-baad-c3c0764a0216-istiod-ca-cert\") pod \"f46dc7e6-e02b-4201-baad-c3c0764a0216\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") "
Nov 03 02:31:40.778878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.778825    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-credential-socket\") pod \"f46dc7e6-e02b-4201-baad-c3c0764a0216\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") "
Nov 03 02:31:40.778878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.778863    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-data\") pod \"f46dc7e6-e02b-4201-baad-c3c0764a0216\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") "
Nov 03 02:31:40.779053 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.778881    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-certs" pod "f46dc7e6-e02b-4201-baad-c3c0764a0216" (UID: "f46dc7e6-e02b-4201-baad-c3c0764a0216"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:31:40.779053 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.778893    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-token\" (UniqueName: \"kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-token\") pod \"f46dc7e6-e02b-4201-baad-c3c0764a0216\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") "
Nov 03 02:31:40.779053 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.778989    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-kube-api-access-wljm4\" (UniqueName: \"kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-kube-api-access-wljm4\") pod \"f46dc7e6-e02b-4201-baad-c3c0764a0216\" (UID: \"f46dc7e6-e02b-4201-baad-c3c0764a0216\") "
Nov 03 02:31:40.779263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.779233    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46dc7e6-e02b-4201-baad-c3c0764a0216-istiod-ca-cert" pod "f46dc7e6-e02b-4201-baad-c3c0764a0216" (UID: "f46dc7e6-e02b-4201-baad-c3c0764a0216"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:31:40.779655 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.779634    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:40.779778 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.779662    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46dc7e6-e02b-4201-baad-c3c0764a0216-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:40.780156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.779937    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-socket" pod "f46dc7e6-e02b-4201-baad-c3c0764a0216" (UID: "f46dc7e6-e02b-4201-baad-c3c0764a0216"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:31:40.780156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.780113    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-credential-socket" pod "f46dc7e6-e02b-4201-baad-c3c0764a0216" (UID: "f46dc7e6-e02b-4201-baad-c3c0764a0216"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:31:40.780156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.780145    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-data" pod "f46dc7e6-e02b-4201-baad-c3c0764a0216" (UID: "f46dc7e6-e02b-4201-baad-c3c0764a0216"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:31:40.781440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.781414    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-envoy" pod "f46dc7e6-e02b-4201-baad-c3c0764a0216" (UID: "f46dc7e6-e02b-4201-baad-c3c0764a0216"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:31:40.781550 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.781480    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-token" pod "f46dc7e6-e02b-4201-baad-c3c0764a0216" (UID: "f46dc7e6-e02b-4201-baad-c3c0764a0216"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:31:40.782175 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.782156    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-kube-api-access-wljm4" pod "f46dc7e6-e02b-4201-baad-c3c0764a0216" (UID: "f46dc7e6-e02b-4201-baad-c3c0764a0216"). InnerVolumeSpecName "kube-api-access-wljm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:31:40.788037 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.788015    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-podinfo" pod "f46dc7e6-e02b-4201-baad-c3c0764a0216" (UID: "f46dc7e6-e02b-4201-baad-c3c0764a0216"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:31:40.880983 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.880937    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:40.880983 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.880972    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:40.880983 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.880987    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:40.881290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.881000    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:40.881290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.881014    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:40.881290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.881028    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:40.881290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:40.881042    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wljm4\" (UniqueName: \"kubernetes.io/projected/f46dc7e6-e02b-4201-baad-c3c0764a0216-kube-api-access-wljm4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:41.660326 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:41.660289    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"
Nov 03 02:31:41.660859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:41.660319    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv" event={"ID":"f46dc7e6-e02b-4201-baad-c3c0764a0216","Type":"ContainerDied","Data":"61c7f5a3c1c0498b4d198e7d4df8b4202e4e455ed054b72f29655f32aa066677"}
Nov 03 02:31:41.660859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:41.660413    5508 scope.go:122] "RemoveContainer" containerID="7e4698ad05328f185f80155390246e55c559ee68758ccbf9e33cf2f4addaf5f8"
Nov 03 02:31:41.829674 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:41.829647    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"
Nov 03 02:31:41.838745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:41.838718    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"]
Nov 03 02:31:41.885265 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:41.885232    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b6c56b-openshift-default-78f969b65c-lx6zv"]
Nov 03 02:31:41.891822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:41.891797    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-openshift-ca\") pod \"98fda50c-5d63-472f-a465-faca6a3b95fb\" (UID: \"98fda50c-5d63-472f-a465-faca6a3b95fb\") "
Nov 03 02:31:41.891987 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:41.891835    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-xks-ca\") pod \"98fda50c-5d63-472f-a465-faca6a3b95fb\" (UID: \"98fda50c-5d63-472f-a465-faca6a3b95fb\") "
Nov 03 02:31:41.892061 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:41.891994    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98fda50c-5d63-472f-a465-faca6a3b95fb-kube-api-access-7bwjn\" (UniqueName: \"kubernetes.io/projected/98fda50c-5d63-472f-a465-faca6a3b95fb-kube-api-access-7bwjn\") pod \"98fda50c-5d63-472f-a465-faca6a3b95fb\" (UID: \"98fda50c-5d63-472f-a465-faca6a3b95fb\") "
Nov 03 02:31:41.892273 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:41.892249    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-openshift-ca" pod "98fda50c-5d63-472f-a465-faca6a3b95fb" (UID: "98fda50c-5d63-472f-a465-faca6a3b95fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:31:41.892575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:41.892550    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:41.894003 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:41.893977    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-xks-ca" pod "98fda50c-5d63-472f-a465-faca6a3b95fb" (UID: "98fda50c-5d63-472f-a465-faca6a3b95fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:31:41.894263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:41.894242    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fda50c-5d63-472f-a465-faca6a3b95fb-kube-api-access-7bwjn" pod "98fda50c-5d63-472f-a465-faca6a3b95fb" (UID: "98fda50c-5d63-472f-a465-faca6a3b95fb"). InnerVolumeSpecName "kube-api-access-7bwjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:31:42.015421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:42.005761    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bwjn\" (UniqueName: \"kubernetes.io/projected/98fda50c-5d63-472f-a465-faca6a3b95fb-kube-api-access-7bwjn\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:42.015421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:42.005838    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fda50c-5d63-472f-a465-faca6a3b95fb-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:31:42.670613 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:42.670574    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255" event={"ID":"98fda50c-5d63-472f-a465-faca6a3b95fb","Type":"ContainerDied","Data":"7f8c717c0843f66cc4d2c20e32edb147aac8caf2779619926e9f25586284c409"}
Nov 03 02:31:42.670613 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:42.670602    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"
Nov 03 02:31:42.671071 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:42.670631    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f8c717c0843f66cc4d2c20e32edb147aac8caf2779619926e9f25586284c409"
Nov 03 02:31:42.693524 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:31:42.693494    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" path="/var/lib/kubelet/pods/f46dc7e6-e02b-4201-baad-c3c0764a0216/volumes"
Nov 03 02:31:43.813010 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:31:43.812975    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:32:13.814775 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:32:13.814732    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:32:19.620113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:19.620030    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"]
Nov 03 02:32:19.620731 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:19.620242    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv" podUID="f3164d26-e210-4ca0-a32b-f169362543fa" containerName="maas-api" containerID="cri-o://cf40420fd0121c24f0f1883942fe8a591042aaccd290c7c60f04ff3bed5ca182" gracePeriod=30
Nov 03 02:32:19.928490 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:19.928466    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:32:19.954551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:19.954524    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3164d26-e210-4ca0-a32b-f169362543fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3164d26-e210-4ca0-a32b-f169362543fa-openshift-service-ca\") pod \"f3164d26-e210-4ca0-a32b-f169362543fa\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") "
Nov 03 02:32:19.954699 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:19.954564    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3164d26-e210-4ca0-a32b-f169362543fa-kube-api-access-t98jb\" (UniqueName: \"kubernetes.io/projected/f3164d26-e210-4ca0-a32b-f169362543fa-kube-api-access-t98jb\") pod \"f3164d26-e210-4ca0-a32b-f169362543fa\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") "
Nov 03 02:32:19.954699 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:19.954585    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-maas-api-tls\") pod \"f3164d26-e210-4ca0-a32b-f169362543fa\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") "
Nov 03 02:32:19.954876 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:19.954851    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3164d26-e210-4ca0-a32b-f169362543fa-openshift-service-ca" pod "f3164d26-e210-4ca0-a32b-f169362543fa" (UID: "f3164d26-e210-4ca0-a32b-f169362543fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:32:19.955072 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:19.955059    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3164d26-e210-4ca0-a32b-f169362543fa-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:32:19.956590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:19.956568    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-maas-api-tls" pod "f3164d26-e210-4ca0-a32b-f169362543fa" (UID: "f3164d26-e210-4ca0-a32b-f169362543fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:32:19.956695 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:19.956675    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3164d26-e210-4ca0-a32b-f169362543fa-kube-api-access-t98jb" pod "f3164d26-e210-4ca0-a32b-f169362543fa" (UID: "f3164d26-e210-4ca0-a32b-f169362543fa"). InnerVolumeSpecName "kube-api-access-t98jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:32:20.012475 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.009427    5508 generic.go:356] "Generic (PLEG): container finished" podID="f3164d26-e210-4ca0-a32b-f169362543fa" containerID="cf40420fd0121c24f0f1883942fe8a591042aaccd290c7c60f04ff3bed5ca182" exitCode=0
Nov 03 02:32:20.012475 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.009483    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv" event={"ID":"f3164d26-e210-4ca0-a32b-f169362543fa","Type":"ContainerDied","Data":"cf40420fd0121c24f0f1883942fe8a591042aaccd290c7c60f04ff3bed5ca182"}
Nov 03 02:32:20.012475 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.009532    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv" event={"ID":"f3164d26-e210-4ca0-a32b-f169362543fa","Type":"ContainerDied","Data":"2edfae43364b091ebe548d393814d00103dbafcf117104e7d694a55af97b12ab"}
Nov 03 02:32:20.012475 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.009554    5508 scope.go:122] "RemoveContainer" containerID="cf40420fd0121c24f0f1883942fe8a591042aaccd290c7c60f04ff3bed5ca182"
Nov 03 02:32:20.012475 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.009703    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"
Nov 03 02:32:20.029495 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.029466    5508 scope.go:122] "RemoveContainer" containerID="cf40420fd0121c24f0f1883942fe8a591042aaccd290c7c60f04ff3bed5ca182"
Nov 03 02:32:20.029976 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:32:20.029942    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf40420fd0121c24f0f1883942fe8a591042aaccd290c7c60f04ff3bed5ca182\": container with ID starting with cf40420fd0121c24f0f1883942fe8a591042aaccd290c7c60f04ff3bed5ca182 not found: ID does not exist" containerID="cf40420fd0121c24f0f1883942fe8a591042aaccd290c7c60f04ff3bed5ca182"
Nov 03 02:32:20.030082 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.030028    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf40420fd0121c24f0f1883942fe8a591042aaccd290c7c60f04ff3bed5ca182"} err="failed to get container status \"cf40420fd0121c24f0f1883942fe8a591042aaccd290c7c60f04ff3bed5ca182\": rpc error: code = NotFound desc = could not find container \"cf40420fd0121c24f0f1883942fe8a591042aaccd290c7c60f04ff3bed5ca182\": container with ID starting with cf40420fd0121c24f0f1883942fe8a591042aaccd290c7c60f04ff3bed5ca182 not found: ID does not exist"
Nov 03 02:32:20.060445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.056871    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-metrics-certs\") pod \"f3164d26-e210-4ca0-a32b-f169362543fa\" (UID: \"f3164d26-e210-4ca0-a32b-f169362543fa\") "
Nov 03 02:32:20.060445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.057399    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t98jb\" (UniqueName: \"kubernetes.io/projected/f3164d26-e210-4ca0-a32b-f169362543fa-kube-api-access-t98jb\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:32:20.060445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.057424    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:32:20.061816 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.061784    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-metrics-certs" pod "f3164d26-e210-4ca0-a32b-f169362543fa" (UID: "f3164d26-e210-4ca0-a32b-f169362543fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:32:20.178455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.165893    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3164d26-e210-4ca0-a32b-f169362543fa-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:32:20.425363 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.425318    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"]
Nov 03 02:32:20.431718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.431644    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e40047-8c5674889-mplcv"]
Nov 03 02:32:20.693629 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:20.693555    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3164d26-e210-4ca0-a32b-f169362543fa" path="/var/lib/kubelet/pods/f3164d26-e210-4ca0-a32b-f169362543fa/volumes"
Nov 03 02:32:26.402292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.402258    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"]
Nov 03 02:32:26.402921 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.402903    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d260f09c-3f76-4662-b580-14e75503724a" containerName="cleanup"
Nov 03 02:32:26.402975 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.402922    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d260f09c-3f76-4662-b580-14e75503724a" containerName="cleanup"
Nov 03 02:32:26.402975 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.402938    5508 cpu_manager.go:414] "containerMap: removing container" podUID="25d517d9-1e6d-4106-bf57-4cf3415707b6" containerName="cleanup"
Nov 03 02:32:26.402975 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.402945    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d517d9-1e6d-4106-bf57-4cf3415707b6" containerName="cleanup"
Nov 03 02:32:26.402975 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.402971    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerName="istio-proxy"
Nov 03 02:32:26.403106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.402977    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerName="istio-proxy"
Nov 03 02:32:26.403106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.402991    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f3164d26-e210-4ca0-a32b-f169362543fa" containerName="maas-api"
Nov 03 02:32:26.403106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.402996    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3164d26-e210-4ca0-a32b-f169362543fa" containerName="maas-api"
Nov 03 02:32:26.403106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403002    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f1e29727-c92f-4f44-8244-a2af9f83a3ac" containerName="cleanup"
Nov 03 02:32:26.403106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403006    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e29727-c92f-4f44-8244-a2af9f83a3ac" containerName="cleanup"
Nov 03 02:32:26.403106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403015    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b265ea0a-729a-4d1c-9421-9d43f4d44aba" containerName="cleanup"
Nov 03 02:32:26.403106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403019    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b265ea0a-729a-4d1c-9421-9d43f4d44aba" containerName="cleanup"
Nov 03 02:32:26.403106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403030    5508 cpu_manager.go:414] "containerMap: removing container" podUID="98fda50c-5d63-472f-a465-faca6a3b95fb" containerName="revoke-keys"
Nov 03 02:32:26.403106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403035    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98fda50c-5d63-472f-a465-faca6a3b95fb" containerName="revoke-keys"
Nov 03 02:32:26.403106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403059    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ed674d66-2642-4b30-a4b3-9873263a1cfc" containerName="cleanup"
Nov 03 02:32:26.403106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403064    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed674d66-2642-4b30-a4b3-9873263a1cfc" containerName="cleanup"
Nov 03 02:32:26.403466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403147    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerName="istio-proxy"
Nov 03 02:32:26.403466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403155    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3164d26-e210-4ca0-a32b-f169362543fa" containerName="maas-api"
Nov 03 02:32:26.403466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403164    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="98fda50c-5d63-472f-a465-faca6a3b95fb" containerName="revoke-keys"
Nov 03 02:32:26.403466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403172    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e29727-c92f-4f44-8244-a2af9f83a3ac" containerName="cleanup"
Nov 03 02:32:26.403466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403178    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d218587-4638-4bf3-b319-5cbafcbb7933" containerName="cleanup"
Nov 03 02:32:26.403466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403184    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d260f09c-3f76-4662-b580-14e75503724a" containerName="cleanup"
Nov 03 02:32:26.403466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403191    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed674d66-2642-4b30-a4b3-9873263a1cfc" containerName="cleanup"
Nov 03 02:32:26.403466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.403197    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b265ea0a-729a-4d1c-9421-9d43f4d44aba" containerName="cleanup"
Nov 03 02:32:26.407528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.407509    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.456983 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.456955    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-a3dce7-openshift-default-dockercfg-h85kn\""
Nov 03 02:32:26.460626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.460598    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-token\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.460763 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.460642    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f9ms\" (UniqueName: \"kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-kube-api-access-6f9ms\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.460827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.460772    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istiod-ca-cert\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.460891 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.460831    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-socket\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.460962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.460939    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-data\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.461024 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.461000    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-certs\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.461123 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.461109    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-podinfo\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.461181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.461166    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-envoy\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.461247 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.461231    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-credential-socket\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.516555 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.516518    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"]
Nov 03 02:32:26.689209 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.689094    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-credential-socket\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.697965 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.697932    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-token\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.698233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.698210    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f9ms\" (UniqueName: \"kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-kube-api-access-6f9ms\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.698422 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.698407    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istiod-ca-cert\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.698562 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.698548    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-socket\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.698756 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.698733    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-data\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.698900 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.698885    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-certs\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.699102 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.699088    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-podinfo\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.699222 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.699208    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-envoy\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.700706 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.694675    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-credential-socket\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.701255 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.701042    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-data\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.701255 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.701130    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-socket\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.701255 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.701128    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istiod-ca-cert\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.701255 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.701225    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-certs\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.712405 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.705586    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-envoy\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.712405 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.706053    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-podinfo\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.757981 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.757942    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-token\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:26.758170 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:26.758081    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f9ms\" (UniqueName: \"kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-kube-api-access-6f9ms\") pod \"e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") " pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:27.021707 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:27.021620    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:27.461781 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:27.461743    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"]
Nov 03 02:32:27.468857 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:32:27.468828    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0865f37_29a7_4ef7_a522_8b7ee7d8bea4.slice/crio-7a74d9e2a1ae9e54c54ec8446662964ed934435fe542ede7360ef5d8052cdd92 WatchSource:0}: Error finding container 7a74d9e2a1ae9e54c54ec8446662964ed934435fe542ede7360ef5d8052cdd92: Status 404 returned error can't find the container with id 7a74d9e2a1ae9e54c54ec8446662964ed934435fe542ede7360ef5d8052cdd92
Nov 03 02:32:27.471478 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:27.471447    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:32:27.471571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:27.471514    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:32:27.471571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:27.471542    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:32:28.076475 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:28.076425    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd" event={"ID":"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4","Type":"ContainerStarted","Data":"a9ecaa08945b651757ca39644425b279884e04eb830e09194588b57eee8a18df"}
Nov 03 02:32:28.076690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:28.076498    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd" event={"ID":"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4","Type":"ContainerStarted","Data":"7a74d9e2a1ae9e54c54ec8446662964ed934435fe542ede7360ef5d8052cdd92"}
Nov 03 02:32:28.190578 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:28.190514    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd" podStartSLOduration=2.190496052 podStartE2EDuration="2.190496052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:28.188712936 +0000 UTC m=+4427.614319404" watchObservedRunningTime="2026-11-03 02:32:28.190496052 +0000 UTC m=+4427.616102517"
Nov 03 02:32:29.022770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:29.022726    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:29.028437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:29.028412    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:29.082901 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:29.082858    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:29.083887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:29.083868    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:32:29.720259 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:29.720223    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:32:38.237418 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.237380    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"]
Nov 03 02:32:38.237994 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.237980    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerName="istio-proxy"
Nov 03 02:32:38.238083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.237994    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerName="istio-proxy"
Nov 03 02:32:38.238120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.238107    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46dc7e6-e02b-4201-baad-c3c0764a0216" containerName="istio-proxy"
Nov 03 02:32:38.242277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.242256    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"
Nov 03 02:32:38.339525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.339491    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-a3dce7\""
Nov 03 02:32:38.340433 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.339801    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-a3dce7-dockercfg-brvm7\""
Nov 03 02:32:38.361377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.355023    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"]
Nov 03 02:32:38.370670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.370248    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/328a97c3-2db1-41c2-b4e2-5d86dadcb453-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7\" (UID: \"328a97c3-2db1-41c2-b4e2-5d86dadcb453\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"
Nov 03 02:32:38.370670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.370548    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqtqh\" (UniqueName: \"kubernetes.io/projected/328a97c3-2db1-41c2-b4e2-5d86dadcb453-kube-api-access-wqtqh\") pod \"payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7\" (UID: \"328a97c3-2db1-41c2-b4e2-5d86dadcb453\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"
Nov 03 02:32:38.405373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.389523    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"]
Nov 03 02:32:38.411095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.411065    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"
Nov 03 02:32:38.463791 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.463754    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"]
Nov 03 02:32:38.473775 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.473565    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll\" (UID: \"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"
Nov 03 02:32:38.473775 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.473638    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/328a97c3-2db1-41c2-b4e2-5d86dadcb453-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7\" (UID: \"328a97c3-2db1-41c2-b4e2-5d86dadcb453\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"
Nov 03 02:32:38.474027 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.473786    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24g9p\" (UniqueName: \"kubernetes.io/projected/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-kube-api-access-24g9p\") pod \"payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll\" (UID: \"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"
Nov 03 02:32:38.474027 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.473836    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqtqh\" (UniqueName: \"kubernetes.io/projected/328a97c3-2db1-41c2-b4e2-5d86dadcb453-kube-api-access-wqtqh\") pod \"payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7\" (UID: \"328a97c3-2db1-41c2-b4e2-5d86dadcb453\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"
Nov 03 02:32:38.475477 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.475328    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/328a97c3-2db1-41c2-b4e2-5d86dadcb453-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7\" (UID: \"328a97c3-2db1-41c2-b4e2-5d86dadcb453\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"
Nov 03 02:32:38.531393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.531300    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqtqh\" (UniqueName: \"kubernetes.io/projected/328a97c3-2db1-41c2-b4e2-5d86dadcb453-kube-api-access-wqtqh\") pod \"payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7\" (UID: \"328a97c3-2db1-41c2-b4e2-5d86dadcb453\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"
Nov 03 02:32:38.556418 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.556381    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"
Nov 03 02:32:38.575079 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.575042    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24g9p\" (UniqueName: \"kubernetes.io/projected/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-kube-api-access-24g9p\") pod \"payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll\" (UID: \"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"
Nov 03 02:32:38.575245 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.575229    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll\" (UID: \"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"
Nov 03 02:32:38.576095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.576066    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll\" (UID: \"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"
Nov 03 02:32:38.637841 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.637808    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24g9p\" (UniqueName: \"kubernetes.io/projected/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-kube-api-access-24g9p\") pod \"payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll\" (UID: \"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"
Nov 03 02:32:38.762675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.762642    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"
Nov 03 02:32:38.941713 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:38.941675    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"]
Nov 03 02:32:38.948633 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:32:38.948601    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328a97c3_2db1_41c2_b4e2_5d86dadcb453.slice/crio-e1a7c73e4731cb2c021cddc71499aa6c0c03a3dfb2521a22ae24ac27346f68d1 WatchSource:0}: Error finding container e1a7c73e4731cb2c021cddc71499aa6c0c03a3dfb2521a22ae24ac27346f68d1: Status 404 returned error can't find the container with id e1a7c73e4731cb2c021cddc71499aa6c0c03a3dfb2521a22ae24ac27346f68d1
Nov 03 02:32:39.141456 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:39.141371    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"]
Nov 03 02:32:39.148479 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:32:39.148443    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b500fc0_6bf5_4eca_8cd4_677f8bc466aa.slice/crio-d0429605acf7521d24f389ec5c9ce24fcf12f88d532852074cdf7f597b5704e2 WatchSource:0}: Error finding container d0429605acf7521d24f389ec5c9ce24fcf12f88d532852074cdf7f597b5704e2: Status 404 returned error can't find the container with id d0429605acf7521d24f389ec5c9ce24fcf12f88d532852074cdf7f597b5704e2
Nov 03 02:32:39.173512 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:39.173481    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll" event={"ID":"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa","Type":"ContainerStarted","Data":"d0429605acf7521d24f389ec5c9ce24fcf12f88d532852074cdf7f597b5704e2"}
Nov 03 02:32:39.175533 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:39.175511    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7" event={"ID":"328a97c3-2db1-41c2-b4e2-5d86dadcb453","Type":"ContainerStarted","Data":"b16daf20fa77d8964a8c73ec02bff9ba729079798be50690aefb5ad1834f367c"}
Nov 03 02:32:39.175637 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:39.175554    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7" event={"ID":"328a97c3-2db1-41c2-b4e2-5d86dadcb453","Type":"ContainerStarted","Data":"e1a7c73e4731cb2c021cddc71499aa6c0c03a3dfb2521a22ae24ac27346f68d1"}
Nov 03 02:32:39.175709 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:39.175655    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"
Nov 03 02:32:39.286191 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:39.286129    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7" podStartSLOduration=1.286110592 podStartE2EDuration="1.286110592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:39.283370789 +0000 UTC m=+4438.708977266" watchObservedRunningTime="2026-11-03 02:32:39.286110592 +0000 UTC m=+4438.711717058"
Nov 03 02:32:40.186205 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:40.186162    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll" event={"ID":"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa","Type":"ContainerStarted","Data":"25102b6b81c25472afbbf41adbcf6b395c1855a966fc1eb5ff7c93d90038e5b9"}
Nov 03 02:32:40.186466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:40.186236    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"
Nov 03 02:32:40.294325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:40.294270    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll" podStartSLOduration=2.294257343 podStartE2EDuration="2.294257343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:32:40.294127225 +0000 UTC m=+4439.719733690" watchObservedRunningTime="2026-11-03 02:32:40.294257343 +0000 UTC m=+4439.719863836"
Nov 03 02:32:43.813281 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:32:43.813239    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:32:50.189984 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:50.189946    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"
Nov 03 02:32:51.198578 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:51.198550    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"
Nov 03 02:32:58.519616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:32:58.519572    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:33:13.813369 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:33:13.813309    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:33:20.510426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.510393    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"]
Nov 03 02:33:20.514531 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.514507    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.620932 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.609050    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-eec2ff70-openshift-default-dockercfg-cpkzp\""
Nov 03 02:33:20.661141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.660506    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"]
Nov 03 02:33:20.661621 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.661590    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-certs\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.661753 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.661639    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-envoy\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.661753 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.661686    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-socket\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.661890 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.661755    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wddkt\" (UniqueName: \"kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-kube-api-access-wddkt\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.661890 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.661786    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-credential-socket\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.661890 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.661813    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-data\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.661890 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.661854    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-podinfo\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.662131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.661906    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a199c50f-4280-402e-b6cd-9433e0639b7a-istiod-ca-cert\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.662131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.661958    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-token\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.803909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.802288    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-podinfo\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.803909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.802399    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a199c50f-4280-402e-b6cd-9433e0639b7a-istiod-ca-cert\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.803909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.802456    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-token\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.803909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.802588    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-certs\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.803909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.802634    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-envoy\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.803909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.802686    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-socket\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.803909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.802763    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wddkt\" (UniqueName: \"kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-kube-api-access-wddkt\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.803909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.802800    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-credential-socket\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.803909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.802832    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-data\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.803909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.803313    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-data\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.807305 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.807271    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-socket\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.807937 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.807906    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-podinfo\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.807937 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.807922    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-certs\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.808263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.808240    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-credential-socket\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.808803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.808778    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a199c50f-4280-402e-b6cd-9433e0639b7a-istiod-ca-cert\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.816582 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.816556    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-envoy\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.865227 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.865184    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-token\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:20.865440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:20.865414    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wddkt\" (UniqueName: \"kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-kube-api-access-wddkt\") pod \"e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") " pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:21.129024 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:21.128935    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:21.570430 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:21.570390    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"]
Nov 03 02:33:21.575824 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:33:21.575795    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda199c50f_4280_402e_b6cd_9433e0639b7a.slice/crio-87e757b20ac67cbf2cbd706279693e026f9707f4be88cbda99421d66ba396848 WatchSource:0}: Error finding container 87e757b20ac67cbf2cbd706279693e026f9707f4be88cbda99421d66ba396848: Status 404 returned error can't find the container with id 87e757b20ac67cbf2cbd706279693e026f9707f4be88cbda99421d66ba396848
Nov 03 02:33:21.579078 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:21.579033    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:33:21.579192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:21.579132    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:33:21.579192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:21.579171    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:33:22.503897 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:22.503854    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" event={"ID":"a199c50f-4280-402e-b6cd-9433e0639b7a","Type":"ContainerStarted","Data":"f2371369c17d6c2eef4ad0a6b48e0ff0ec4ece40a3a30f4339c1a4c8a204ccb0"}
Nov 03 02:33:22.504230 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:22.503921    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" event={"ID":"a199c50f-4280-402e-b6cd-9433e0639b7a","Type":"ContainerStarted","Data":"87e757b20ac67cbf2cbd706279693e026f9707f4be88cbda99421d66ba396848"}
Nov 03 02:33:22.666605 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:22.666507    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" podStartSLOduration=2.666491461 podStartE2EDuration="2.666491461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:22.66256224 +0000 UTC m=+4482.088168725" watchObservedRunningTime="2026-11-03 02:33:22.666491461 +0000 UTC m=+4482.092097925"
Nov 03 02:33:23.129827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:23.129788    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:23.136469 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:23.136437    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:23.511294 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:23.511265    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:23.512401 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:23.512377    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:33:28.999479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:28.999439    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"]
Nov 03 02:33:29.004227 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:29.004201    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"
Nov 03 02:33:29.020493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:29.020466    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rt4\" (UniqueName: \"kubernetes.io/projected/f18a2aed-f7a2-4326-810f-56a0b97d114c-kube-api-access-l4rt4\") pod \"payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz\" (UID: \"f18a2aed-f7a2-4326-810f-56a0b97d114c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"
Nov 03 02:33:29.020637 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:29.020610    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18a2aed-f7a2-4326-810f-56a0b97d114c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz\" (UID: \"f18a2aed-f7a2-4326-810f-56a0b97d114c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"
Nov 03 02:33:29.100943 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:29.100900    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-eec2ff70-dockercfg-9r26r\""
Nov 03 02:33:29.101194 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:29.101156    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-eec2ff70\""
Nov 03 02:33:29.150288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:29.150245    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"]
Nov 03 02:33:29.155532 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:29.155243    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18a2aed-f7a2-4326-810f-56a0b97d114c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz\" (UID: \"f18a2aed-f7a2-4326-810f-56a0b97d114c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"
Nov 03 02:33:29.159732 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:29.159702    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rt4\" (UniqueName: \"kubernetes.io/projected/f18a2aed-f7a2-4326-810f-56a0b97d114c-kube-api-access-l4rt4\") pod \"payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz\" (UID: \"f18a2aed-f7a2-4326-810f-56a0b97d114c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"
Nov 03 02:33:29.160416 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:29.157540    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18a2aed-f7a2-4326-810f-56a0b97d114c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz\" (UID: \"f18a2aed-f7a2-4326-810f-56a0b97d114c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"
Nov 03 02:33:29.225504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:29.222302    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rt4\" (UniqueName: \"kubernetes.io/projected/f18a2aed-f7a2-4326-810f-56a0b97d114c-kube-api-access-l4rt4\") pod \"payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz\" (UID: \"f18a2aed-f7a2-4326-810f-56a0b97d114c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"
Nov 03 02:33:29.323922 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:29.323834    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"
Nov 03 02:33:29.702413 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:29.702377    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"]
Nov 03 02:33:29.710255 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:33:29.710226    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18a2aed_f7a2_4326_810f_56a0b97d114c.slice/crio-0e71f973863ea3b24fbf999e3f527fa4f45b4ceed56d2fa3cf94214ea6d3ffe8 WatchSource:0}: Error finding container 0e71f973863ea3b24fbf999e3f527fa4f45b4ceed56d2fa3cf94214ea6d3ffe8: Status 404 returned error can't find the container with id 0e71f973863ea3b24fbf999e3f527fa4f45b4ceed56d2fa3cf94214ea6d3ffe8
Nov 03 02:33:30.596299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:30.596256    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz" event={"ID":"f18a2aed-f7a2-4326-810f-56a0b97d114c","Type":"ContainerStarted","Data":"f808cc119c932166db155f8d5d0f0120e50bcbff4cb27bc4cc65b5651caa7ab7"}
Nov 03 02:33:30.596879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:30.596393    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"
Nov 03 02:33:30.596879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:30.596419    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz" event={"ID":"f18a2aed-f7a2-4326-810f-56a0b97d114c","Type":"ContainerStarted","Data":"0e71f973863ea3b24fbf999e3f527fa4f45b4ceed56d2fa3cf94214ea6d3ffe8"}
Nov 03 02:33:30.707501 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:30.707449    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz" podStartSLOduration=2.707433857 podStartE2EDuration="2.707433857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:30.704195975 +0000 UTC m=+4490.129802460" watchObservedRunningTime="2026-11-03 02:33:30.707433857 +0000 UTC m=+4490.133040326"
Nov 03 02:33:31.935151 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:31.935114    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"]
Nov 03 02:33:32.035130 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:32.035100    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"]
Nov 03 02:33:36.936134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:36.936090    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h" podUID="0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7" containerName="payload-processing" containerID="cri-o://a327cb27e682415180bda9b716590bef3d10aa1afd12ffd6a66558b4f1e52f44" gracePeriod=25
Nov 03 02:33:37.035829 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:37.035785    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2" podUID="d0cfd5a0-8059-492b-b503-aa3644196ec4" containerName="payload-pre-processing" containerID="cri-o://4111aec62f8726db30af8cdb95818a5c61cd88be477689c8303af1de2d905d5b" gracePeriod=25
Nov 03 02:33:38.904240 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:38.904201    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"]
Nov 03 02:33:38.909801 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:38.909779    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:38.958972 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:38.958938    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-a3dce7-serving-cert\""
Nov 03 02:33:39.008671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.008631    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"]
Nov 03 02:33:39.047498 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.047462    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d393ed80-639b-4ae5-8c63-207fda9ec10f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:39.047691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.047513    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:39.047691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.047667    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:39.047838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.047816    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfdl4\" (UniqueName: \"kubernetes.io/projected/d393ed80-639b-4ae5-8c63-207fda9ec10f-kube-api-access-hfdl4\") pod \"maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:39.149704 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.149578    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d393ed80-639b-4ae5-8c63-207fda9ec10f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:39.149704 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.149641    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:39.149704 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.149689    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:39.150024 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.149915    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfdl4\" (UniqueName: \"kubernetes.io/projected/d393ed80-639b-4ae5-8c63-207fda9ec10f-kube-api-access-hfdl4\") pod \"maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:39.153389 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.151019    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d393ed80-639b-4ae5-8c63-207fda9ec10f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:39.153389 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:33:39.151135    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-serving-cert: secret "maas-api-e2e-shared-a-w0-a3dce7-serving-cert" not found
Nov 03 02:33:39.153389 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:33:39.151206    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-maas-api-tls podName:d393ed80-639b-4ae5-8c63-207fda9ec10f nodeName:}" failed. No retries permitted until 2026-11-03 02:33:39.651184433 +0000 UTC m=+4499.076790877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-maas-api-tls") pod "maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679" (UID: "d393ed80-639b-4ae5-8c63-207fda9ec10f") : secret "maas-api-e2e-shared-a-w0-a3dce7-serving-cert" not found
Nov 03 02:33:39.185295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.185186    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:39.211120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.211088    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfdl4\" (UniqueName: \"kubernetes.io/projected/d393ed80-639b-4ae5-8c63-207fda9ec10f-kube-api-access-hfdl4\") pod \"maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:39.657728 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.657681    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:39.660928 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.660887    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:39.824510 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:39.824472    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:40.203470 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:40.203431    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"]
Nov 03 02:33:40.211565 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:33:40.211540    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd393ed80_639b_4ae5_8c63_207fda9ec10f.slice/crio-579ba95a58911f13ce80c82676416449119ba304dbdfa841719871ea7ffb9325 WatchSource:0}: Error finding container 579ba95a58911f13ce80c82676416449119ba304dbdfa841719871ea7ffb9325: Status 404 returned error can't find the container with id 579ba95a58911f13ce80c82676416449119ba304dbdfa841719871ea7ffb9325
Nov 03 02:33:40.213016 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:40.212998    5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:33:40.673936 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:40.673898    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679" event={"ID":"d393ed80-639b-4ae5-8c63-207fda9ec10f","Type":"ContainerStarted","Data":"579ba95a58911f13ce80c82676416449119ba304dbdfa841719871ea7ffb9325"}
Nov 03 02:33:41.570301 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:41.570211    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:33:41.607415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:41.607390    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"
Nov 03 02:33:41.683899 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:41.683867    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679" event={"ID":"d393ed80-639b-4ae5-8c63-207fda9ec10f","Type":"ContainerStarted","Data":"5fc04583748bd15a0d5d245c31ea32a27b9ced95b5b4aec73873c167dfaacfa5"}
Nov 03 02:33:41.684443 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:41.684420    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:41.693827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:41.693806    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:33:41.829920 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:41.829804    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679" podStartSLOduration=2.851263035 podStartE2EDuration="3.829784607s" totalImagesPullingTime="978.521572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:33:41.825627991 +0000 UTC m=+4501.251234475" watchObservedRunningTime="2026-11-03 02:33:41.829784607 +0000 UTC m=+4501.255391073"
Nov 03 02:33:43.831188 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:33:43.831154    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:33:45.850318 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:45.850224    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h" podUID="0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:33:45.850831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:45.850323    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2" podUID="d0cfd5a0-8059-492b-b503-aa3644196ec4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:33:48.781651 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:48.781623    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:33:54.512141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:54.512104    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"]
Nov 03 02:33:54.512697 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:54.512469    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerName="istio-proxy" containerID="cri-o://f743c1dee3a59b38f511f65a798b1f62e9f00306512cfebbb18cc8c4f197b335" gracePeriod=30
Nov 03 02:33:55.849618 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:55.849582    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2" podUID="d0cfd5a0-8059-492b-b503-aa3644196ec4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:33:55.850116 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:55.849653    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h" podUID="0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:33:57.242385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.240232    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"
Nov 03 02:33:57.305386 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.305317    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"
Nov 03 02:33:57.344128 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.344096    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-kube-api-access-479zv\" (UniqueName: \"kubernetes.io/projected/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-kube-api-access-479zv\") pod \"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7\" (UID: \"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7\") "
Nov 03 02:33:57.344314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.344150    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-plugins-config-volume\") pod \"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7\" (UID: \"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7\") "
Nov 03 02:33:57.344578 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.344548    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-plugins-config-volume" pod "0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7" (UID: "0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:33:57.344719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.344668    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:57.346252 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.346226    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-kube-api-access-479zv" pod "0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7" (UID: "0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7"). InnerVolumeSpecName "kube-api-access-479zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:33:57.445744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.445708    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0cfd5a0-8059-492b-b503-aa3644196ec4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0cfd5a0-8059-492b-b503-aa3644196ec4-plugins-config-volume\") pod \"d0cfd5a0-8059-492b-b503-aa3644196ec4\" (UID: \"d0cfd5a0-8059-492b-b503-aa3644196ec4\") "
Nov 03 02:33:57.445943 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.445884    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0cfd5a0-8059-492b-b503-aa3644196ec4-kube-api-access-cwrqk\" (UniqueName: \"kubernetes.io/projected/d0cfd5a0-8059-492b-b503-aa3644196ec4-kube-api-access-cwrqk\") pod \"d0cfd5a0-8059-492b-b503-aa3644196ec4\" (UID: \"d0cfd5a0-8059-492b-b503-aa3644196ec4\") "
Nov 03 02:33:57.446131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.446105    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0cfd5a0-8059-492b-b503-aa3644196ec4-plugins-config-volume" pod "d0cfd5a0-8059-492b-b503-aa3644196ec4" (UID: "d0cfd5a0-8059-492b-b503-aa3644196ec4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:33:57.446359 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.446325    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0cfd5a0-8059-492b-b503-aa3644196ec4-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:57.446501 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.446364    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-479zv\" (UniqueName: \"kubernetes.io/projected/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7-kube-api-access-479zv\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:57.448042 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.448019    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0cfd5a0-8059-492b-b503-aa3644196ec4-kube-api-access-cwrqk" pod "d0cfd5a0-8059-492b-b503-aa3644196ec4" (UID: "d0cfd5a0-8059-492b-b503-aa3644196ec4"). InnerVolumeSpecName "kube-api-access-cwrqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:33:57.547811 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.547727    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwrqk\" (UniqueName: \"kubernetes.io/projected/d0cfd5a0-8059-492b-b503-aa3644196ec4-kube-api-access-cwrqk\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:57.877784 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.877690    5508 generic.go:356] "Generic (PLEG): container finished" podID="d0cfd5a0-8059-492b-b503-aa3644196ec4" containerID="4111aec62f8726db30af8cdb95818a5c61cd88be477689c8303af1de2d905d5b" exitCode=0
Nov 03 02:33:57.877784 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.877745    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"
Nov 03 02:33:57.877784 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.877770    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2" event={"ID":"d0cfd5a0-8059-492b-b503-aa3644196ec4","Type":"ContainerDied","Data":"4111aec62f8726db30af8cdb95818a5c61cd88be477689c8303af1de2d905d5b"}
Nov 03 02:33:57.878084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.877825    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2" event={"ID":"d0cfd5a0-8059-492b-b503-aa3644196ec4","Type":"ContainerDied","Data":"b6cb3d31fa301dd678f3acdbff8d5e58b1874cab9f49f866318b144a3450c5c9"}
Nov 03 02:33:57.878084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.877855    5508 scope.go:122] "RemoveContainer" containerID="4111aec62f8726db30af8cdb95818a5c61cd88be477689c8303af1de2d905d5b"
Nov 03 02:33:57.879611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.879585    5508 generic.go:356] "Generic (PLEG): container finished" podID="0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7" containerID="a327cb27e682415180bda9b716590bef3d10aa1afd12ffd6a66558b4f1e52f44" exitCode=0
Nov 03 02:33:57.879754 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.879624    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h" event={"ID":"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7","Type":"ContainerDied","Data":"a327cb27e682415180bda9b716590bef3d10aa1afd12ffd6a66558b4f1e52f44"}
Nov 03 02:33:57.879754 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.879642    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"
Nov 03 02:33:57.879754 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.879675    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h" event={"ID":"0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7","Type":"ContainerDied","Data":"5002592bde4b324ca849730fc7e43af2062eae0bcdaaba9baeee5dcdc1dde269"}
Nov 03 02:33:57.890222 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.890204    5508 scope.go:122] "RemoveContainer" containerID="4111aec62f8726db30af8cdb95818a5c61cd88be477689c8303af1de2d905d5b"
Nov 03 02:33:57.890616 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:33:57.890545    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4111aec62f8726db30af8cdb95818a5c61cd88be477689c8303af1de2d905d5b\": container with ID starting with 4111aec62f8726db30af8cdb95818a5c61cd88be477689c8303af1de2d905d5b not found: ID does not exist" containerID="4111aec62f8726db30af8cdb95818a5c61cd88be477689c8303af1de2d905d5b"
Nov 03 02:33:57.890616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.890577    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4111aec62f8726db30af8cdb95818a5c61cd88be477689c8303af1de2d905d5b"} err="failed to get container status \"4111aec62f8726db30af8cdb95818a5c61cd88be477689c8303af1de2d905d5b\": rpc error: code = NotFound desc = could not find container \"4111aec62f8726db30af8cdb95818a5c61cd88be477689c8303af1de2d905d5b\": container with ID starting with 4111aec62f8726db30af8cdb95818a5c61cd88be477689c8303af1de2d905d5b not found: ID does not exist"
Nov 03 02:33:57.890616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.890600    5508 scope.go:122] "RemoveContainer" containerID="a327cb27e682415180bda9b716590bef3d10aa1afd12ffd6a66558b4f1e52f44"
Nov 03 02:33:57.900081 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.900067    5508 scope.go:122] "RemoveContainer" containerID="a327cb27e682415180bda9b716590bef3d10aa1afd12ffd6a66558b4f1e52f44"
Nov 03 02:33:57.900381 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:33:57.900334    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a327cb27e682415180bda9b716590bef3d10aa1afd12ffd6a66558b4f1e52f44\": container with ID starting with a327cb27e682415180bda9b716590bef3d10aa1afd12ffd6a66558b4f1e52f44 not found: ID does not exist" containerID="a327cb27e682415180bda9b716590bef3d10aa1afd12ffd6a66558b4f1e52f44"
Nov 03 02:33:57.900496 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:57.900384    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a327cb27e682415180bda9b716590bef3d10aa1afd12ffd6a66558b4f1e52f44"} err="failed to get container status \"a327cb27e682415180bda9b716590bef3d10aa1afd12ffd6a66558b4f1e52f44\": rpc error: code = NotFound desc = could not find container \"a327cb27e682415180bda9b716590bef3d10aa1afd12ffd6a66558b4f1e52f44\": container with ID starting with a327cb27e682415180bda9b716590bef3d10aa1afd12ffd6a66558b4f1e52f44 not found: ID does not exist"
Nov 03 02:33:58.032768 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:58.032733    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"]
Nov 03 02:33:58.078890 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:58.078859    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a54cab-5d8ddd9f5-bblt2"]
Nov 03 02:33:58.139259 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:58.139132    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"]
Nov 03 02:33:58.142660 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:58.142635    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a54cab-7bd48bb9c5-jhd2h"]
Nov 03 02:33:58.692945 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:58.692910    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7" path="/var/lib/kubelet/pods/0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7/volumes"
Nov 03 02:33:58.693448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:58.693430    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0cfd5a0-8059-492b-b503-aa3644196ec4" path="/var/lib/kubelet/pods/d0cfd5a0-8059-492b-b503-aa3644196ec4/volumes"
Nov 03 02:33:59.833929 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.833902    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:33:59.904852 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.904811    5508 generic.go:356] "Generic (PLEG): container finished" podID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerID="f743c1dee3a59b38f511f65a798b1f62e9f00306512cfebbb18cc8c4f197b335" exitCode=0
Nov 03 02:33:59.905064 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.904869    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"
Nov 03 02:33:59.905064 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.904896    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" event={"ID":"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053","Type":"ContainerDied","Data":"f743c1dee3a59b38f511f65a798b1f62e9f00306512cfebbb18cc8c4f197b335"}
Nov 03 02:33:59.905064 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.904990    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2" event={"ID":"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053","Type":"ContainerDied","Data":"a99d597dabc42e3797ae1c092bad2731f0c622911aa579f991af3a4b5023db9d"}
Nov 03 02:33:59.905064 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.905005    5508 scope.go:122] "RemoveContainer" containerID="f743c1dee3a59b38f511f65a798b1f62e9f00306512cfebbb18cc8c4f197b335"
Nov 03 02:33:59.915426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.915404    5508 scope.go:122] "RemoveContainer" containerID="6f9451f1166b46677e76f7963db18b3894b036b3f8c8020265697d8937d4f7bf"
Nov 03 02:33:59.924522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.924504    5508 scope.go:122] "RemoveContainer" containerID="f743c1dee3a59b38f511f65a798b1f62e9f00306512cfebbb18cc8c4f197b335"
Nov 03 02:33:59.924793 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:33:59.924775    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f743c1dee3a59b38f511f65a798b1f62e9f00306512cfebbb18cc8c4f197b335\": container with ID starting with f743c1dee3a59b38f511f65a798b1f62e9f00306512cfebbb18cc8c4f197b335 not found: ID does not exist" containerID="f743c1dee3a59b38f511f65a798b1f62e9f00306512cfebbb18cc8c4f197b335"
Nov 03 02:33:59.924858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.924800    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f743c1dee3a59b38f511f65a798b1f62e9f00306512cfebbb18cc8c4f197b335"} err="failed to get container status \"f743c1dee3a59b38f511f65a798b1f62e9f00306512cfebbb18cc8c4f197b335\": rpc error: code = NotFound desc = could not find container \"f743c1dee3a59b38f511f65a798b1f62e9f00306512cfebbb18cc8c4f197b335\": container with ID starting with f743c1dee3a59b38f511f65a798b1f62e9f00306512cfebbb18cc8c4f197b335 not found: ID does not exist"
Nov 03 02:33:59.924858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.924817    5508 scope.go:122] "RemoveContainer" containerID="6f9451f1166b46677e76f7963db18b3894b036b3f8c8020265697d8937d4f7bf"
Nov 03 02:33:59.925038 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:33:59.925021    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f9451f1166b46677e76f7963db18b3894b036b3f8c8020265697d8937d4f7bf\": container with ID starting with 6f9451f1166b46677e76f7963db18b3894b036b3f8c8020265697d8937d4f7bf not found: ID does not exist" containerID="6f9451f1166b46677e76f7963db18b3894b036b3f8c8020265697d8937d4f7bf"
Nov 03 02:33:59.925103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.925040    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f9451f1166b46677e76f7963db18b3894b036b3f8c8020265697d8937d4f7bf"} err="failed to get container status \"6f9451f1166b46677e76f7963db18b3894b036b3f8c8020265697d8937d4f7bf\": rpc error: code = NotFound desc = could not find container \"6f9451f1166b46677e76f7963db18b3894b036b3f8c8020265697d8937d4f7bf\": container with ID starting with 6f9451f1166b46677e76f7963db18b3894b036b3f8c8020265697d8937d4f7bf not found: ID does not exist"
Nov 03 02:33:59.990214 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.990186    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-socket\") pod \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") "
Nov 03 02:33:59.990467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.990224    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-certs\") pod \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") "
Nov 03 02:33:59.990467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.990320    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-data\") pod \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") "
Nov 03 02:33:59.990467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.990397    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-podinfo\") pod \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") "
Nov 03 02:33:59.990467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.990432    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-envoy\") pod \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") "
Nov 03 02:33:59.990718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.990496    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-socket" pod "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" (UID: "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:59.990718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.990520    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-token\" (UniqueName: \"kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-token\") pod \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") "
Nov 03 02:33:59.990718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.990568    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istiod-ca-cert\") pod \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") "
Nov 03 02:33:59.990718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.990603    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-data" pod "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" (UID: "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:59.990718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.990613    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-certs" pod "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" (UID: "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:59.990718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.990645    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-credential-socket\") pod \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") "
Nov 03 02:33:59.990718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.990699    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-kube-api-access-rcx4m\" (UniqueName: \"kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-kube-api-access-rcx4m\") pod \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\" (UID: \"2ed3911d-2e2f-463d-93aa-cd8ef0bf9053\") "
Nov 03 02:33:59.991112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.991033    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istiod-ca-cert" pod "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" (UID: "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:33:59.991326 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.991301    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:59.991443 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.991332    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:59.991443 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.991373    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:59.991443 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.991389    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:33:59.991667 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.991652    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-credential-socket" pod "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" (UID: "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:33:59.992810 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.992785    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-token" pod "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" (UID: "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:33:59.992968 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.992891    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-kube-api-access-rcx4m" pod "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" (UID: "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053"). InnerVolumeSpecName "kube-api-access-rcx4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:33:59.993144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.993112    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-podinfo" pod "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" (UID: "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:33:59.993362 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:33:59.993205    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-envoy" pod "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" (UID: "2ed3911d-2e2f-463d-93aa-cd8ef0bf9053"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:34:00.092485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:00.092403    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:34:00.092485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:00.092431    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:34:00.092485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:00.092442    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcx4m\" (UniqueName: \"kubernetes.io/projected/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-kube-api-access-rcx4m\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:34:00.092485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:00.092453    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:34:00.092485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:00.092461    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:34:00.322423 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:00.322387    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"]
Nov 03 02:34:00.325534 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:00.325499    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a54cab-openshift-default-7d465dfd85-w59r2"]
Nov 03 02:34:00.693151 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:00.693121    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" path="/var/lib/kubelet/pods/2ed3911d-2e2f-463d-93aa-cd8ef0bf9053/volumes"
Nov 03 02:34:00.707021 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:00.706988    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"]
Nov 03 02:34:00.809970 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:00.809930    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"]
Nov 03 02:34:05.707827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:05.707757    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb" podUID="911e2aa1-8c98-428d-91fc-988b3131d2de" containerName="payload-processing" containerID="cri-o://f27879941a0fdeaa028401a48408702c0955045f988c0f2e6cb037605555dfb1" gracePeriod=25
Nov 03 02:34:05.810416 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:05.810319    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr" podUID="7aaaad46-1501-477a-b1e4-7cd7d758e92f" containerName="payload-pre-processing" containerID="cri-o://f04faaf51f2fa73a00aa18355df45012404e558b4da28a65a29810540c3f1eff" gracePeriod=25
Nov 03 02:34:08.838548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:08.838509    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:34:11.802884 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:11.802847    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb" podUID="911e2aa1-8c98-428d-91fc-988b3131d2de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:34:11.803305 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:11.802853    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr" podUID="7aaaad46-1501-477a-b1e4-7cd7d758e92f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:34:13.812828 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:34:13.812782    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:34:20.069436 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:20.069302    5508 generic.go:356] "Generic (PLEG): container finished" podID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerID="a9ecaa08945b651757ca39644425b279884e04eb830e09194588b57eee8a18df" exitCode=0
Nov 03 02:34:20.069436 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:20.069394    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd" event={"ID":"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4","Type":"ContainerDied","Data":"a9ecaa08945b651757ca39644425b279884e04eb830e09194588b57eee8a18df"}
Nov 03 02:34:20.070052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:20.069835    5508 scope.go:122] "RemoveContainer" containerID="a9ecaa08945b651757ca39644425b279884e04eb830e09194588b57eee8a18df"
Nov 03 02:34:20.071618 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:20.071585    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:34:20.071761 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:20.071677    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:34:20.071761 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:20.071730    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:34:21.084319 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:21.084282    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd" event={"ID":"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4","Type":"ContainerStarted","Data":"411f20fa3c295e7c4719d94c954d8bcd4dfdc484ddd931c29bd4a3b2415066f0"}
Nov 03 02:34:21.803527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:21.803480    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb" podUID="911e2aa1-8c98-428d-91fc-988b3131d2de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:34:21.803731 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:21.803480    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr" podUID="7aaaad46-1501-477a-b1e4-7cd7d758e92f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:34:22.022315 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:22.022277    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:34:22.022597 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:22.022570    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body=
Nov 03 02:34:22.022664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:22.022631    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd" podUID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused"
Nov 03 02:34:23.022829 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:23.022796    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body=
Nov 03 02:34:23.023233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:23.022851    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd" podUID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused"
Nov 03 02:34:24.022329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:24.022293    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body=
Nov 03 02:34:24.022539 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:24.022368    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd" podUID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused"
Nov 03 02:34:25.022036 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:25.021996    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body=
Nov 03 02:34:25.022507 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:25.022059    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd" podUID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused"
Nov 03 02:34:26.023659 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.023010    5508 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused" start-of-body=
Nov 03 02:34:26.023659 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.023074    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd" podUID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.86:15021/healthz/ready\": dial tcp 10.133.1.86:15021: connect: connection refused"
Nov 03 02:34:26.094422 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.094381    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"
Nov 03 02:34:26.122185 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.121989    5508 generic.go:356] "Generic (PLEG): container finished" podID="7aaaad46-1501-477a-b1e4-7cd7d758e92f" containerID="f04faaf51f2fa73a00aa18355df45012404e558b4da28a65a29810540c3f1eff" exitCode=0
Nov 03 02:34:26.122185 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.121995    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr" event={"ID":"7aaaad46-1501-477a-b1e4-7cd7d758e92f","Type":"ContainerDied","Data":"f04faaf51f2fa73a00aa18355df45012404e558b4da28a65a29810540c3f1eff"}
Nov 03 02:34:26.125615 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.125581    5508 generic.go:356] "Generic (PLEG): container finished" podID="911e2aa1-8c98-428d-91fc-988b3131d2de" containerID="f27879941a0fdeaa028401a48408702c0955045f988c0f2e6cb037605555dfb1" exitCode=0
Nov 03 02:34:26.125802 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.125762    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"
Nov 03 02:34:26.125802 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.125771    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb" event={"ID":"911e2aa1-8c98-428d-91fc-988b3131d2de","Type":"ContainerDied","Data":"f27879941a0fdeaa028401a48408702c0955045f988c0f2e6cb037605555dfb1"}
Nov 03 02:34:26.125912 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.125827    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb" event={"ID":"911e2aa1-8c98-428d-91fc-988b3131d2de","Type":"ContainerDied","Data":"2c183cc2530e8a23aa06f8022ebc03e7baba6bb44b89f99311a1e450166b5159"}
Nov 03 02:34:26.126023 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.126004    5508 scope.go:122] "RemoveContainer" containerID="f27879941a0fdeaa028401a48408702c0955045f988c0f2e6cb037605555dfb1"
Nov 03 02:34:26.172736 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.160521    5508 scope.go:122] "RemoveContainer" containerID="f27879941a0fdeaa028401a48408702c0955045f988c0f2e6cb037605555dfb1"
Nov 03 02:34:26.174635 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:34:26.174551    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27879941a0fdeaa028401a48408702c0955045f988c0f2e6cb037605555dfb1\": container with ID starting with f27879941a0fdeaa028401a48408702c0955045f988c0f2e6cb037605555dfb1 not found: ID does not exist" containerID="f27879941a0fdeaa028401a48408702c0955045f988c0f2e6cb037605555dfb1"
Nov 03 02:34:26.174635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.174603    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27879941a0fdeaa028401a48408702c0955045f988c0f2e6cb037605555dfb1"} err="failed to get container status \"f27879941a0fdeaa028401a48408702c0955045f988c0f2e6cb037605555dfb1\": rpc error: code = NotFound desc = could not find container \"f27879941a0fdeaa028401a48408702c0955045f988c0f2e6cb037605555dfb1\": container with ID starting with f27879941a0fdeaa028401a48408702c0955045f988c0f2e6cb037605555dfb1 not found: ID does not exist"
Nov 03 02:34:26.180108 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.180082    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"
Nov 03 02:34:26.218911 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.218856    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/911e2aa1-8c98-428d-91fc-988b3131d2de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911e2aa1-8c98-428d-91fc-988b3131d2de-plugins-config-volume\") pod \"911e2aa1-8c98-428d-91fc-988b3131d2de\" (UID: \"911e2aa1-8c98-428d-91fc-988b3131d2de\") "
Nov 03 02:34:26.219115 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.219049    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911e2aa1-8c98-428d-91fc-988b3131d2de-kube-api-access-fhxxd\" (UniqueName: \"kubernetes.io/projected/911e2aa1-8c98-428d-91fc-988b3131d2de-kube-api-access-fhxxd\") pod \"911e2aa1-8c98-428d-91fc-988b3131d2de\" (UID: \"911e2aa1-8c98-428d-91fc-988b3131d2de\") "
Nov 03 02:34:26.219705 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.219676    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911e2aa1-8c98-428d-91fc-988b3131d2de-plugins-config-volume" pod "911e2aa1-8c98-428d-91fc-988b3131d2de" (UID: "911e2aa1-8c98-428d-91fc-988b3131d2de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:34:26.219832 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.219804    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/911e2aa1-8c98-428d-91fc-988b3131d2de-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:34:26.221730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.221702    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911e2aa1-8c98-428d-91fc-988b3131d2de-kube-api-access-fhxxd" pod "911e2aa1-8c98-428d-91fc-988b3131d2de" (UID: "911e2aa1-8c98-428d-91fc-988b3131d2de"). InnerVolumeSpecName "kube-api-access-fhxxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:34:26.321012 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.320917    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aaaad46-1501-477a-b1e4-7cd7d758e92f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaaad46-1501-477a-b1e4-7cd7d758e92f-plugins-config-volume\") pod \"7aaaad46-1501-477a-b1e4-7cd7d758e92f\" (UID: \"7aaaad46-1501-477a-b1e4-7cd7d758e92f\") "
Nov 03 02:34:26.321208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.321050    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aaaad46-1501-477a-b1e4-7cd7d758e92f-kube-api-access-r2ljl\" (UniqueName: \"kubernetes.io/projected/7aaaad46-1501-477a-b1e4-7cd7d758e92f-kube-api-access-r2ljl\") pod \"7aaaad46-1501-477a-b1e4-7cd7d758e92f\" (UID: \"7aaaad46-1501-477a-b1e4-7cd7d758e92f\") "
Nov 03 02:34:26.321412 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.321384    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aaaad46-1501-477a-b1e4-7cd7d758e92f-plugins-config-volume" pod "7aaaad46-1501-477a-b1e4-7cd7d758e92f" (UID: "7aaaad46-1501-477a-b1e4-7cd7d758e92f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:34:26.321529 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.321508    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7aaaad46-1501-477a-b1e4-7cd7d758e92f-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:34:26.321585 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.321536    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhxxd\" (UniqueName: \"kubernetes.io/projected/911e2aa1-8c98-428d-91fc-988b3131d2de-kube-api-access-fhxxd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:34:26.323424 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.323396    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aaaad46-1501-477a-b1e4-7cd7d758e92f-kube-api-access-r2ljl" pod "7aaaad46-1501-477a-b1e4-7cd7d758e92f" (UID: "7aaaad46-1501-477a-b1e4-7cd7d758e92f"). InnerVolumeSpecName "kube-api-access-r2ljl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:34:26.423238 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.423174    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2ljl\" (UniqueName: \"kubernetes.io/projected/7aaaad46-1501-477a-b1e4-7cd7d758e92f-kube-api-access-r2ljl\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:34:26.540563 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.540523    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"]
Nov 03 02:34:26.545247 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.545216    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1c9b4e-8d885787c-zj7kb"]
Nov 03 02:34:26.694929 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:26.694888    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911e2aa1-8c98-428d-91fc-988b3131d2de" path="/var/lib/kubelet/pods/911e2aa1-8c98-428d-91fc-988b3131d2de/volumes"
Nov 03 02:34:27.026893 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:27.026814    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:34:27.027306 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:27.027252    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:34:27.028167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:27.028148    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:34:27.135259 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:27.135227    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"
Nov 03 02:34:27.135259 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:27.135248    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr" event={"ID":"7aaaad46-1501-477a-b1e4-7cd7d758e92f","Type":"ContainerDied","Data":"5515dbbca6daec7e87b1d42cfa67215f1398bd19b08e2940ccfb46da032226b8"}
Nov 03 02:34:27.135594 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:27.135315    5508 scope.go:122] "RemoveContainer" containerID="f04faaf51f2fa73a00aa18355df45012404e558b4da28a65a29810540c3f1eff"
Nov 03 02:34:27.268765 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:27.268725    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"]
Nov 03 02:34:27.273168 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:27.273134    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1c9b4e-d679cb6c9-xrszr"]
Nov 03 02:34:28.694661 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:28.694621    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aaaad46-1501-477a-b1e4-7cd7d758e92f" path="/var/lib/kubelet/pods/7aaaad46-1501-477a-b1e4-7cd7d758e92f/volumes"
Nov 03 02:34:33.172571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.172526    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"]
Nov 03 02:34:33.173434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173413    5508 cpu_manager.go:414] "containerMap: removing container" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerName="istio-proxy"
Nov 03 02:34:33.173491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173436    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerName="istio-proxy"
Nov 03 02:34:33.173491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173452    5508 cpu_manager.go:414] "containerMap: removing container" podUID="911e2aa1-8c98-428d-91fc-988b3131d2de" containerName="payload-processing"
Nov 03 02:34:33.173491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173459    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911e2aa1-8c98-428d-91fc-988b3131d2de" containerName="payload-processing"
Nov 03 02:34:33.173601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173493    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d0cfd5a0-8059-492b-b503-aa3644196ec4" containerName="payload-pre-processing"
Nov 03 02:34:33.173601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173502    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0cfd5a0-8059-492b-b503-aa3644196ec4" containerName="payload-pre-processing"
Nov 03 02:34:33.173601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173521    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7aaaad46-1501-477a-b1e4-7cd7d758e92f" containerName="payload-pre-processing"
Nov 03 02:34:33.173601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173529    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aaaad46-1501-477a-b1e4-7cd7d758e92f" containerName="payload-pre-processing"
Nov 03 02:34:33.173601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173560    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7" containerName="payload-processing"
Nov 03 02:34:33.173601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173567    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7" containerName="payload-processing"
Nov 03 02:34:33.173793 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173686    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerName="istio-proxy"
Nov 03 02:34:33.173793 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173697    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="911e2aa1-8c98-428d-91fc-988b3131d2de" containerName="payload-processing"
Nov 03 02:34:33.173793 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173712    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0cfd5a0-8059-492b-b503-aa3644196ec4" containerName="payload-pre-processing"
Nov 03 02:34:33.173793 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173724    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aaaad46-1501-477a-b1e4-7cd7d758e92f" containerName="payload-pre-processing"
Nov 03 02:34:33.173793 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.173738    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1b2ae6-4a5f-4d90-90a6-24a434c0f5d7" containerName="payload-processing"
Nov 03 02:34:33.179200 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.179176    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:33.217226 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.217171    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-maas-api-tls\") pod \"maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:33.217468 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.217384    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-metrics-certs\") pod \"maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:33.217468 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.217426    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50fad98-b746-40f6-853e-887f2da38314-openshift-service-ca\") pod \"maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:33.217605 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.217489    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxnh7\" (UniqueName: \"kubernetes.io/projected/c50fad98-b746-40f6-853e-887f2da38314-kube-api-access-vxnh7\") pod \"maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:33.229880 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.229843    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-eec2ff70-serving-cert\""
Nov 03 02:34:33.281887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.281835    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"]
Nov 03 02:34:33.318542 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.318504    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-metrics-certs\") pod \"maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:33.318746 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.318561    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50fad98-b746-40f6-853e-887f2da38314-openshift-service-ca\") pod \"maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:33.318746 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.318605    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxnh7\" (UniqueName: \"kubernetes.io/projected/c50fad98-b746-40f6-853e-887f2da38314-kube-api-access-vxnh7\") pod \"maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:33.318746 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.318712    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-maas-api-tls\") pod \"maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:33.318945 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:34:33.318848    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-serving-cert: secret "maas-api-e2e-mt-eec2ff70-serving-cert" not found
Nov 03 02:34:33.318945 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:34:33.318912    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-maas-api-tls podName:c50fad98-b746-40f6-853e-887f2da38314 nodeName:}" failed. No retries permitted until 2026-11-03 02:34:33.818891125 +0000 UTC m=+4553.244497592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-maas-api-tls") pod "maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw" (UID: "c50fad98-b746-40f6-853e-887f2da38314") : secret "maas-api-e2e-mt-eec2ff70-serving-cert" not found
Nov 03 02:34:33.319482 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.319453    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50fad98-b746-40f6-853e-887f2da38314-openshift-service-ca\") pod \"maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:33.321881 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.321853    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-metrics-certs\") pod \"maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:33.377437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.377403    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxnh7\" (UniqueName: \"kubernetes.io/projected/c50fad98-b746-40f6-853e-887f2da38314-kube-api-access-vxnh7\") pod \"maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:33.835466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.835429    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-maas-api-tls\") pod \"maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:33.838188 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:33.838159    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-maas-api-tls\") pod \"maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:34.095540 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:34.095435    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:34.481974 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:34.481933    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"]
Nov 03 02:34:34.488983 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:34:34.488945    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50fad98_b746_40f6_853e_887f2da38314.slice/crio-457e2fc63e476068de910eb4f86bc1df38d304f657da93387862135ebd73a071 WatchSource:0}: Error finding container 457e2fc63e476068de910eb4f86bc1df38d304f657da93387862135ebd73a071: Status 404 returned error can't find the container with id 457e2fc63e476068de910eb4f86bc1df38d304f657da93387862135ebd73a071
Nov 03 02:34:35.197603 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:35.197569    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw" event={"ID":"c50fad98-b746-40f6-853e-887f2da38314","Type":"ContainerStarted","Data":"457e2fc63e476068de910eb4f86bc1df38d304f657da93387862135ebd73a071"}
Nov 03 02:34:36.226201 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:36.226160    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw" event={"ID":"c50fad98-b746-40f6-853e-887f2da38314","Type":"ContainerStarted","Data":"93aad97c433cc0c1d7219580f0b94561996ad59946a8aaa02ddc3fc15671b99a"}
Nov 03 02:34:36.227870 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:36.227843    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:36.340396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:36.340332    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw" podStartSLOduration=2.46717135 podStartE2EDuration="3.340317367s" totalImagesPullingTime="873.146017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:34:36.337584244 +0000 UTC m=+4555.763190709" watchObservedRunningTime="2026-11-03 02:34:36.340317367 +0000 UTC m=+4555.765923831"
Nov 03 02:34:43.047298 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:43.047257    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"]
Nov 03 02:34:43.067875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:43.067839    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a83ac7-7qrqs"]
Nov 03 02:34:43.246120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:43.246089    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:34:43.813798 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:34:43.813761    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:34:44.694559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:44.694527    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76169ad3-96ee-48f1-824c-9defe8f83540" path="/var/lib/kubelet/pods/76169ad3-96ee-48f1-824c-9defe8f83540/volumes"
Nov 03 02:34:46.366596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:34:46.366564    5508 scope.go:122] "RemoveContainer" containerID="e5cbe36f26d5a6b47dc120a8c33c3816c277b14734f2d25f5f76b72afe423e8c"
Nov 03 02:35:00.443793 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.443757    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"]
Nov 03 02:35:00.444518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.444494    5508 cpu_manager.go:414] "containerMap: removing container" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerName="istio-proxy"
Nov 03 02:35:00.444734 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.444709    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerName="istio-proxy"
Nov 03 02:35:00.444939 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.444925    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed3911d-2e2f-463d-93aa-cd8ef0bf9053" containerName="istio-proxy"
Nov 03 02:35:00.450285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.450263    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.568376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.555418    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"]
Nov 03 02:35:00.568376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.566815    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-f0391a-openshift-default-dockercfg-rj9vj\""
Nov 03 02:35:00.658687 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.651686    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8227j\" (UniqueName: \"kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-kube-api-access-8227j\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.658687 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.651763    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-token\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.658687 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.651813    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-socket\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.658687 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.651839    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istiod-ca-cert\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.658687 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.651883    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-envoy\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.658687 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.651917    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-podinfo\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.658687 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.651991    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-data\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.658687 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.652119    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-credential-socket\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.658687 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.652209    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-certs\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.753986 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.753897    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-certs\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.754188 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.753985    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8227j\" (UniqueName: \"kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-kube-api-access-8227j\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.754188 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.754021    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-token\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.754188 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.754046    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-socket\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.754188 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.754067    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istiod-ca-cert\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.754491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.754258    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-envoy\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.754491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.754309    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-podinfo\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.754491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.754339    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-certs\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.754491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.754429    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-data\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.754491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.754445    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-socket\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.754783 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.754552    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-credential-socket\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.754783 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.754696    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-data\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.754905 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.754821    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istiod-ca-cert\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.754905 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.754866    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-credential-socket\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.757110 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.757082    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-envoy\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.757311 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.757284    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-podinfo\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.811256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.811219    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-token\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:00.811446 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:00.811225    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8227j\" (UniqueName: \"kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-kube-api-access-8227j\") pod \"e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") " pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:01.061245 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:01.061148    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"]
Nov 03 02:35:01.064751 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:01.064671    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d956463-h47zf"]
Nov 03 02:35:01.069470 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:01.069447    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:01.242129 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:01.241983    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:35:01.564559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:01.564433    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"]
Nov 03 02:35:01.570672 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:35:01.570645    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04df94de_2714_4273_b4c2_3f38f5b3fbd4.slice/crio-b262dc570e4d3b4054ddbe4312252d00b453040a4d74165f80a85bc81847fd41 WatchSource:0}: Error finding container b262dc570e4d3b4054ddbe4312252d00b453040a4d74165f80a85bc81847fd41: Status 404 returned error can't find the container with id b262dc570e4d3b4054ddbe4312252d00b453040a4d74165f80a85bc81847fd41
Nov 03 02:35:01.573341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:01.573309    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:35:01.573491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:01.573429    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:35:01.573491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:01.573476    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:35:02.427700 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:02.427663    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" event={"ID":"04df94de-2714-4273-b4c2-3f38f5b3fbd4","Type":"ContainerStarted","Data":"177fda0f752940c72f56c326490e8dcc649ee163eb0ab90584c313d4cf267784"}
Nov 03 02:35:02.427921 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:02.427713    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" event={"ID":"04df94de-2714-4273-b4c2-3f38f5b3fbd4","Type":"ContainerStarted","Data":"b262dc570e4d3b4054ddbe4312252d00b453040a4d74165f80a85bc81847fd41"}
Nov 03 02:35:02.543798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:02.543333    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" podStartSLOduration=2.54330976 podStartE2EDuration="2.54330976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:02.539996494 +0000 UTC m=+4581.965602970" watchObservedRunningTime="2026-11-03 02:35:02.54330976 +0000 UTC m=+4581.968916226"
Nov 03 02:35:02.709472 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:02.709380    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501b9d64-d4d3-403f-a321-17935b78eaf2" path="/var/lib/kubelet/pods/501b9d64-d4d3-403f-a321-17935b78eaf2/volumes"
Nov 03 02:35:03.071431 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:03.070729    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:03.079338 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:03.079305    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:03.434307 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:03.434276    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:03.435230 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:03.435211    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:35:06.985922 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:06.985881    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"]
Nov 03 02:35:06.986449 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:06.986138    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9" podUID="fa6cdc5d-e308-4560-96fb-8a1fa621c19c" containerName="maas-api" containerID="cri-o://5bb60a6fcff120bd72280ccadea7d9b4816bc1423faa202a738d37ff2fd02cc0" gracePeriod=30
Nov 03 02:35:07.527886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.527848    5508 generic.go:356] "Generic (PLEG): container finished" podID="fa6cdc5d-e308-4560-96fb-8a1fa621c19c" containerID="5bb60a6fcff120bd72280ccadea7d9b4816bc1423faa202a738d37ff2fd02cc0" exitCode=0
Nov 03 02:35:07.528101 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.527920    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9" event={"ID":"fa6cdc5d-e308-4560-96fb-8a1fa621c19c","Type":"ContainerDied","Data":"5bb60a6fcff120bd72280ccadea7d9b4816bc1423faa202a738d37ff2fd02cc0"}
Nov 03 02:35:07.528101 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.527970    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9" event={"ID":"fa6cdc5d-e308-4560-96fb-8a1fa621c19c","Type":"ContainerDied","Data":"2dd7a6f5810c5e57c8e96301c60465ab1f1bd5f5da72c9853f6f2b4c3468da2f"}
Nov 03 02:35:07.528101 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.527993    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dd7a6f5810c5e57c8e96301c60465ab1f1bd5f5da72c9853f6f2b4c3468da2f"
Nov 03 02:35:07.530103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.530079    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:35:07.606064 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.606028    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-metrics-certs\") pod \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") "
Nov 03 02:35:07.606837 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.606816    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-maas-api-tls\") pod \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") "
Nov 03 02:35:07.607249 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.607231    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-kube-api-access-skw6w\" (UniqueName: \"kubernetes.io/projected/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-kube-api-access-skw6w\") pod \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") "
Nov 03 02:35:07.607424 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.607407    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-openshift-service-ca\") pod \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\" (UID: \"fa6cdc5d-e308-4560-96fb-8a1fa621c19c\") "
Nov 03 02:35:07.607799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.607775    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-openshift-service-ca" pod "fa6cdc5d-e308-4560-96fb-8a1fa621c19c" (UID: "fa6cdc5d-e308-4560-96fb-8a1fa621c19c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:35:07.608259 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.608239    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-metrics-certs" pod "fa6cdc5d-e308-4560-96fb-8a1fa621c19c" (UID: "fa6cdc5d-e308-4560-96fb-8a1fa621c19c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:35:07.608384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.608268    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:07.608777 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.608762    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-maas-api-tls" pod "fa6cdc5d-e308-4560-96fb-8a1fa621c19c" (UID: "fa6cdc5d-e308-4560-96fb-8a1fa621c19c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:35:07.610714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.610695    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-kube-api-access-skw6w" pod "fa6cdc5d-e308-4560-96fb-8a1fa621c19c" (UID: "fa6cdc5d-e308-4560-96fb-8a1fa621c19c"). InnerVolumeSpecName "kube-api-access-skw6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:35:07.710485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.710399    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:07.710485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.710436    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skw6w\" (UniqueName: \"kubernetes.io/projected/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-kube-api-access-skw6w\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:07.710485 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:07.710449    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa6cdc5d-e308-4560-96fb-8a1fa621c19c-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:08.535374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:08.535317    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"
Nov 03 02:35:08.693828 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:08.693752    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"]
Nov 03 02:35:08.739380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:08.739334    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-1c9b4e-787f7f7d4b-mq5k9"]
Nov 03 02:35:09.193664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.193631    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"]
Nov 03 02:35:09.194387 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.194336    5508 cpu_manager.go:414] "containerMap: removing container" podUID="fa6cdc5d-e308-4560-96fb-8a1fa621c19c" containerName="maas-api"
Nov 03 02:35:09.194576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.194553    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6cdc5d-e308-4560-96fb-8a1fa621c19c" containerName="maas-api"
Nov 03 02:35:09.194837 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.194822    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa6cdc5d-e308-4560-96fb-8a1fa621c19c" containerName="maas-api"
Nov 03 02:35:09.199080 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.199060    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"
Nov 03 02:35:09.248052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.248020    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-f0391a\""
Nov 03 02:35:09.248287 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.248206    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-f0391a-dockercfg-6h5d4\""
Nov 03 02:35:09.262742 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.262699    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"]
Nov 03 02:35:09.342385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.341586    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6502ec5-52cb-4824-894e-d77e374f67a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8\" (UID: \"e6502ec5-52cb-4824-894e-d77e374f67a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"
Nov 03 02:35:09.342385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.341725    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqzp9\" (UniqueName: \"kubernetes.io/projected/e6502ec5-52cb-4824-894e-d77e374f67a4-kube-api-access-vqzp9\") pod \"payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8\" (UID: \"e6502ec5-52cb-4824-894e-d77e374f67a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"
Nov 03 02:35:09.413372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.400178    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"]
Nov 03 02:35:09.460425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.455677    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6502ec5-52cb-4824-894e-d77e374f67a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8\" (UID: \"e6502ec5-52cb-4824-894e-d77e374f67a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"
Nov 03 02:35:09.460425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.455803    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqzp9\" (UniqueName: \"kubernetes.io/projected/e6502ec5-52cb-4824-894e-d77e374f67a4-kube-api-access-vqzp9\") pod \"payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8\" (UID: \"e6502ec5-52cb-4824-894e-d77e374f67a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"
Nov 03 02:35:09.460425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.456969    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6502ec5-52cb-4824-894e-d77e374f67a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8\" (UID: \"e6502ec5-52cb-4824-894e-d77e374f67a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"
Nov 03 02:35:09.460996 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.460973    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"
Nov 03 02:35:09.473605 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.468750    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"]
Nov 03 02:35:09.518531 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.518498    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqzp9\" (UniqueName: \"kubernetes.io/projected/e6502ec5-52cb-4824-894e-d77e374f67a4-kube-api-access-vqzp9\") pod \"payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8\" (UID: \"e6502ec5-52cb-4824-894e-d77e374f67a4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"
Nov 03 02:35:09.557381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.553487    5508 patch_prober.go:29] interesting pod/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:35:09.557381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.553545    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:35:09.557381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.557019    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms\" (UID: \"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"
Nov 03 02:35:09.557381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.557068    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7s26\" (UniqueName: \"kubernetes.io/projected/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-kube-api-access-x7s26\") pod \"payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms\" (UID: \"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"
Nov 03 02:35:09.658591 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.658546    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms\" (UID: \"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"
Nov 03 02:35:09.658822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.658603    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7s26\" (UniqueName: \"kubernetes.io/projected/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-kube-api-access-x7s26\") pod \"payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms\" (UID: \"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"
Nov 03 02:35:09.659341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.659310    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms\" (UID: \"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"
Nov 03 02:35:09.718534 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.718458    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7s26\" (UniqueName: \"kubernetes.io/projected/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-kube-api-access-x7s26\") pod \"payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms\" (UID: \"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"
Nov 03 02:35:09.788424 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.788391    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"
Nov 03 02:35:09.813919 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:09.813878    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"
Nov 03 02:35:10.168496 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:10.168455    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"]
Nov 03 02:35:10.174823 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:35:10.174793    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8701d1_ea9a_4aa3_87c3_095d8eb6ff2b.slice/crio-08ec46881f0d826724b79e53a550779e60adf98905fead1ccf44690f7ff6b62e WatchSource:0}: Error finding container 08ec46881f0d826724b79e53a550779e60adf98905fead1ccf44690f7ff6b62e: Status 404 returned error can't find the container with id 08ec46881f0d826724b79e53a550779e60adf98905fead1ccf44690f7ff6b62e
Nov 03 02:35:10.221559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:10.221473    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"]
Nov 03 02:35:10.241939 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:35:10.241911    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6502ec5_52cb_4824_894e_d77e374f67a4.slice/crio-edebe6af840824ac70a39fbfd6d64154be1275c47cc635980a1da79ab3cb00e3 WatchSource:0}: Error finding container edebe6af840824ac70a39fbfd6d64154be1275c47cc635980a1da79ab3cb00e3: Status 404 returned error can't find the container with id edebe6af840824ac70a39fbfd6d64154be1275c47cc635980a1da79ab3cb00e3
Nov 03 02:35:10.556193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:10.556092    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms" event={"ID":"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b","Type":"ContainerStarted","Data":"3be1340a0966e6fbdd60467384e961f66bd1a18ed9ecad2a1808c3d33255f81a"}
Nov 03 02:35:10.556193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:10.556167    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"
Nov 03 02:35:10.556193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:10.556190    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms" event={"ID":"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b","Type":"ContainerStarted","Data":"08ec46881f0d826724b79e53a550779e60adf98905fead1ccf44690f7ff6b62e"}
Nov 03 02:35:10.558031 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:10.558005    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8" event={"ID":"e6502ec5-52cb-4824-894e-d77e374f67a4","Type":"ContainerStarted","Data":"8b04ad9506c7612475fe7fd8d48770d5ae05c142e3a9237a3cf1f9b4f893438d"}
Nov 03 02:35:10.558494 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:10.558044    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8" event={"ID":"e6502ec5-52cb-4824-894e-d77e374f67a4","Type":"ContainerStarted","Data":"edebe6af840824ac70a39fbfd6d64154be1275c47cc635980a1da79ab3cb00e3"}
Nov 03 02:35:10.558494 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:10.558156    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"
Nov 03 02:35:10.669234 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:10.669158    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms" podStartSLOduration=1.669136486 podStartE2EDuration="1.669136486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:10.664479048 +0000 UTC m=+4590.090085513" watchObservedRunningTime="2026-11-03 02:35:10.669136486 +0000 UTC m=+4590.094742955"
Nov 03 02:35:10.696798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:10.696760    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa6cdc5d-e308-4560-96fb-8a1fa621c19c" path="/var/lib/kubelet/pods/fa6cdc5d-e308-4560-96fb-8a1fa621c19c/volumes"
Nov 03 02:35:10.774445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:10.774388    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8" podStartSLOduration=1.774373966 podStartE2EDuration="1.774373966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:10.771898384 +0000 UTC m=+4590.197504849" watchObservedRunningTime="2026-11-03 02:35:10.774373966 +0000 UTC m=+4590.199980432"
Nov 03 02:35:12.575142 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:12.575105    5508 generic.go:356] "Generic (PLEG): container finished" podID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerID="f2371369c17d6c2eef4ad0a6b48e0ff0ec4ece40a3a30f4339c1a4c8a204ccb0" exitCode=0
Nov 03 02:35:12.575651 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:12.575178    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" event={"ID":"a199c50f-4280-402e-b6cd-9433e0639b7a","Type":"ContainerDied","Data":"f2371369c17d6c2eef4ad0a6b48e0ff0ec4ece40a3a30f4339c1a4c8a204ccb0"}
Nov 03 02:35:12.575651 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:12.575628    5508 scope.go:122] "RemoveContainer" containerID="f2371369c17d6c2eef4ad0a6b48e0ff0ec4ece40a3a30f4339c1a4c8a204ccb0"
Nov 03 02:35:12.577326 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:12.577298    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:35:12.577429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:12.577381    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:35:12.577429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:12.577414    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:35:13.584558 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:13.584525    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" event={"ID":"a199c50f-4280-402e-b6cd-9433e0639b7a","Type":"ContainerStarted","Data":"f3004218e39a35d4d01bba3c254ec3ec6462d058f796dccfe1021fb5553b2813"}
Nov 03 02:35:13.856428 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:35:13.853734    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:35:14.129499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:14.129415    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:35:14.129675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:14.129639    5508 patch_prober.go:29] interesting pod/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Nov 03 02:35:14.129736 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:14.129685    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Nov 03 02:35:15.129856 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:15.129818    5508 patch_prober.go:29] interesting pod/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Nov 03 02:35:15.130337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:15.129876    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Nov 03 02:35:16.129838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:16.129756    5508 patch_prober.go:29] interesting pod/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Nov 03 02:35:16.129838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:16.129811    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Nov 03 02:35:17.130197 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:17.130157    5508 patch_prober.go:29] interesting pod/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Nov 03 02:35:17.130643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:17.130222    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Nov 03 02:35:18.129804 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.129762    5508 patch_prober.go:29] interesting pod/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Nov 03 02:35:18.130007 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.129820    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Nov 03 02:35:18.635038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.635000    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"]
Nov 03 02:35:18.639527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.639504    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"
Nov 03 02:35:18.688200 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.688170    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:35:18.710545 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.710516    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:35:18.737247 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.737206    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41bc39c3-jj6kg\" (UID: \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"
Nov 03 02:35:18.737452 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.737277    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdfhb\" (UniqueName: \"kubernetes.io/projected/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-kube-api-access-jdfhb\") pod \"maas-api-revoke-sub-41bc39c3-jj6kg\" (UID: \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"
Nov 03 02:35:18.737504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.737465    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41bc39c3-jj6kg\" (UID: \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"
Nov 03 02:35:18.739748 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.739717    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"]
Nov 03 02:35:18.838709 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.838671    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41bc39c3-jj6kg\" (UID: \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"
Nov 03 02:35:18.838909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.838827    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41bc39c3-jj6kg\" (UID: \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"
Nov 03 02:35:18.838909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.838869    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdfhb\" (UniqueName: \"kubernetes.io/projected/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-kube-api-access-jdfhb\") pod \"maas-api-revoke-sub-41bc39c3-jj6kg\" (UID: \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"
Nov 03 02:35:18.839599 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.839570    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41bc39c3-jj6kg\" (UID: \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"
Nov 03 02:35:18.841201 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.841171    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41bc39c3-jj6kg\" (UID: \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"
Nov 03 02:35:18.896400 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.896369    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdfhb\" (UniqueName: \"kubernetes.io/projected/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-kube-api-access-jdfhb\") pod \"maas-api-revoke-sub-41bc39c3-jj6kg\" (UID: \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"
Nov 03 02:35:18.954106 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:18.954067    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"
Nov 03 02:35:19.134520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:19.134492    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:35:19.135031 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:19.135006    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:35:19.135790 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:19.135769    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:35:19.381601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:19.381163    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"]
Nov 03 02:35:19.699864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:19.699827    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg" event={"ID":"31fa90d8-9ef4-41bd-b11d-c903b734e8f2","Type":"ContainerStarted","Data":"abd36e1ab769149fc3764b0d42733d7f35b3a421c84932d237a9ca0e4ba35828"}
Nov 03 02:35:19.700364 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:19.699890    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg" event={"ID":"31fa90d8-9ef4-41bd-b11d-c903b734e8f2","Type":"ContainerStarted","Data":"83b57e98d951d02a9e8f19d66a0c1260dcee0663e11e2c677935b035494f10ff"}
Nov 03 02:35:19.809065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:19.809012    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg" podStartSLOduration=1.808998296 podStartE2EDuration="1.808998296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:19.808083244 +0000 UTC m=+4599.233689709" watchObservedRunningTime="2026-11-03 02:35:19.808998296 +0000 UTC m=+4599.234604761"
Nov 03 02:35:20.711526 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:20.711493    5508 generic.go:356] "Generic (PLEG): container finished" podID="31fa90d8-9ef4-41bd-b11d-c903b734e8f2" containerID="abd36e1ab769149fc3764b0d42733d7f35b3a421c84932d237a9ca0e4ba35828" exitCode=0
Nov 03 02:35:20.712038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:20.711563    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg" event={"ID":"31fa90d8-9ef4-41bd-b11d-c903b734e8f2","Type":"ContainerDied","Data":"abd36e1ab769149fc3764b0d42733d7f35b3a421c84932d237a9ca0e4ba35828"}
Nov 03 02:35:21.570237 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:21.570206    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"
Nov 03 02:35:21.570461 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:21.570290    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"
Nov 03 02:35:21.907044 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:21.907008    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"
Nov 03 02:35:21.981560 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:21.981527    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-kube-api-access-jdfhb\" (UniqueName: \"kubernetes.io/projected/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-kube-api-access-jdfhb\") pod \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\" (UID: \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\") "
Nov 03 02:35:21.981760 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:21.981579    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-openshift-ca\") pod \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\" (UID: \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\") "
Nov 03 02:35:21.981760 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:21.981613    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-xks-ca\") pod \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\" (UID: \"31fa90d8-9ef4-41bd-b11d-c903b734e8f2\") "
Nov 03 02:35:21.982014 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:21.981985    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-openshift-ca" pod "31fa90d8-9ef4-41bd-b11d-c903b734e8f2" (UID: "31fa90d8-9ef4-41bd-b11d-c903b734e8f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:35:21.982307 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:21.982281    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:21.984292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:21.984265    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-xks-ca" pod "31fa90d8-9ef4-41bd-b11d-c903b734e8f2" (UID: "31fa90d8-9ef4-41bd-b11d-c903b734e8f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:35:21.984385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:21.984285    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-kube-api-access-jdfhb" pod "31fa90d8-9ef4-41bd-b11d-c903b734e8f2" (UID: "31fa90d8-9ef4-41bd-b11d-c903b734e8f2"). InnerVolumeSpecName "kube-api-access-jdfhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:35:22.083745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:22.083703    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdfhb\" (UniqueName: \"kubernetes.io/projected/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-kube-api-access-jdfhb\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:22.083745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:22.083740    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31fa90d8-9ef4-41bd-b11d-c903b734e8f2-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:22.728414 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:22.728383    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"
Nov 03 02:35:22.728595 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:22.728405    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg" event={"ID":"31fa90d8-9ef4-41bd-b11d-c903b734e8f2","Type":"ContainerDied","Data":"83b57e98d951d02a9e8f19d66a0c1260dcee0663e11e2c677935b035494f10ff"}
Nov 03 02:35:22.728595 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:22.728459    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83b57e98d951d02a9e8f19d66a0c1260dcee0663e11e2c677935b035494f10ff"
Nov 03 02:35:23.006392 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:23.006261    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"]
Nov 03 02:35:31.398654 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.398621    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"]
Nov 03 02:35:31.399295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.399277    5508 cpu_manager.go:414] "containerMap: removing container" podUID="31fa90d8-9ef4-41bd-b11d-c903b734e8f2" containerName="revoke-keys"
Nov 03 02:35:31.399295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.399295    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31fa90d8-9ef4-41bd-b11d-c903b734e8f2" containerName="revoke-keys"
Nov 03 02:35:31.399718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.399473    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="31fa90d8-9ef4-41bd-b11d-c903b734e8f2" containerName="revoke-keys"
Nov 03 02:35:31.402800 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.402782    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"
Nov 03 02:35:31.452203 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.452159    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:35:31.498501 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.498464    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:35:31.502318 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.502288    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"]
Nov 03 02:35:31.525936 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.525908    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv5bg\" (UniqueName: \"kubernetes.io/projected/0411036c-38ea-4b9e-be73-475c2b9491c7-kube-api-access-vv5bg\") pod \"maas-api-revoke-sub-a524fe7a-z2629\" (UID: \"0411036c-38ea-4b9e-be73-475c2b9491c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"
Nov 03 02:35:31.526068 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.525992    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a524fe7a-z2629\" (UID: \"0411036c-38ea-4b9e-be73-475c2b9491c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"
Nov 03 02:35:31.526068 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.526026    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a524fe7a-z2629\" (UID: \"0411036c-38ea-4b9e-be73-475c2b9491c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"
Nov 03 02:35:31.627731 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.627693    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv5bg\" (UniqueName: \"kubernetes.io/projected/0411036c-38ea-4b9e-be73-475c2b9491c7-kube-api-access-vv5bg\") pod \"maas-api-revoke-sub-a524fe7a-z2629\" (UID: \"0411036c-38ea-4b9e-be73-475c2b9491c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"
Nov 03 02:35:31.627919 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.627758    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a524fe7a-z2629\" (UID: \"0411036c-38ea-4b9e-be73-475c2b9491c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"
Nov 03 02:35:31.627919 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.627782    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a524fe7a-z2629\" (UID: \"0411036c-38ea-4b9e-be73-475c2b9491c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"
Nov 03 02:35:31.628575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.628443    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a524fe7a-z2629\" (UID: \"0411036c-38ea-4b9e-be73-475c2b9491c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"
Nov 03 02:35:31.630174 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.630149    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a524fe7a-z2629\" (UID: \"0411036c-38ea-4b9e-be73-475c2b9491c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"
Nov 03 02:35:31.667684 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.667621    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"]
Nov 03 02:35:31.685093 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.685059    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv5bg\" (UniqueName: \"kubernetes.io/projected/0411036c-38ea-4b9e-be73-475c2b9491c7-kube-api-access-vv5bg\") pod \"maas-api-revoke-sub-a524fe7a-z2629\" (UID: \"0411036c-38ea-4b9e-be73-475c2b9491c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"
Nov 03 02:35:31.717574 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:31.717547    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"
Nov 03 02:35:32.105306 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:32.105222    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"]
Nov 03 02:35:32.112087 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:35:32.112060    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0411036c_38ea_4b9e_be73_475c2b9491c7.slice/crio-3405f45548488b35c09cfcad7af5d32e771cc6fd9a7e575928494933df09c023 WatchSource:0}: Error finding container 3405f45548488b35c09cfcad7af5d32e771cc6fd9a7e575928494933df09c023: Status 404 returned error can't find the container with id 3405f45548488b35c09cfcad7af5d32e771cc6fd9a7e575928494933df09c023
Nov 03 02:35:32.812730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:32.812635    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629" event={"ID":"0411036c-38ea-4b9e-be73-475c2b9491c7","Type":"ContainerStarted","Data":"7ffe44d3d678cff4583cd845df0da434dc511dd5d51985328fa580f0505e7a2e"}
Nov 03 02:35:32.812730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:32.812694    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629" event={"ID":"0411036c-38ea-4b9e-be73-475c2b9491c7","Type":"ContainerStarted","Data":"3405f45548488b35c09cfcad7af5d32e771cc6fd9a7e575928494933df09c023"}
Nov 03 02:35:32.921078 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:32.921014    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629" podStartSLOduration=1.9209976979999999 podStartE2EDuration="1.920997698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:35:32.919939763 +0000 UTC m=+4612.345546229" watchObservedRunningTime="2026-11-03 02:35:32.920997698 +0000 UTC m=+4612.346604164"
Nov 03 02:35:33.822903 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:33.822866    5508 generic.go:356] "Generic (PLEG): container finished" podID="0411036c-38ea-4b9e-be73-475c2b9491c7" containerID="7ffe44d3d678cff4583cd845df0da434dc511dd5d51985328fa580f0505e7a2e" exitCode=0
Nov 03 02:35:33.823333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:33.822952    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629" event={"ID":"0411036c-38ea-4b9e-be73-475c2b9491c7","Type":"ContainerDied","Data":"7ffe44d3d678cff4583cd845df0da434dc511dd5d51985328fa580f0505e7a2e"}
Nov 03 02:35:34.984177 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:34.984147    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"
Nov 03 02:35:35.182072 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:35.182040    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-xks-ca\") pod \"0411036c-38ea-4b9e-be73-475c2b9491c7\" (UID: \"0411036c-38ea-4b9e-be73-475c2b9491c7\") "
Nov 03 02:35:35.182273 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:35.182260    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0411036c-38ea-4b9e-be73-475c2b9491c7-kube-api-access-vv5bg\" (UniqueName: \"kubernetes.io/projected/0411036c-38ea-4b9e-be73-475c2b9491c7-kube-api-access-vv5bg\") pod \"0411036c-38ea-4b9e-be73-475c2b9491c7\" (UID: \"0411036c-38ea-4b9e-be73-475c2b9491c7\") "
Nov 03 02:35:35.182344 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:35.182333    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-openshift-ca\") pod \"0411036c-38ea-4b9e-be73-475c2b9491c7\" (UID: \"0411036c-38ea-4b9e-be73-475c2b9491c7\") "
Nov 03 02:35:35.182774 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:35.182741    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-openshift-ca" pod "0411036c-38ea-4b9e-be73-475c2b9491c7" (UID: "0411036c-38ea-4b9e-be73-475c2b9491c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:35:35.184228 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:35.184206    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-xks-ca" pod "0411036c-38ea-4b9e-be73-475c2b9491c7" (UID: "0411036c-38ea-4b9e-be73-475c2b9491c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:35:35.184385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:35.184344    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0411036c-38ea-4b9e-be73-475c2b9491c7-kube-api-access-vv5bg" pod "0411036c-38ea-4b9e-be73-475c2b9491c7" (UID: "0411036c-38ea-4b9e-be73-475c2b9491c7"). InnerVolumeSpecName "kube-api-access-vv5bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:35:35.283787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:35.283753    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:35.283787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:35.283785    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv5bg\" (UniqueName: \"kubernetes.io/projected/0411036c-38ea-4b9e-be73-475c2b9491c7-kube-api-access-vv5bg\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:35.283787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:35.283796    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0411036c-38ea-4b9e-be73-475c2b9491c7-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:35.837941 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:35.837907    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"
Nov 03 02:35:35.837941 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:35.837931    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629" event={"ID":"0411036c-38ea-4b9e-be73-475c2b9491c7","Type":"ContainerDied","Data":"3405f45548488b35c09cfcad7af5d32e771cc6fd9a7e575928494933df09c023"}
Nov 03 02:35:35.838192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:35.837983    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3405f45548488b35c09cfcad7af5d32e771cc6fd9a7e575928494933df09c023"
Nov 03 02:35:36.667990 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:36.667944    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6" podUID="27a3ebaf-96d0-4664-9775-2207983e2a96" containerName="payload-processing" containerID="cri-o://f4e81d9417c12e7eece1e3f98fdce565d248175be1b526dba3e3a09718a6aea2" gracePeriod=25
Nov 03 02:35:36.850381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:36.848309    5508 generic.go:356] "Generic (PLEG): container finished" podID="27a3ebaf-96d0-4664-9775-2207983e2a96" containerID="f4e81d9417c12e7eece1e3f98fdce565d248175be1b526dba3e3a09718a6aea2" exitCode=0
Nov 03 02:35:36.850381 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:36.848438    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6" event={"ID":"27a3ebaf-96d0-4664-9775-2207983e2a96","Type":"ContainerDied","Data":"f4e81d9417c12e7eece1e3f98fdce565d248175be1b526dba3e3a09718a6aea2"}
Nov 03 02:35:36.942622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:36.942597    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"
Nov 03 02:35:37.005473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:37.005445    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a3ebaf-96d0-4664-9775-2207983e2a96-kube-api-access-rjsz4\" (UniqueName: \"kubernetes.io/projected/27a3ebaf-96d0-4664-9775-2207983e2a96-kube-api-access-rjsz4\") pod \"27a3ebaf-96d0-4664-9775-2207983e2a96\" (UID: \"27a3ebaf-96d0-4664-9775-2207983e2a96\") "
Nov 03 02:35:37.005657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:37.005533    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27a3ebaf-96d0-4664-9775-2207983e2a96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a3ebaf-96d0-4664-9775-2207983e2a96-plugins-config-volume\") pod \"27a3ebaf-96d0-4664-9775-2207983e2a96\" (UID: \"27a3ebaf-96d0-4664-9775-2207983e2a96\") "
Nov 03 02:35:37.006051 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:37.006026    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a3ebaf-96d0-4664-9775-2207983e2a96-plugins-config-volume" pod "27a3ebaf-96d0-4664-9775-2207983e2a96" (UID: "27a3ebaf-96d0-4664-9775-2207983e2a96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:35:37.008002 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:37.007981    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a3ebaf-96d0-4664-9775-2207983e2a96-kube-api-access-rjsz4" pod "27a3ebaf-96d0-4664-9775-2207983e2a96" (UID: "27a3ebaf-96d0-4664-9775-2207983e2a96"). InnerVolumeSpecName "kube-api-access-rjsz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:35:37.106724 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:37.106692    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a3ebaf-96d0-4664-9775-2207983e2a96-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:37.106724 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:37.106720    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjsz4\" (UniqueName: \"kubernetes.io/projected/27a3ebaf-96d0-4664-9775-2207983e2a96-kube-api-access-rjsz4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:37.858604 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:37.858569    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"
Nov 03 02:35:37.858604 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:37.858580    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6" event={"ID":"27a3ebaf-96d0-4664-9775-2207983e2a96","Type":"ContainerDied","Data":"4f7c70adf67a2bf5906dc06f6f99188741d164398f5ae807c5f85a6cae2d0941"}
Nov 03 02:35:37.859189 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:37.858650    5508 scope.go:122] "RemoveContainer" containerID="f4e81d9417c12e7eece1e3f98fdce565d248175be1b526dba3e3a09718a6aea2"
Nov 03 02:35:38.021911 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.021872    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"]
Nov 03 02:35:38.027779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.027740    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main" containerID="cri-o://5ec5a6471468a361081683eb32dc521adfe418e8fae4f39c1d6f513737986fb9" gracePeriod=45
Nov 03 02:35:38.068260 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.068228    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e1793d-68496fc9c9-n6kr6"]
Nov 03 02:35:38.367758 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.367729    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 02:35:38.424710 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.424678    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tls-certs\") pod \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") "
Nov 03 02:35:38.424926 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.424792    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-home\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-home\") pod \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") "
Nov 03 02:35:38.424926 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.424823    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tmp-dir\") pod \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") "
Nov 03 02:35:38.424926 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.424861    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-dshm\") pod \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") "
Nov 03 02:35:38.424926 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.424893    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-kube-api-access-kdc8x\" (UniqueName: \"kubernetes.io/projected/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-kube-api-access-kdc8x\") pod \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") "
Nov 03 02:35:38.425144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.424945    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-model-cache\") pod \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\" (UID: \"3fb5a7e5-d503-4be7-a6be-fe9f7f936028\") "
Nov 03 02:35:38.425144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.425062    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tmp-dir" pod "3fb5a7e5-d503-4be7-a6be-fe9f7f936028" (UID: "3fb5a7e5-d503-4be7-a6be-fe9f7f936028"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:35:38.425144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.425100    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-home" pod "3fb5a7e5-d503-4be7-a6be-fe9f7f936028" (UID: "3fb5a7e5-d503-4be7-a6be-fe9f7f936028"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:35:38.425319 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.425285    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-model-cache" pod "3fb5a7e5-d503-4be7-a6be-fe9f7f936028" (UID: "3fb5a7e5-d503-4be7-a6be-fe9f7f936028"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:35:38.425577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.425557    5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-home\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:38.425661 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.425581    5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tmp-dir\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:38.425661 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.425596    5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-model-cache\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:38.427115 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.427093    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tls-certs" pod "3fb5a7e5-d503-4be7-a6be-fe9f7f936028" (UID: "3fb5a7e5-d503-4be7-a6be-fe9f7f936028"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:35:38.427245 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.427135    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-dshm" pod "3fb5a7e5-d503-4be7-a6be-fe9f7f936028" (UID: "3fb5a7e5-d503-4be7-a6be-fe9f7f936028"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:35:38.427303 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.427234    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-kube-api-access-kdc8x" pod "3fb5a7e5-d503-4be7-a6be-fe9f7f936028" (UID: "3fb5a7e5-d503-4be7-a6be-fe9f7f936028"). InnerVolumeSpecName "kube-api-access-kdc8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:35:38.527295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.527256    5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-tls-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:38.527295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.527290    5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-dshm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:38.527295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.527300    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdc8x\" (UniqueName: \"kubernetes.io/projected/3fb5a7e5-d503-4be7-a6be-fe9f7f936028-kube-api-access-kdc8x\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:35:38.692794 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.692715    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a3ebaf-96d0-4664-9775-2207983e2a96" path="/var/lib/kubelet/pods/27a3ebaf-96d0-4664-9775-2207983e2a96/volumes"
Nov 03 02:35:38.866804 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.866768    5508 generic.go:356] "Generic (PLEG): container finished" podID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerID="5ec5a6471468a361081683eb32dc521adfe418e8fae4f39c1d6f513737986fb9" exitCode=0
Nov 03 02:35:38.867288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.866823    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"
Nov 03 02:35:38.867288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.866858    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" event={"ID":"3fb5a7e5-d503-4be7-a6be-fe9f7f936028","Type":"ContainerDied","Data":"5ec5a6471468a361081683eb32dc521adfe418e8fae4f39c1d6f513737986fb9"}
Nov 03 02:35:38.867288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.866912    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9" event={"ID":"3fb5a7e5-d503-4be7-a6be-fe9f7f936028","Type":"ContainerDied","Data":"3e7afc3b18559d587371632865a0a82347a35ea62aae07e5852f0bd878616bb6"}
Nov 03 02:35:38.867288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.866934    5508 scope.go:122] "RemoveContainer" containerID="5ec5a6471468a361081683eb32dc521adfe418e8fae4f39c1d6f513737986fb9"
Nov 03 02:35:38.877116 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.877097    5508 scope.go:122] "RemoveContainer" containerID="5ec5a6471468a361081683eb32dc521adfe418e8fae4f39c1d6f513737986fb9"
Nov 03 02:35:38.877414 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:35:38.877395    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec5a6471468a361081683eb32dc521adfe418e8fae4f39c1d6f513737986fb9\": container with ID starting with 5ec5a6471468a361081683eb32dc521adfe418e8fae4f39c1d6f513737986fb9 not found: ID does not exist" containerID="5ec5a6471468a361081683eb32dc521adfe418e8fae4f39c1d6f513737986fb9"
Nov 03 02:35:38.877491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:38.877424    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec5a6471468a361081683eb32dc521adfe418e8fae4f39c1d6f513737986fb9"} err="failed to get container status \"5ec5a6471468a361081683eb32dc521adfe418e8fae4f39c1d6f513737986fb9\": rpc error: code = NotFound desc = could not find container \"5ec5a6471468a361081683eb32dc521adfe418e8fae4f39c1d6f513737986fb9\": container with ID starting with 5ec5a6471468a361081683eb32dc521adfe418e8fae4f39c1d6f513737986fb9 not found: ID does not exist"
Nov 03 02:35:39.022107 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:39.022023    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"]
Nov 03 02:35:39.068432 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:39.068401    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e1793d/premium-w1-e1793d-kserve-7dd5954c8-kkgw9"]
Nov 03 02:35:40.695407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:40.695372    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" path="/var/lib/kubelet/pods/3fb5a7e5-d503-4be7-a6be-fe9f7f936028/volumes"
Nov 03 02:35:43.816376 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:35:43.815406    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:35:43.975414 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:43.975378    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:35:46.419007 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:46.418977    5508 scope.go:122] "RemoveContainer" containerID="5bb60a6fcff120bd72280ccadea7d9b4816bc1423faa202a738d37ff2fd02cc0"
Nov 03 02:35:46.429122 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:35:46.429099    5508 scope.go:122] "RemoveContainer" containerID="9748169732ceb4db756f230c0ef44dba47069ee0e0ade337e74cc217e99174ea"
Nov 03 02:36:13.814132 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:36:13.814097    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:36:39.213686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:39.213644    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:36:42.044634 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:42.044599    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"]
Nov 03 02:36:42.090532 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:42.090499    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0be01c8e-lw255"]
Nov 03 02:36:42.692148 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:42.692112    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98fda50c-5d63-472f-a465-faca6a3b95fb" path="/var/lib/kubelet/pods/98fda50c-5d63-472f-a465-faca6a3b95fb/volumes"
Nov 03 02:36:43.813285 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:36:43.813248    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:36:46.470809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:46.470771    5508 scope.go:122] "RemoveContainer" containerID="2c9f518788ae595031d41f377b6926c99d06af3cbc216d72a68eff4d1a7ae9a8"
Nov 03 02:36:49.475499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:49.475459    5508 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:36:49.475969 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:49.475535    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:36:52.429588 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:52.429556    5508 generic.go:356] "Generic (PLEG): container finished" podID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerID="177fda0f752940c72f56c326490e8dcc649ee163eb0ab90584c313d4cf267784" exitCode=0
Nov 03 02:36:52.430033 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:52.429617    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" event={"ID":"04df94de-2714-4273-b4c2-3f38f5b3fbd4","Type":"ContainerDied","Data":"177fda0f752940c72f56c326490e8dcc649ee163eb0ab90584c313d4cf267784"}
Nov 03 02:36:52.430271 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:52.430209    5508 scope.go:122] "RemoveContainer" containerID="177fda0f752940c72f56c326490e8dcc649ee163eb0ab90584c313d4cf267784"
Nov 03 02:36:52.432157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:52.432126    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:36:52.432242 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:52.432197    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:36:52.432242 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:52.432228    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:36:53.439568 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:53.439523    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" event={"ID":"04df94de-2714-4273-b4c2-3f38f5b3fbd4","Type":"ContainerStarted","Data":"108b4b81af888a5fc075b8c18b3222921eca8a386a149c1506438127bb9ebc65"}
Nov 03 02:36:54.071790 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:54.071755    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:36:54.072231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:54.072208    5508 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body=
Nov 03 02:36:54.072423 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:54.072394    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused"
Nov 03 02:36:55.076867 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:55.076822    5508 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body=
Nov 03 02:36:55.077512 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:55.077479    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused"
Nov 03 02:36:56.071876 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:56.071837    5508 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body=
Nov 03 02:36:56.072202 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:56.072172    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused"
Nov 03 02:36:57.075157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:57.075119    5508 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body=
Nov 03 02:36:57.076571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:57.076540    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused"
Nov 03 02:36:58.074373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:58.073758    5508 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused" start-of-body=
Nov 03 02:36:58.074373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:58.073826    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.93:15021/healthz/ready\": dial tcp 10.133.1.93:15021: connect: connection refused"
Nov 03 02:36:59.074790 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:59.074758    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:36:59.075331 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:59.075210    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:36:59.076160 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:36:59.076142    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:37:13.192646 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:13.192609    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:37:13.825394 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:37:13.825102    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:37:22.931834 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:22.931795    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"]
Nov 03 02:37:22.932857 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:22.932827    5508 cpu_manager.go:414] "containerMap: removing container" podUID="27a3ebaf-96d0-4664-9775-2207983e2a96" containerName="payload-processing"
Nov 03 02:37:22.932857 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:22.932857    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a3ebaf-96d0-4664-9775-2207983e2a96" containerName="payload-processing"
Nov 03 02:37:22.933075 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:22.932891    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0411036c-38ea-4b9e-be73-475c2b9491c7" containerName="revoke-keys"
Nov 03 02:37:22.933075 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:22.932899    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0411036c-38ea-4b9e-be73-475c2b9491c7" containerName="revoke-keys"
Nov 03 02:37:22.933075 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:22.932951    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main"
Nov 03 02:37:22.933075 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:22.932958    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main"
Nov 03 02:37:22.933227 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:22.933081    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a3ebaf-96d0-4664-9775-2207983e2a96" containerName="payload-processing"
Nov 03 02:37:22.933227 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:22.933091    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0411036c-38ea-4b9e-be73-475c2b9491c7" containerName="revoke-keys"
Nov 03 02:37:22.933227 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:22.933102    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb5a7e5-d503-4be7-a6be-fe9f7f936028" containerName="main"
Nov 03 02:37:22.936915 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:22.936895    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"
Nov 03 02:37:23.031940 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.031896    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:37:23.031940 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.031919    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:37:23.037658 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.037630    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"]
Nov 03 02:37:23.077173 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.077136    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f58757-8386-475c-a39a-623194abcef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18bf9aaa-bc54d\" (UID: \"59f58757-8386-475c-a39a-623194abcef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"
Nov 03 02:37:23.077421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.077179    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f58757-8386-475c-a39a-623194abcef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18bf9aaa-bc54d\" (UID: \"59f58757-8386-475c-a39a-623194abcef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"
Nov 03 02:37:23.077487 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.077425    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7csgq\" (UniqueName: \"kubernetes.io/projected/59f58757-8386-475c-a39a-623194abcef7-kube-api-access-7csgq\") pod \"maas-api-revoke-sub-18bf9aaa-bc54d\" (UID: \"59f58757-8386-475c-a39a-623194abcef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"
Nov 03 02:37:23.179079 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.179035    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7csgq\" (UniqueName: \"kubernetes.io/projected/59f58757-8386-475c-a39a-623194abcef7-kube-api-access-7csgq\") pod \"maas-api-revoke-sub-18bf9aaa-bc54d\" (UID: \"59f58757-8386-475c-a39a-623194abcef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"
Nov 03 02:37:23.179317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.179163    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f58757-8386-475c-a39a-623194abcef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18bf9aaa-bc54d\" (UID: \"59f58757-8386-475c-a39a-623194abcef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"
Nov 03 02:37:23.179317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.179189    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f58757-8386-475c-a39a-623194abcef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18bf9aaa-bc54d\" (UID: \"59f58757-8386-475c-a39a-623194abcef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"
Nov 03 02:37:23.179942 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.179916    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f58757-8386-475c-a39a-623194abcef7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18bf9aaa-bc54d\" (UID: \"59f58757-8386-475c-a39a-623194abcef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"
Nov 03 02:37:23.181640 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.181614    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f58757-8386-475c-a39a-623194abcef7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18bf9aaa-bc54d\" (UID: \"59f58757-8386-475c-a39a-623194abcef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"
Nov 03 02:37:23.236764 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.236692    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7csgq\" (UniqueName: \"kubernetes.io/projected/59f58757-8386-475c-a39a-623194abcef7-kube-api-access-7csgq\") pod \"maas-api-revoke-sub-18bf9aaa-bc54d\" (UID: \"59f58757-8386-475c-a39a-623194abcef7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"
Nov 03 02:37:23.250493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.250464    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"
Nov 03 02:37:23.635875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.635792    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"]
Nov 03 02:37:23.642806 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:37:23.642764    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f58757_8386_475c_a39a_623194abcef7.slice/crio-d5aba6eec9814ef166e0498498eb4d958b7531fcd9073ae3ebd3c5aa7a9adf86 WatchSource:0}: Error finding container d5aba6eec9814ef166e0498498eb4d958b7531fcd9073ae3ebd3c5aa7a9adf86: Status 404 returned error can't find the container with id d5aba6eec9814ef166e0498498eb4d958b7531fcd9073ae3ebd3c5aa7a9adf86
Nov 03 02:37:23.682422 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:23.682389    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d" event={"ID":"59f58757-8386-475c-a39a-623194abcef7","Type":"ContainerStarted","Data":"d5aba6eec9814ef166e0498498eb4d958b7531fcd9073ae3ebd3c5aa7a9adf86"}
Nov 03 02:37:24.703035 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:24.702994    5508 generic.go:356] "Generic (PLEG): container finished" podID="59f58757-8386-475c-a39a-623194abcef7" containerID="f24122ad335596d4d1669318203b7bb91202128bb3bcb90b2fe060770c688150" exitCode=0
Nov 03 02:37:24.705440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:24.705397    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d" event={"ID":"59f58757-8386-475c-a39a-623194abcef7","Type":"ContainerDied","Data":"f24122ad335596d4d1669318203b7bb91202128bb3bcb90b2fe060770c688150"}
Nov 03 02:37:25.863693 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:25.863664    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"
Nov 03 02:37:26.024831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:26.024748    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f58757-8386-475c-a39a-623194abcef7-kube-api-access-7csgq\" (UniqueName: \"kubernetes.io/projected/59f58757-8386-475c-a39a-623194abcef7-kube-api-access-7csgq\") pod \"59f58757-8386-475c-a39a-623194abcef7\" (UID: \"59f58757-8386-475c-a39a-623194abcef7\") "
Nov 03 02:37:26.025000 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:26.024867    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59f58757-8386-475c-a39a-623194abcef7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f58757-8386-475c-a39a-623194abcef7-maas-api-xks-ca\") pod \"59f58757-8386-475c-a39a-623194abcef7\" (UID: \"59f58757-8386-475c-a39a-623194abcef7\") "
Nov 03 02:37:26.025048 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:26.025023    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f58757-8386-475c-a39a-623194abcef7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f58757-8386-475c-a39a-623194abcef7-maas-api-openshift-ca\") pod \"59f58757-8386-475c-a39a-623194abcef7\" (UID: \"59f58757-8386-475c-a39a-623194abcef7\") "
Nov 03 02:37:26.025405 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:26.025377    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f58757-8386-475c-a39a-623194abcef7-maas-api-openshift-ca" pod "59f58757-8386-475c-a39a-623194abcef7" (UID: "59f58757-8386-475c-a39a-623194abcef7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:37:26.027069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:26.027039    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f58757-8386-475c-a39a-623194abcef7-maas-api-xks-ca" pod "59f58757-8386-475c-a39a-623194abcef7" (UID: "59f58757-8386-475c-a39a-623194abcef7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:37:26.027318 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:26.027293    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f58757-8386-475c-a39a-623194abcef7-kube-api-access-7csgq" pod "59f58757-8386-475c-a39a-623194abcef7" (UID: "59f58757-8386-475c-a39a-623194abcef7"). InnerVolumeSpecName "kube-api-access-7csgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:37:26.126984 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:26.126946    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f58757-8386-475c-a39a-623194abcef7-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:37:26.126984 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:26.126978    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f58757-8386-475c-a39a-623194abcef7-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:37:26.126984 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:26.126988    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7csgq\" (UniqueName: \"kubernetes.io/projected/59f58757-8386-475c-a39a-623194abcef7-kube-api-access-7csgq\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:37:26.720038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:26.719999    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d" event={"ID":"59f58757-8386-475c-a39a-623194abcef7","Type":"ContainerDied","Data":"d5aba6eec9814ef166e0498498eb4d958b7531fcd9073ae3ebd3c5aa7a9adf86"}
Nov 03 02:37:26.720038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:26.720032    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"
Nov 03 02:37:26.720288 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:26.720055    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5aba6eec9814ef166e0498498eb4d958b7531fcd9073ae3ebd3c5aa7a9adf86"
Nov 03 02:37:30.939071 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:30.939038    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"]
Nov 03 02:37:33.750464 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.750330    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"]
Nov 03 02:37:33.751907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.751882    5508 cpu_manager.go:414] "containerMap: removing container" podUID="59f58757-8386-475c-a39a-623194abcef7" containerName="revoke-keys"
Nov 03 02:37:33.752077 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.752064    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f58757-8386-475c-a39a-623194abcef7" containerName="revoke-keys"
Nov 03 02:37:33.752302 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.752290    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f58757-8386-475c-a39a-623194abcef7" containerName="revoke-keys"
Nov 03 02:37:33.756978 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.756953    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"
Nov 03 02:37:33.810862 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.810821    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:37:33.811552 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.811524    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"]
Nov 03 02:37:33.854112 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.854081    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:37:33.854895 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.854865    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54bc4a76-r82bj\" (UID: \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"
Nov 03 02:37:33.855089 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.855058    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54bc4a76-r82bj\" (UID: \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"
Nov 03 02:37:33.855171 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.855112    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8t4j\" (UniqueName: \"kubernetes.io/projected/ba64a7a8-9c90-4c77-ac37-57a6904c5549-kube-api-access-l8t4j\") pod \"maas-api-revoke-sub-54bc4a76-r82bj\" (UID: \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"
Nov 03 02:37:33.962639 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.962595    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54bc4a76-r82bj\" (UID: \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"
Nov 03 02:37:33.975516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.975478    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8t4j\" (UniqueName: \"kubernetes.io/projected/ba64a7a8-9c90-4c77-ac37-57a6904c5549-kube-api-access-l8t4j\") pod \"maas-api-revoke-sub-54bc4a76-r82bj\" (UID: \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"
Nov 03 02:37:33.976047 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.976016    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54bc4a76-r82bj\" (UID: \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"
Nov 03 02:37:33.976269 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.976241    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54bc4a76-r82bj\" (UID: \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"
Nov 03 02:37:33.986308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:33.976985    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54bc4a76-r82bj\" (UID: \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"
Nov 03 02:37:34.040336 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:34.040259    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8t4j\" (UniqueName: \"kubernetes.io/projected/ba64a7a8-9c90-4c77-ac37-57a6904c5549-kube-api-access-l8t4j\") pod \"maas-api-revoke-sub-54bc4a76-r82bj\" (UID: \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"
Nov 03 02:37:34.070728 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:34.070689    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"
Nov 03 02:37:34.452323 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:34.452289    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"]
Nov 03 02:37:34.460915 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:37:34.460886    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba64a7a8_9c90_4c77_ac37_57a6904c5549.slice/crio-136c3e4acbdd1da35871b518e479f767da4e8d7d2f3e60421d2ee96e119e4949 WatchSource:0}: Error finding container 136c3e4acbdd1da35871b518e479f767da4e8d7d2f3e60421d2ee96e119e4949: Status 404 returned error can't find the container with id 136c3e4acbdd1da35871b518e479f767da4e8d7d2f3e60421d2ee96e119e4949
Nov 03 02:37:34.781953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:34.781911    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj" event={"ID":"ba64a7a8-9c90-4c77-ac37-57a6904c5549","Type":"ContainerStarted","Data":"cbaa63ccc742c056245d9bf3012ec3c6970bbfd690b88c49cbe4242078f3a570"}
Nov 03 02:37:34.782396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:34.781983    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj" event={"ID":"ba64a7a8-9c90-4c77-ac37-57a6904c5549","Type":"ContainerStarted","Data":"136c3e4acbdd1da35871b518e479f767da4e8d7d2f3e60421d2ee96e119e4949"}
Nov 03 02:37:34.892628 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:34.892572    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj" podStartSLOduration=1.892556833 podStartE2EDuration="1.892556833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:37:34.891341661 +0000 UTC m=+4734.316948128" watchObservedRunningTime="2026-11-03 02:37:34.892556833 +0000 UTC m=+4734.318163349"
Nov 03 02:37:35.790001 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:35.789962    5508 generic.go:356] "Generic (PLEG): container finished" podID="ba64a7a8-9c90-4c77-ac37-57a6904c5549" containerID="cbaa63ccc742c056245d9bf3012ec3c6970bbfd690b88c49cbe4242078f3a570" exitCode=0
Nov 03 02:37:35.790437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:35.790053    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj" event={"ID":"ba64a7a8-9c90-4c77-ac37-57a6904c5549","Type":"ContainerDied","Data":"cbaa63ccc742c056245d9bf3012ec3c6970bbfd690b88c49cbe4242078f3a570"}
Nov 03 02:37:35.940519 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:35.940473    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz" podUID="f18a2aed-f7a2-4326-810f-56a0b97d114c" containerName="payload-pre-processing" containerID="cri-o://f808cc119c932166db155f8d5d0f0120e50bcbff4cb27bc4cc65b5651caa7ab7" gracePeriod=25
Nov 03 02:37:36.216685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.216656    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"
Nov 03 02:37:36.313806 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.313723    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f18a2aed-f7a2-4326-810f-56a0b97d114c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18a2aed-f7a2-4326-810f-56a0b97d114c-plugins-config-volume\") pod \"f18a2aed-f7a2-4326-810f-56a0b97d114c\" (UID: \"f18a2aed-f7a2-4326-810f-56a0b97d114c\") "
Nov 03 02:37:36.313967 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.313915    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f18a2aed-f7a2-4326-810f-56a0b97d114c-kube-api-access-l4rt4\" (UniqueName: \"kubernetes.io/projected/f18a2aed-f7a2-4326-810f-56a0b97d114c-kube-api-access-l4rt4\") pod \"f18a2aed-f7a2-4326-810f-56a0b97d114c\" (UID: \"f18a2aed-f7a2-4326-810f-56a0b97d114c\") "
Nov 03 02:37:36.314186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.314164    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18a2aed-f7a2-4326-810f-56a0b97d114c-plugins-config-volume" pod "f18a2aed-f7a2-4326-810f-56a0b97d114c" (UID: "f18a2aed-f7a2-4326-810f-56a0b97d114c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:37:36.314447 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.314424    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18a2aed-f7a2-4326-810f-56a0b97d114c-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:37:36.316022 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.315996    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18a2aed-f7a2-4326-810f-56a0b97d114c-kube-api-access-l4rt4" pod "f18a2aed-f7a2-4326-810f-56a0b97d114c" (UID: "f18a2aed-f7a2-4326-810f-56a0b97d114c"). InnerVolumeSpecName "kube-api-access-l4rt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:37:36.416019 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.415977    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4rt4\" (UniqueName: \"kubernetes.io/projected/f18a2aed-f7a2-4326-810f-56a0b97d114c-kube-api-access-l4rt4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:37:36.797903 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.797864    5508 generic.go:356] "Generic (PLEG): container finished" podID="f18a2aed-f7a2-4326-810f-56a0b97d114c" containerID="f808cc119c932166db155f8d5d0f0120e50bcbff4cb27bc4cc65b5651caa7ab7" exitCode=0
Nov 03 02:37:36.798442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.797940    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"
Nov 03 02:37:36.798442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.797952    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz" event={"ID":"f18a2aed-f7a2-4326-810f-56a0b97d114c","Type":"ContainerDied","Data":"f808cc119c932166db155f8d5d0f0120e50bcbff4cb27bc4cc65b5651caa7ab7"}
Nov 03 02:37:36.798442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.798022    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz" event={"ID":"f18a2aed-f7a2-4326-810f-56a0b97d114c","Type":"ContainerDied","Data":"0e71f973863ea3b24fbf999e3f527fa4f45b4ceed56d2fa3cf94214ea6d3ffe8"}
Nov 03 02:37:36.798442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.798044    5508 scope.go:122] "RemoveContainer" containerID="f808cc119c932166db155f8d5d0f0120e50bcbff4cb27bc4cc65b5651caa7ab7"
Nov 03 02:37:36.810183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.810153    5508 scope.go:122] "RemoveContainer" containerID="f808cc119c932166db155f8d5d0f0120e50bcbff4cb27bc4cc65b5651caa7ab7"
Nov 03 02:37:36.810525 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:37:36.810501    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f808cc119c932166db155f8d5d0f0120e50bcbff4cb27bc4cc65b5651caa7ab7\": container with ID starting with f808cc119c932166db155f8d5d0f0120e50bcbff4cb27bc4cc65b5651caa7ab7 not found: ID does not exist" containerID="f808cc119c932166db155f8d5d0f0120e50bcbff4cb27bc4cc65b5651caa7ab7"
Nov 03 02:37:36.810607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.810538    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f808cc119c932166db155f8d5d0f0120e50bcbff4cb27bc4cc65b5651caa7ab7"} err="failed to get container status \"f808cc119c932166db155f8d5d0f0120e50bcbff4cb27bc4cc65b5651caa7ab7\": rpc error: code = NotFound desc = could not find container \"f808cc119c932166db155f8d5d0f0120e50bcbff4cb27bc4cc65b5651caa7ab7\": container with ID starting with f808cc119c932166db155f8d5d0f0120e50bcbff4cb27bc4cc65b5651caa7ab7 not found: ID does not exist"
Nov 03 02:37:36.955796 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.955721    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"]
Nov 03 02:37:36.964565 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:36.964539    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"
Nov 03 02:37:37.002145 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:37.002114    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-eec2ff70-fbb6544c5-xvfnz"]
Nov 03 02:37:37.023550 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:37.023520    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-openshift-ca\") pod \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\" (UID: \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\") "
Nov 03 02:37:37.023738 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:37.023566    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-xks-ca\") pod \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\" (UID: \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\") "
Nov 03 02:37:37.023799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:37.023762    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba64a7a8-9c90-4c77-ac37-57a6904c5549-kube-api-access-l8t4j\" (UniqueName: \"kubernetes.io/projected/ba64a7a8-9c90-4c77-ac37-57a6904c5549-kube-api-access-l8t4j\") pod \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\" (UID: \"ba64a7a8-9c90-4c77-ac37-57a6904c5549\") "
Nov 03 02:37:37.023908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:37.023881    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-openshift-ca" pod "ba64a7a8-9c90-4c77-ac37-57a6904c5549" (UID: "ba64a7a8-9c90-4c77-ac37-57a6904c5549"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:37:37.024516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:37.024492    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:37:37.025718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:37.025689    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-xks-ca" pod "ba64a7a8-9c90-4c77-ac37-57a6904c5549" (UID: "ba64a7a8-9c90-4c77-ac37-57a6904c5549"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:37:37.025820 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:37.025803    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba64a7a8-9c90-4c77-ac37-57a6904c5549-kube-api-access-l8t4j" pod "ba64a7a8-9c90-4c77-ac37-57a6904c5549" (UID: "ba64a7a8-9c90-4c77-ac37-57a6904c5549"). InnerVolumeSpecName "kube-api-access-l8t4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:37:37.126274 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:37.126233    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba64a7a8-9c90-4c77-ac37-57a6904c5549-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:37:37.126274 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:37.126267    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8t4j\" (UniqueName: \"kubernetes.io/projected/ba64a7a8-9c90-4c77-ac37-57a6904c5549-kube-api-access-l8t4j\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:37:37.806438 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:37.806411    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"
Nov 03 02:37:37.806438 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:37.806429    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj" event={"ID":"ba64a7a8-9c90-4c77-ac37-57a6904c5549","Type":"ContainerDied","Data":"136c3e4acbdd1da35871b518e479f767da4e8d7d2f3e60421d2ee96e119e4949"}
Nov 03 02:37:37.806966 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:37.806490    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="136c3e4acbdd1da35871b518e479f767da4e8d7d2f3e60421d2ee96e119e4949"
Nov 03 02:37:38.693720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:38.693683    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f18a2aed-f7a2-4326-810f-56a0b97d114c" path="/var/lib/kubelet/pods/f18a2aed-f7a2-4326-810f-56a0b97d114c/volumes"
Nov 03 02:37:43.813530 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:37:43.813479    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:37:46.496479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:46.496446    5508 scope.go:122] "RemoveContainer" containerID="2e22bb0083b83ca727e0a1bef9f01b0c73edbfab123cae65699a370634d04130"
Nov 03 02:37:59.097018 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.096981    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"]
Nov 03 02:37:59.097599 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.097200    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw" podUID="c50fad98-b746-40f6-853e-887f2da38314" containerName="maas-api" containerID="cri-o://93aad97c433cc0c1d7219580f0b94561996ad59946a8aaa02ddc3fc15671b99a" gracePeriod=30
Nov 03 02:37:59.435638 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.435610    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:37:59.576411 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.553418    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50fad98-b746-40f6-853e-887f2da38314-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50fad98-b746-40f6-853e-887f2da38314-openshift-service-ca\") pod \"c50fad98-b746-40f6-853e-887f2da38314\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") "
Nov 03 02:37:59.577158 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.560150    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50fad98-b746-40f6-853e-887f2da38314-openshift-service-ca" pod "c50fad98-b746-40f6-853e-887f2da38314" (UID: "c50fad98-b746-40f6-853e-887f2da38314"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:37:59.577439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.577417    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-metrics-certs\") pod \"c50fad98-b746-40f6-853e-887f2da38314\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") "
Nov 03 02:37:59.577631 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.577616    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50fad98-b746-40f6-853e-887f2da38314-kube-api-access-vxnh7\" (UniqueName: \"kubernetes.io/projected/c50fad98-b746-40f6-853e-887f2da38314-kube-api-access-vxnh7\") pod \"c50fad98-b746-40f6-853e-887f2da38314\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") "
Nov 03 02:37:59.577831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.577815    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-maas-api-tls\") pod \"c50fad98-b746-40f6-853e-887f2da38314\" (UID: \"c50fad98-b746-40f6-853e-887f2da38314\") "
Nov 03 02:37:59.578844 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.578822    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50fad98-b746-40f6-853e-887f2da38314-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:37:59.585793 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.585764    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-maas-api-tls" pod "c50fad98-b746-40f6-853e-887f2da38314" (UID: "c50fad98-b746-40f6-853e-887f2da38314"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:37:59.594452 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.594415    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-metrics-certs" pod "c50fad98-b746-40f6-853e-887f2da38314" (UID: "c50fad98-b746-40f6-853e-887f2da38314"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:37:59.595621 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.595569    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50fad98-b746-40f6-853e-887f2da38314-kube-api-access-vxnh7" pod "c50fad98-b746-40f6-853e-887f2da38314" (UID: "c50fad98-b746-40f6-853e-887f2da38314"). InnerVolumeSpecName "kube-api-access-vxnh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:37:59.683435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.680434    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:37:59.683435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.680469    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c50fad98-b746-40f6-853e-887f2da38314-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:37:59.683435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.680485    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxnh7\" (UniqueName: \"kubernetes.io/projected/c50fad98-b746-40f6-853e-887f2da38314-kube-api-access-vxnh7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:37:59.968479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.968384    5508 generic.go:356] "Generic (PLEG): container finished" podID="c50fad98-b746-40f6-853e-887f2da38314" containerID="93aad97c433cc0c1d7219580f0b94561996ad59946a8aaa02ddc3fc15671b99a" exitCode=0
Nov 03 02:37:59.968479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.968435    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"
Nov 03 02:37:59.968722 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.968465    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw" event={"ID":"c50fad98-b746-40f6-853e-887f2da38314","Type":"ContainerDied","Data":"93aad97c433cc0c1d7219580f0b94561996ad59946a8aaa02ddc3fc15671b99a"}
Nov 03 02:37:59.968722 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.968535    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw" event={"ID":"c50fad98-b746-40f6-853e-887f2da38314","Type":"ContainerDied","Data":"457e2fc63e476068de910eb4f86bc1df38d304f657da93387862135ebd73a071"}
Nov 03 02:37:59.968722 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.968565    5508 scope.go:122] "RemoveContainer" containerID="93aad97c433cc0c1d7219580f0b94561996ad59946a8aaa02ddc3fc15671b99a"
Nov 03 02:37:59.981135 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.981113    5508 scope.go:122] "RemoveContainer" containerID="93aad97c433cc0c1d7219580f0b94561996ad59946a8aaa02ddc3fc15671b99a"
Nov 03 02:37:59.981486 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:37:59.981464    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93aad97c433cc0c1d7219580f0b94561996ad59946a8aaa02ddc3fc15671b99a\": container with ID starting with 93aad97c433cc0c1d7219580f0b94561996ad59946a8aaa02ddc3fc15671b99a not found: ID does not exist" containerID="93aad97c433cc0c1d7219580f0b94561996ad59946a8aaa02ddc3fc15671b99a"
Nov 03 02:37:59.981571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:37:59.981495    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93aad97c433cc0c1d7219580f0b94561996ad59946a8aaa02ddc3fc15671b99a"} err="failed to get container status \"93aad97c433cc0c1d7219580f0b94561996ad59946a8aaa02ddc3fc15671b99a\": rpc error: code = NotFound desc = could not find container \"93aad97c433cc0c1d7219580f0b94561996ad59946a8aaa02ddc3fc15671b99a\": container with ID starting with 93aad97c433cc0c1d7219580f0b94561996ad59946a8aaa02ddc3fc15671b99a not found: ID does not exist"
Nov 03 02:38:00.148418 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:00.135086    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"]
Nov 03 02:38:00.148418 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:00.144071    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-eec2ff70-5f7f75f764-brpbw"]
Nov 03 02:38:00.692042 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:00.692010    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50fad98-b746-40f6-853e-887f2da38314" path="/var/lib/kubelet/pods/c50fad98-b746-40f6-853e-887f2da38314/volumes"
Nov 03 02:38:01.628847 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:01.628808    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:38:06.294439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.294400    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"]
Nov 03 02:38:06.295057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.295035    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c50fad98-b746-40f6-853e-887f2da38314" containerName="maas-api"
Nov 03 02:38:06.295211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.295131    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50fad98-b746-40f6-853e-887f2da38314" containerName="maas-api"
Nov 03 02:38:06.295211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.295187    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f18a2aed-f7a2-4326-810f-56a0b97d114c" containerName="payload-pre-processing"
Nov 03 02:38:06.295211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.295195    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f18a2aed-f7a2-4326-810f-56a0b97d114c" containerName="payload-pre-processing"
Nov 03 02:38:06.295418 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.295264    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ba64a7a8-9c90-4c77-ac37-57a6904c5549" containerName="revoke-keys"
Nov 03 02:38:06.295418 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.295273    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba64a7a8-9c90-4c77-ac37-57a6904c5549" containerName="revoke-keys"
Nov 03 02:38:06.295531 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.295467    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba64a7a8-9c90-4c77-ac37-57a6904c5549" containerName="revoke-keys"
Nov 03 02:38:06.295531 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.295485    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50fad98-b746-40f6-853e-887f2da38314" containerName="maas-api"
Nov 03 02:38:06.295531 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.295502    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f18a2aed-f7a2-4326-810f-56a0b97d114c" containerName="payload-pre-processing"
Nov 03 02:38:06.301085 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.301063    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"
Nov 03 02:38:06.351922 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.351889    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:38:06.398262 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.398229    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:38:06.403675 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.403646    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"]
Nov 03 02:38:06.404938 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.404917    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52a12add-g4t86\" (UID: \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"
Nov 03 02:38:06.405027 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.404958    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52a12add-g4t86\" (UID: \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"
Nov 03 02:38:06.405223 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.405200    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w25ss\" (UniqueName: \"kubernetes.io/projected/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-kube-api-access-w25ss\") pod \"maas-api-revoke-sub-52a12add-g4t86\" (UID: \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"
Nov 03 02:38:06.506581 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.506546    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52a12add-g4t86\" (UID: \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"
Nov 03 02:38:06.506808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.506595    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52a12add-g4t86\" (UID: \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"
Nov 03 02:38:06.506808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.506737    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w25ss\" (UniqueName: \"kubernetes.io/projected/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-kube-api-access-w25ss\") pod \"maas-api-revoke-sub-52a12add-g4t86\" (UID: \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"
Nov 03 02:38:06.507277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.507250    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52a12add-g4t86\" (UID: \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"
Nov 03 02:38:06.509088 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.509065    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52a12add-g4t86\" (UID: \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"
Nov 03 02:38:06.564686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.564614    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w25ss\" (UniqueName: \"kubernetes.io/projected/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-kube-api-access-w25ss\") pod \"maas-api-revoke-sub-52a12add-g4t86\" (UID: \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"
Nov 03 02:38:06.618525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:06.618488    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"
Nov 03 02:38:07.007147 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:07.007110    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"]
Nov 03 02:38:07.039275 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:38:07.039231    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb3932d_77a4_4cd8_9d06_0de4dcc0ddc3.slice/crio-97d778fb74068ede1c73dfc4ae79d8c692535c9d8943de264808d3f49c28fb2c WatchSource:0}: Error finding container 97d778fb74068ede1c73dfc4ae79d8c692535c9d8943de264808d3f49c28fb2c: Status 404 returned error can't find the container with id 97d778fb74068ede1c73dfc4ae79d8c692535c9d8943de264808d3f49c28fb2c
Nov 03 02:38:07.057603 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:07.057573    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86" event={"ID":"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3","Type":"ContainerStarted","Data":"97d778fb74068ede1c73dfc4ae79d8c692535c9d8943de264808d3f49c28fb2c"}
Nov 03 02:38:08.076224 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:08.076191    5508 generic.go:356] "Generic (PLEG): container finished" podID="ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3" containerID="ed3745b78c86e7893d4418adeac297b632b4635665ac3611202e0c6bd915b690" exitCode=0
Nov 03 02:38:08.076642 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:08.076278    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86" event={"ID":"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3","Type":"ContainerDied","Data":"ed3745b78c86e7893d4418adeac297b632b4635665ac3611202e0c6bd915b690"}
Nov 03 02:38:09.118401 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:09.118363    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"]
Nov 03 02:38:09.118811 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:09.118673    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerName="istio-proxy" containerID="cri-o://f3004218e39a35d4d01bba3c254ec3ec6462d058f796dccfe1021fb5553b2813" gracePeriod=30
Nov 03 02:38:09.372379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:09.367076    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"
Nov 03 02:38:09.470018 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:09.469983    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-kube-api-access-w25ss\" (UniqueName: \"kubernetes.io/projected/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-kube-api-access-w25ss\") pod \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\" (UID: \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\") "
Nov 03 02:38:09.470237 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:09.470030    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-openshift-ca\") pod \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\" (UID: \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\") "
Nov 03 02:38:09.470237 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:09.470175    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-xks-ca\") pod \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\" (UID: \"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3\") "
Nov 03 02:38:09.470516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:09.470483    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-openshift-ca" pod "ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3" (UID: "ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:38:09.470735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:09.470714    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:09.472242 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:09.472215    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-xks-ca" pod "ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3" (UID: "ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:38:09.472367 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:09.472306    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-kube-api-access-w25ss" pod "ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3" (UID: "ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3"). InnerVolumeSpecName "kube-api-access-w25ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:38:09.572192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:09.572140    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w25ss\" (UniqueName: \"kubernetes.io/projected/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-kube-api-access-w25ss\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:09.572192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:09.572192    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:10.091547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:10.091521    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"
Nov 03 02:38:10.091817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:10.091552    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86" event={"ID":"ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3","Type":"ContainerDied","Data":"97d778fb74068ede1c73dfc4ae79d8c692535c9d8943de264808d3f49c28fb2c"}
Nov 03 02:38:10.091817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:10.091600    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97d778fb74068ede1c73dfc4ae79d8c692535c9d8943de264808d3f49c28fb2c"
Nov 03 02:38:13.812532 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:38:13.812492    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:38:14.393740 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.393665    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:38:14.549289 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.549252    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-envoy\") pod \"a199c50f-4280-402e-b6cd-9433e0639b7a\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") "
Nov 03 02:38:14.549289 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.549293    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-certs\") pod \"a199c50f-4280-402e-b6cd-9433e0639b7a\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") "
Nov 03 02:38:14.549630 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.549329    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-podinfo\") pod \"a199c50f-4280-402e-b6cd-9433e0639b7a\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") "
Nov 03 02:38:14.549630 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.549400    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-data\") pod \"a199c50f-4280-402e-b6cd-9433e0639b7a\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") "
Nov 03 02:38:14.549630 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.549513    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-kube-api-access-wddkt\" (UniqueName: \"kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-kube-api-access-wddkt\") pod \"a199c50f-4280-402e-b6cd-9433e0639b7a\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") "
Nov 03 02:38:14.549818 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.549630    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-socket\") pod \"a199c50f-4280-402e-b6cd-9433e0639b7a\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") "
Nov 03 02:38:14.549818 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.549661    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-token\" (UniqueName: \"kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-token\") pod \"a199c50f-4280-402e-b6cd-9433e0639b7a\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") "
Nov 03 02:38:14.549818 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.549659    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-certs" pod "a199c50f-4280-402e-b6cd-9433e0639b7a" (UID: "a199c50f-4280-402e-b6cd-9433e0639b7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:38:14.549818 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.549698    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a199c50f-4280-402e-b6cd-9433e0639b7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a199c50f-4280-402e-b6cd-9433e0639b7a-istiod-ca-cert\") pod \"a199c50f-4280-402e-b6cd-9433e0639b7a\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") "
Nov 03 02:38:14.550056 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.549819    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-credential-socket\") pod \"a199c50f-4280-402e-b6cd-9433e0639b7a\" (UID: \"a199c50f-4280-402e-b6cd-9433e0639b7a\") "
Nov 03 02:38:14.550056 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.549896    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-socket" pod "a199c50f-4280-402e-b6cd-9433e0639b7a" (UID: "a199c50f-4280-402e-b6cd-9433e0639b7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:38:14.550272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.550202    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a199c50f-4280-402e-b6cd-9433e0639b7a-istiod-ca-cert" pod "a199c50f-4280-402e-b6cd-9433e0639b7a" (UID: "a199c50f-4280-402e-b6cd-9433e0639b7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:38:14.550272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.550207    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-credential-socket" pod "a199c50f-4280-402e-b6cd-9433e0639b7a" (UID: "a199c50f-4280-402e-b6cd-9433e0639b7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:38:14.550272 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.550236    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-data" pod "a199c50f-4280-402e-b6cd-9433e0639b7a" (UID: "a199c50f-4280-402e-b6cd-9433e0639b7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:38:14.550553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.550429    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:14.550553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.550448    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:14.550553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.550462    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:14.550553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.550476    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:14.550553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.550489    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a199c50f-4280-402e-b6cd-9433e0639b7a-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:14.551810 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.551785    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-podinfo" pod "a199c50f-4280-402e-b6cd-9433e0639b7a" (UID: "a199c50f-4280-402e-b6cd-9433e0639b7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:38:14.552044 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.552025    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-envoy" pod "a199c50f-4280-402e-b6cd-9433e0639b7a" (UID: "a199c50f-4280-402e-b6cd-9433e0639b7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:38:14.552174 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.552087    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-kube-api-access-wddkt" pod "a199c50f-4280-402e-b6cd-9433e0639b7a" (UID: "a199c50f-4280-402e-b6cd-9433e0639b7a"). InnerVolumeSpecName "kube-api-access-wddkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:38:14.552412 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.552390    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-token" pod "a199c50f-4280-402e-b6cd-9433e0639b7a" (UID: "a199c50f-4280-402e-b6cd-9433e0639b7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:38:14.651577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.651536    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wddkt\" (UniqueName: \"kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-kube-api-access-wddkt\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:14.651577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.651573    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:14.651577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.651583    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:14.651914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:14.651592    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a199c50f-4280-402e-b6cd-9433e0639b7a-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:15.130573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:15.130482    5508 generic.go:356] "Generic (PLEG): container finished" podID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerID="f3004218e39a35d4d01bba3c254ec3ec6462d058f796dccfe1021fb5553b2813" exitCode=0
Nov 03 02:38:15.130573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:15.130537    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"
Nov 03 02:38:15.130573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:15.130542    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" event={"ID":"a199c50f-4280-402e-b6cd-9433e0639b7a","Type":"ContainerDied","Data":"f3004218e39a35d4d01bba3c254ec3ec6462d058f796dccfe1021fb5553b2813"}
Nov 03 02:38:15.131177 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:15.130603    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj" event={"ID":"a199c50f-4280-402e-b6cd-9433e0639b7a","Type":"ContainerDied","Data":"87e757b20ac67cbf2cbd706279693e026f9707f4be88cbda99421d66ba396848"}
Nov 03 02:38:15.131177 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:15.130625    5508 scope.go:122] "RemoveContainer" containerID="f3004218e39a35d4d01bba3c254ec3ec6462d058f796dccfe1021fb5553b2813"
Nov 03 02:38:15.141873 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:15.141850    5508 scope.go:122] "RemoveContainer" containerID="f2371369c17d6c2eef4ad0a6b48e0ff0ec4ece40a3a30f4339c1a4c8a204ccb0"
Nov 03 02:38:15.151513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:15.151491    5508 scope.go:122] "RemoveContainer" containerID="f3004218e39a35d4d01bba3c254ec3ec6462d058f796dccfe1021fb5553b2813"
Nov 03 02:38:15.151783 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:38:15.151763    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3004218e39a35d4d01bba3c254ec3ec6462d058f796dccfe1021fb5553b2813\": container with ID starting with f3004218e39a35d4d01bba3c254ec3ec6462d058f796dccfe1021fb5553b2813 not found: ID does not exist" containerID="f3004218e39a35d4d01bba3c254ec3ec6462d058f796dccfe1021fb5553b2813"
Nov 03 02:38:15.151835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:15.151794    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3004218e39a35d4d01bba3c254ec3ec6462d058f796dccfe1021fb5553b2813"} err="failed to get container status \"f3004218e39a35d4d01bba3c254ec3ec6462d058f796dccfe1021fb5553b2813\": rpc error: code = NotFound desc = could not find container \"f3004218e39a35d4d01bba3c254ec3ec6462d058f796dccfe1021fb5553b2813\": container with ID starting with f3004218e39a35d4d01bba3c254ec3ec6462d058f796dccfe1021fb5553b2813 not found: ID does not exist"
Nov 03 02:38:15.151835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:15.151811    5508 scope.go:122] "RemoveContainer" containerID="f2371369c17d6c2eef4ad0a6b48e0ff0ec4ece40a3a30f4339c1a4c8a204ccb0"
Nov 03 02:38:15.152052 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:38:15.152034    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2371369c17d6c2eef4ad0a6b48e0ff0ec4ece40a3a30f4339c1a4c8a204ccb0\": container with ID starting with f2371369c17d6c2eef4ad0a6b48e0ff0ec4ece40a3a30f4339c1a4c8a204ccb0 not found: ID does not exist" containerID="f2371369c17d6c2eef4ad0a6b48e0ff0ec4ece40a3a30f4339c1a4c8a204ccb0"
Nov 03 02:38:15.152099 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:15.152060    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2371369c17d6c2eef4ad0a6b48e0ff0ec4ece40a3a30f4339c1a4c8a204ccb0"} err="failed to get container status \"f2371369c17d6c2eef4ad0a6b48e0ff0ec4ece40a3a30f4339c1a4c8a204ccb0\": rpc error: code = NotFound desc = could not find container \"f2371369c17d6c2eef4ad0a6b48e0ff0ec4ece40a3a30f4339c1a4c8a204ccb0\": container with ID starting with f2371369c17d6c2eef4ad0a6b48e0ff0ec4ece40a3a30f4339c1a4c8a204ccb0 not found: ID does not exist"
Nov 03 02:38:15.286017 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:15.285981    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"]
Nov 03 02:38:15.290038 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:15.290005    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-eec2ff70-openshift-default-668fc5b565-mpdqj"]
Nov 03 02:38:16.631956 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:16.631863    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"]
Nov 03 02:38:16.694400 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:16.694343    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" path="/var/lib/kubelet/pods/a199c50f-4280-402e-b6cd-9433e0639b7a/volumes"
Nov 03 02:38:17.017052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.017018    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"]
Nov 03 02:38:17.017643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.017614    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerName="istio-proxy"
Nov 03 02:38:17.017643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.017633    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerName="istio-proxy"
Nov 03 02:38:17.017838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.017680    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerName="istio-proxy"
Nov 03 02:38:17.017838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.017689    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerName="istio-proxy"
Nov 03 02:38:17.017838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.017705    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3" containerName="revoke-keys"
Nov 03 02:38:17.017838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.017712    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3" containerName="revoke-keys"
Nov 03 02:38:17.017984 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.017864    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerName="istio-proxy"
Nov 03 02:38:17.017984 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.017879    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3" containerName="revoke-keys"
Nov 03 02:38:17.022923 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.022901    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"
Nov 03 02:38:17.082398 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.082339    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:38:17.082849 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.082833    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:38:17.094548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.094510    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"]
Nov 03 02:38:17.190282 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.190244    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d64720-gd84b\" (UID: \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"
Nov 03 02:38:17.190492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.190315    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d64720-gd84b\" (UID: \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"
Nov 03 02:38:17.190492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.190458    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdh5b\" (UniqueName: \"kubernetes.io/projected/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-kube-api-access-zdh5b\") pod \"maas-api-revoke-sub-59d64720-gd84b\" (UID: \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"
Nov 03 02:38:17.292312 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.292204    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d64720-gd84b\" (UID: \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"
Nov 03 02:38:17.292312 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.292309    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdh5b\" (UniqueName: \"kubernetes.io/projected/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-kube-api-access-zdh5b\") pod \"maas-api-revoke-sub-59d64720-gd84b\" (UID: \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"
Nov 03 02:38:17.292593 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.292505    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d64720-gd84b\" (UID: \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"
Nov 03 02:38:17.293121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.293091    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59d64720-gd84b\" (UID: \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"
Nov 03 02:38:17.295162 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.295135    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59d64720-gd84b\" (UID: \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"
Nov 03 02:38:17.349640 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.349605    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdh5b\" (UniqueName: \"kubernetes.io/projected/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-kube-api-access-zdh5b\") pod \"maas-api-revoke-sub-59d64720-gd84b\" (UID: \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"
Nov 03 02:38:17.645390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:17.645293    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"
Nov 03 02:38:18.042000 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:18.041965    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"]
Nov 03 02:38:18.049711 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:38:18.049679    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb719a8_4f64_4414_8cb4_e94c97c2c66e.slice/crio-e3735a236161c699f463b5753ab3125ad2d5e2895bb296cdc32e98354ca58d2d WatchSource:0}: Error finding container e3735a236161c699f463b5753ab3125ad2d5e2895bb296cdc32e98354ca58d2d: Status 404 returned error can't find the container with id e3735a236161c699f463b5753ab3125ad2d5e2895bb296cdc32e98354ca58d2d
Nov 03 02:38:18.180095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:18.180055    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b" event={"ID":"7eb719a8-4f64-4414-8cb4-e94c97c2c66e","Type":"ContainerStarted","Data":"e3735a236161c699f463b5753ab3125ad2d5e2895bb296cdc32e98354ca58d2d"}
Nov 03 02:38:19.192903 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:19.192863    5508 generic.go:356] "Generic (PLEG): container finished" podID="7eb719a8-4f64-4414-8cb4-e94c97c2c66e" containerID="e9ef46d4da057c34a85d2f568bed8c49d3d40eff75501f4faba5de6cfb62ef3a" exitCode=0
Nov 03 02:38:19.193404 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:19.193006    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b" event={"ID":"7eb719a8-4f64-4414-8cb4-e94c97c2c66e","Type":"ContainerDied","Data":"e9ef46d4da057c34a85d2f568bed8c49d3d40eff75501f4faba5de6cfb62ef3a"}
Nov 03 02:38:20.418064 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:20.418033    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"
Nov 03 02:38:20.559758 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:20.559653    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-kube-api-access-zdh5b\" (UniqueName: \"kubernetes.io/projected/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-kube-api-access-zdh5b\") pod \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\" (UID: \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\") "
Nov 03 02:38:20.559959 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:20.559759    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-openshift-ca\") pod \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\" (UID: \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\") "
Nov 03 02:38:20.559959 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:20.559818    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-xks-ca\") pod \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\" (UID: \"7eb719a8-4f64-4414-8cb4-e94c97c2c66e\") "
Nov 03 02:38:20.560838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:20.560812    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-openshift-ca" pod "7eb719a8-4f64-4414-8cb4-e94c97c2c66e" (UID: "7eb719a8-4f64-4414-8cb4-e94c97c2c66e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:38:20.572379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:20.563474    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-kube-api-access-zdh5b" pod "7eb719a8-4f64-4414-8cb4-e94c97c2c66e" (UID: "7eb719a8-4f64-4414-8cb4-e94c97c2c66e"). InnerVolumeSpecName "kube-api-access-zdh5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:38:20.572379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:20.569033    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-xks-ca" pod "7eb719a8-4f64-4414-8cb4-e94c97c2c66e" (UID: "7eb719a8-4f64-4414-8cb4-e94c97c2c66e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:38:20.662225 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:20.662182    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:20.662467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:20.662225    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:20.662467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:20.662256    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdh5b\" (UniqueName: \"kubernetes.io/projected/7eb719a8-4f64-4414-8cb4-e94c97c2c66e-kube-api-access-zdh5b\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:21.208109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:21.208076    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"
Nov 03 02:38:21.208109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:21.208101    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b" event={"ID":"7eb719a8-4f64-4414-8cb4-e94c97c2c66e","Type":"ContainerDied","Data":"e3735a236161c699f463b5753ab3125ad2d5e2895bb296cdc32e98354ca58d2d"}
Nov 03 02:38:21.208455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:21.208161    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3735a236161c699f463b5753ab3125ad2d5e2895bb296cdc32e98354ca58d2d"
Nov 03 02:38:21.632223 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:21.632123    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4" podUID="acb627be-2d98-4b57-86e6-99ed6ff1e3b2" containerName="payload-pre-processing" containerID="cri-o://453c243334650e5112c2d23a36196c481903de801f5d87f0aef7545584988541" gracePeriod=25
Nov 03 02:38:26.717735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:26.717699    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4" podUID="acb627be-2d98-4b57-86e6-99ed6ff1e3b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:38:36.718263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:36.718226    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4" podUID="acb627be-2d98-4b57-86e6-99ed6ff1e3b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:38:41.621434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:41.621403    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:38:41.748339 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:41.748314    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:38:41.911116 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:41.911075    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"
Nov 03 02:38:42.050588 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.050552    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-kube-api-access-ph2gt\" (UniqueName: \"kubernetes.io/projected/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-kube-api-access-ph2gt\") pod \"acb627be-2d98-4b57-86e6-99ed6ff1e3b2\" (UID: \"acb627be-2d98-4b57-86e6-99ed6ff1e3b2\") "
Nov 03 02:38:42.050758 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.050673    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-plugins-config-volume\") pod \"acb627be-2d98-4b57-86e6-99ed6ff1e3b2\" (UID: \"acb627be-2d98-4b57-86e6-99ed6ff1e3b2\") "
Nov 03 02:38:42.051165 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.051133    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-plugins-config-volume" pod "acb627be-2d98-4b57-86e6-99ed6ff1e3b2" (UID: "acb627be-2d98-4b57-86e6-99ed6ff1e3b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:38:42.053720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.053690    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-kube-api-access-ph2gt" pod "acb627be-2d98-4b57-86e6-99ed6ff1e3b2" (UID: "acb627be-2d98-4b57-86e6-99ed6ff1e3b2"). InnerVolumeSpecName "kube-api-access-ph2gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:38:42.152764 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.152728    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph2gt\" (UniqueName: \"kubernetes.io/projected/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-kube-api-access-ph2gt\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:42.152764 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.152762    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb627be-2d98-4b57-86e6-99ed6ff1e3b2-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:38:42.363795 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.363751    5508 generic.go:356] "Generic (PLEG): container finished" podID="acb627be-2d98-4b57-86e6-99ed6ff1e3b2" containerID="453c243334650e5112c2d23a36196c481903de801f5d87f0aef7545584988541" exitCode=0
Nov 03 02:38:42.363795 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.363796    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"
Nov 03 02:38:42.364034 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.363833    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4" event={"ID":"acb627be-2d98-4b57-86e6-99ed6ff1e3b2","Type":"ContainerDied","Data":"453c243334650e5112c2d23a36196c481903de801f5d87f0aef7545584988541"}
Nov 03 02:38:42.364034 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.363889    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4" event={"ID":"acb627be-2d98-4b57-86e6-99ed6ff1e3b2","Type":"ContainerDied","Data":"7405caedd967ba455c15ae657b3b1a9c9e28f6a4dec2d670248498fa06bb87f6"}
Nov 03 02:38:42.364034 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.363913    5508 scope.go:122] "RemoveContainer" containerID="453c243334650e5112c2d23a36196c481903de801f5d87f0aef7545584988541"
Nov 03 02:38:42.374610 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.374587    5508 scope.go:122] "RemoveContainer" containerID="453c243334650e5112c2d23a36196c481903de801f5d87f0aef7545584988541"
Nov 03 02:38:42.374912 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:38:42.374890    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453c243334650e5112c2d23a36196c481903de801f5d87f0aef7545584988541\": container with ID starting with 453c243334650e5112c2d23a36196c481903de801f5d87f0aef7545584988541 not found: ID does not exist" containerID="453c243334650e5112c2d23a36196c481903de801f5d87f0aef7545584988541"
Nov 03 02:38:42.374961 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.374921    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453c243334650e5112c2d23a36196c481903de801f5d87f0aef7545584988541"} err="failed to get container status \"453c243334650e5112c2d23a36196c481903de801f5d87f0aef7545584988541\": rpc error: code = NotFound desc = could not find container \"453c243334650e5112c2d23a36196c481903de801f5d87f0aef7545584988541\": container with ID starting with 453c243334650e5112c2d23a36196c481903de801f5d87f0aef7545584988541 not found: ID does not exist"
Nov 03 02:38:42.519504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.519407    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"]
Nov 03 02:38:42.565410 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.565343    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-059b23-544965757f-fcwl4"]
Nov 03 02:38:42.693935 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:42.693894    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acb627be-2d98-4b57-86e6-99ed6ff1e3b2" path="/var/lib/kubelet/pods/acb627be-2d98-4b57-86e6-99ed6ff1e3b2/volumes"
Nov 03 02:38:43.813832 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:38:43.813791    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:38:53.889045 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:38:53.889003    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:39:03.370688 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.370653    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"]
Nov 03 02:39:03.371233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.370929    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5" podUID="eb6b9575-4487-4e4d-92c5-55a93baff2f4" containerName="maas-api" containerID="cri-o://c191e5dd171f7f2ccb8c073b57789e824db2421f307a772a80e59f85845eac5a" gracePeriod=30
Nov 03 02:39:03.509332 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.509300    5508 generic.go:356] "Generic (PLEG): container finished" podID="eb6b9575-4487-4e4d-92c5-55a93baff2f4" containerID="c191e5dd171f7f2ccb8c073b57789e824db2421f307a772a80e59f85845eac5a" exitCode=0
Nov 03 02:39:03.509541 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.509385    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5" event={"ID":"eb6b9575-4487-4e4d-92c5-55a93baff2f4","Type":"ContainerDied","Data":"c191e5dd171f7f2ccb8c073b57789e824db2421f307a772a80e59f85845eac5a"}
Nov 03 02:39:03.677392 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.675468    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:39:03.760481 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.760437    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb6b9575-4487-4e4d-92c5-55a93baff2f4-kube-api-access-8q66m\" (UniqueName: \"kubernetes.io/projected/eb6b9575-4487-4e4d-92c5-55a93baff2f4-kube-api-access-8q66m\") pod \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") "
Nov 03 02:39:03.760699 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.760580    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb6b9575-4487-4e4d-92c5-55a93baff2f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6b9575-4487-4e4d-92c5-55a93baff2f4-openshift-service-ca\") pod \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") "
Nov 03 02:39:03.760768 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.760723    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-maas-api-tls\") pod \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") "
Nov 03 02:39:03.760836 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.760784    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-metrics-certs\") pod \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\" (UID: \"eb6b9575-4487-4e4d-92c5-55a93baff2f4\") "
Nov 03 02:39:03.764758 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.764715    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb6b9575-4487-4e4d-92c5-55a93baff2f4-openshift-service-ca" pod "eb6b9575-4487-4e4d-92c5-55a93baff2f4" (UID: "eb6b9575-4487-4e4d-92c5-55a93baff2f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:39:03.770674 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.770632    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb6b9575-4487-4e4d-92c5-55a93baff2f4-kube-api-access-8q66m" pod "eb6b9575-4487-4e4d-92c5-55a93baff2f4" (UID: "eb6b9575-4487-4e4d-92c5-55a93baff2f4"). InnerVolumeSpecName "kube-api-access-8q66m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:39:03.787670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.786576    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-metrics-certs" pod "eb6b9575-4487-4e4d-92c5-55a93baff2f4" (UID: "eb6b9575-4487-4e4d-92c5-55a93baff2f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:39:03.788628 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.788569    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-maas-api-tls" pod "eb6b9575-4487-4e4d-92c5-55a93baff2f4" (UID: "eb6b9575-4487-4e4d-92c5-55a93baff2f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:39:03.862996 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.862958    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6b9575-4487-4e4d-92c5-55a93baff2f4-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:03.863258 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.863240    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:03.863401 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.863386    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6b9575-4487-4e4d-92c5-55a93baff2f4-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:03.863517 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:03.863503    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q66m\" (UniqueName: \"kubernetes.io/projected/eb6b9575-4487-4e4d-92c5-55a93baff2f4-kube-api-access-8q66m\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:04.516889 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:04.516857    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"
Nov 03 02:39:04.516889 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:04.516876    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5" event={"ID":"eb6b9575-4487-4e4d-92c5-55a93baff2f4","Type":"ContainerDied","Data":"b58e3cebfb9b9098cb10622498ac844a37a1673360c53e6554e8478876bece04"}
Nov 03 02:39:04.517465 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:04.516962    5508 scope.go:122] "RemoveContainer" containerID="c191e5dd171f7f2ccb8c073b57789e824db2421f307a772a80e59f85845eac5a"
Nov 03 02:39:04.673664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:04.673618    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"]
Nov 03 02:39:04.718134 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:04.718096    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-059b23-6d46cf684b-m58f5"]
Nov 03 02:39:06.695584 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:06.695545    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb6b9575-4487-4e4d-92c5-55a93baff2f4" path="/var/lib/kubelet/pods/eb6b9575-4487-4e4d-92c5-55a93baff2f4/volumes"
Nov 03 02:39:08.419556 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:08.419523    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"]
Nov 03 02:39:08.420023 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:08.419717    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7" podUID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerName="istio-proxy" containerID="cri-o://b93fcc21544e9a20eeca86f6603e518709fdd690407b301a802a6979ab3ed11c" gracePeriod=30
Nov 03 02:39:11.919008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:11.918962    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"]
Nov 03 02:39:11.919637 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:11.919239    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" podUID="9fefda07-12d0-4158-a08d-4e0772428124" containerName="istio-proxy" containerID="cri-o://578a7bfd301a84d7d0c278f49385fcee1abcca9806a00ddf806ced907d4c3db6" gracePeriod=30
Nov 03 02:39:13.628243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.628206    5508 generic.go:356] "Generic (PLEG): container finished" podID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerID="b93fcc21544e9a20eeca86f6603e518709fdd690407b301a802a6979ab3ed11c" exitCode=0
Nov 03 02:39:13.628755 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.628288    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7" event={"ID":"0b83f3c5-8dbb-4187-ae4d-37b745bde034","Type":"ContainerDied","Data":"b93fcc21544e9a20eeca86f6603e518709fdd690407b301a802a6979ab3ed11c"}
Nov 03 02:39:13.628755 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.628387    5508 scope.go:122] "RemoveContainer" containerID="2505c88ff6f85b73aa5244b4732d0d1ad2f39b708a91f63803b2f7f7bd07e778"
Nov 03 02:39:13.683445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.683382    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:39:13.729595 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.729556    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-podinfo\") pod \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") "
Nov 03 02:39:13.729779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.729669    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-token\" (UniqueName: \"kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-token\") pod \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") "
Nov 03 02:39:13.729779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.729692    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-certs\") pod \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") "
Nov 03 02:39:13.729779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.729728    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-data\") pod \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") "
Nov 03 02:39:13.729945 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.729805    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-credential-socket\") pod \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") "
Nov 03 02:39:13.729945 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.729862    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-socket\") pod \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") "
Nov 03 02:39:13.730066 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.729943    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-kube-api-access-vqn78\" (UniqueName: \"kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-kube-api-access-vqn78\") pod \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") "
Nov 03 02:39:13.730066 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.729983    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istiod-ca-cert\") pod \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") "
Nov 03 02:39:13.730066 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.730021    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-envoy\") pod \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\" (UID: \"0b83f3c5-8dbb-4187-ae4d-37b745bde034\") "
Nov 03 02:39:13.730234 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.730076    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-data" pod "0b83f3c5-8dbb-4187-ae4d-37b745bde034" (UID: "0b83f3c5-8dbb-4187-ae4d-37b745bde034"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:13.730234 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.730103    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-certs" pod "0b83f3c5-8dbb-4187-ae4d-37b745bde034" (UID: "0b83f3c5-8dbb-4187-ae4d-37b745bde034"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:13.730234 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.730123    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-credential-socket" pod "0b83f3c5-8dbb-4187-ae4d-37b745bde034" (UID: "0b83f3c5-8dbb-4187-ae4d-37b745bde034"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:13.730434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.730330    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-socket" pod "0b83f3c5-8dbb-4187-ae4d-37b745bde034" (UID: "0b83f3c5-8dbb-4187-ae4d-37b745bde034"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:13.730508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.730446    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istiod-ca-cert" pod "0b83f3c5-8dbb-4187-ae4d-37b745bde034" (UID: "0b83f3c5-8dbb-4187-ae4d-37b745bde034"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:39:13.730719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.730688    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:13.730809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.730717    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:13.730809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.730734    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:13.730809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.730750    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:13.730809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.730765    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:13.732002 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.731977    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-token" pod "0b83f3c5-8dbb-4187-ae4d-37b745bde034" (UID: "0b83f3c5-8dbb-4187-ae4d-37b745bde034"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:39:13.732087 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.731978    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-podinfo" pod "0b83f3c5-8dbb-4187-ae4d-37b745bde034" (UID: "0b83f3c5-8dbb-4187-ae4d-37b745bde034"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:39:13.732587 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.732555    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-envoy" pod "0b83f3c5-8dbb-4187-ae4d-37b745bde034" (UID: "0b83f3c5-8dbb-4187-ae4d-37b745bde034"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:13.732587 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.732574    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-kube-api-access-vqn78" pod "0b83f3c5-8dbb-4187-ae4d-37b745bde034" (UID: "0b83f3c5-8dbb-4187-ae4d-37b745bde034"). InnerVolumeSpecName "kube-api-access-vqn78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:39:13.816189 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:39:13.816146    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:39:13.832213 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.832179    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqn78\" (UniqueName: \"kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-kube-api-access-vqn78\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:13.832213 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.832209    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:13.832484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.832223    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:13.832484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:13.832233    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b83f3c5-8dbb-4187-ae4d-37b745bde034-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:14.303981 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:14.303945    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"]
Nov 03 02:39:14.404068 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:14.404025    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"]
Nov 03 02:39:14.636911 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:14.636836    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"
Nov 03 02:39:14.636911 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:14.636863    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7" event={"ID":"0b83f3c5-8dbb-4187-ae4d-37b745bde034","Type":"ContainerDied","Data":"7d7baade3bacc8493cf2ed09d5f9c127a65529130f29e8474d53b684932c6932"}
Nov 03 02:39:14.637478 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:14.636929    5508 scope.go:122] "RemoveContainer" containerID="b93fcc21544e9a20eeca86f6603e518709fdd690407b301a802a6979ab3ed11c"
Nov 03 02:39:14.792672 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:14.792635    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"]
Nov 03 02:39:14.796049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:14.796021    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-059b23-openshift-default-54d9dcf779-crts7"]
Nov 03 02:39:14.961179 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:14.961139    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-mjk8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:39:14.961415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:14.961207    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" podUID="9fefda07-12d0-4158-a08d-4e0772428124" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:39:16.691669 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:16.691636    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" path="/var/lib/kubelet/pods/0b83f3c5-8dbb-4187-ae4d-37b745bde034/volumes"
Nov 03 02:39:17.198091 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.198065    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:39:17.304842 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.304752    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fefda07-12d0-4158-a08d-4e0772428124-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fefda07-12d0-4158-a08d-4e0772428124-istiod-ca-cert\") pod \"9fefda07-12d0-4158-a08d-4e0772428124\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") "
Nov 03 02:39:17.304842 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.304804    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-kube-api-access-gqml9\" (UniqueName: \"kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-kube-api-access-gqml9\") pod \"9fefda07-12d0-4158-a08d-4e0772428124\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") "
Nov 03 02:39:17.305069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.304911    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-envoy\") pod \"9fefda07-12d0-4158-a08d-4e0772428124\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") "
Nov 03 02:39:17.305069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.305022    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-socket\") pod \"9fefda07-12d0-4158-a08d-4e0772428124\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") "
Nov 03 02:39:17.305166 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.305087    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-data\") pod \"9fefda07-12d0-4158-a08d-4e0772428124\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") "
Nov 03 02:39:17.305166 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.305117    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fefda07-12d0-4158-a08d-4e0772428124-istiod-ca-cert" pod "9fefda07-12d0-4158-a08d-4e0772428124" (UID: "9fefda07-12d0-4158-a08d-4e0772428124"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:39:17.305166 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.305124    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-certs\") pod \"9fefda07-12d0-4158-a08d-4e0772428124\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") "
Nov 03 02:39:17.305334 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.305227    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fefda07-12d0-4158-a08d-4e0772428124-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fefda07-12d0-4158-a08d-4e0772428124-istio-podinfo\") pod \"9fefda07-12d0-4158-a08d-4e0772428124\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") "
Nov 03 02:39:17.305425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.305371    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-certs" pod "9fefda07-12d0-4158-a08d-4e0772428124" (UID: "9fefda07-12d0-4158-a08d-4e0772428124"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:17.305862 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.305485    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-credential-socket\") pod \"9fefda07-12d0-4158-a08d-4e0772428124\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") "
Nov 03 02:39:17.305862 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.305533    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-istio-token\" (UniqueName: \"kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-istio-token\") pod \"9fefda07-12d0-4158-a08d-4e0772428124\" (UID: \"9fefda07-12d0-4158-a08d-4e0772428124\") "
Nov 03 02:39:17.305862 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.305561    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-data" pod "9fefda07-12d0-4158-a08d-4e0772428124" (UID: "9fefda07-12d0-4158-a08d-4e0772428124"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:17.305862 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.305635    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-socket" pod "9fefda07-12d0-4158-a08d-4e0772428124" (UID: "9fefda07-12d0-4158-a08d-4e0772428124"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:17.305862 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.305777    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-credential-socket" pod "9fefda07-12d0-4158-a08d-4e0772428124" (UID: "9fefda07-12d0-4158-a08d-4e0772428124"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:17.306240 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.306049    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:17.306240 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.306067    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:17.306240 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.306081    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:17.306240 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.306095    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:17.306240 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.306111    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fefda07-12d0-4158-a08d-4e0772428124-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:17.307691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.307660    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-envoy" pod "9fefda07-12d0-4158-a08d-4e0772428124" (UID: "9fefda07-12d0-4158-a08d-4e0772428124"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:17.307804 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.307703    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-kube-api-access-gqml9" pod "9fefda07-12d0-4158-a08d-4e0772428124" (UID: "9fefda07-12d0-4158-a08d-4e0772428124"). InnerVolumeSpecName "kube-api-access-gqml9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:39:17.307979 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.307951    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-istio-token" pod "9fefda07-12d0-4158-a08d-4e0772428124" (UID: "9fefda07-12d0-4158-a08d-4e0772428124"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:39:17.308173 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.308153    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fefda07-12d0-4158-a08d-4e0772428124-istio-podinfo" pod "9fefda07-12d0-4158-a08d-4e0772428124" (UID: "9fefda07-12d0-4158-a08d-4e0772428124"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:39:17.407803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.407765    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqml9\" (UniqueName: \"kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-kube-api-access-gqml9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:17.407803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.407803    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fefda07-12d0-4158-a08d-4e0772428124-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:17.408047 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.407820    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fefda07-12d0-4158-a08d-4e0772428124-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:17.408047 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.407834    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fefda07-12d0-4158-a08d-4e0772428124-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:17.663207 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.663169    5508 generic.go:356] "Generic (PLEG): container finished" podID="9fefda07-12d0-4158-a08d-4e0772428124" containerID="578a7bfd301a84d7d0c278f49385fcee1abcca9806a00ddf806ced907d4c3db6" exitCode=0
Nov 03 02:39:17.663551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.663224    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"
Nov 03 02:39:17.663551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.663259    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" event={"ID":"9fefda07-12d0-4158-a08d-4e0772428124","Type":"ContainerDied","Data":"578a7bfd301a84d7d0c278f49385fcee1abcca9806a00ddf806ced907d4c3db6"}
Nov 03 02:39:17.663551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.663323    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j" event={"ID":"9fefda07-12d0-4158-a08d-4e0772428124","Type":"ContainerDied","Data":"b4aebbe5a6c3401f65e2be1fa3d1f74f776bbc757a4914969b2c5476236c13e7"}
Nov 03 02:39:17.663551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.663379    5508 scope.go:122] "RemoveContainer" containerID="578a7bfd301a84d7d0c278f49385fcee1abcca9806a00ddf806ced907d4c3db6"
Nov 03 02:39:17.674925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.674899    5508 scope.go:122] "RemoveContainer" containerID="578a7bfd301a84d7d0c278f49385fcee1abcca9806a00ddf806ced907d4c3db6"
Nov 03 02:39:17.675237 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:39:17.675216    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578a7bfd301a84d7d0c278f49385fcee1abcca9806a00ddf806ced907d4c3db6\": container with ID starting with 578a7bfd301a84d7d0c278f49385fcee1abcca9806a00ddf806ced907d4c3db6 not found: ID does not exist" containerID="578a7bfd301a84d7d0c278f49385fcee1abcca9806a00ddf806ced907d4c3db6"
Nov 03 02:39:17.675310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.675250    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578a7bfd301a84d7d0c278f49385fcee1abcca9806a00ddf806ced907d4c3db6"} err="failed to get container status \"578a7bfd301a84d7d0c278f49385fcee1abcca9806a00ddf806ced907d4c3db6\": rpc error: code = NotFound desc = could not find container \"578a7bfd301a84d7d0c278f49385fcee1abcca9806a00ddf806ced907d4c3db6\": container with ID starting with 578a7bfd301a84d7d0c278f49385fcee1abcca9806a00ddf806ced907d4c3db6 not found: ID does not exist"
Nov 03 02:39:17.818326 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.818286    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"]
Nov 03 02:39:17.864609 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:17.864574    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-mjk8j"]
Nov 03 02:39:18.691613 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:18.691573    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fefda07-12d0-4158-a08d-4e0772428124" path="/var/lib/kubelet/pods/9fefda07-12d0-4158-a08d-4e0772428124/volumes"
Nov 03 02:39:19.304943 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:19.304895    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l" podUID="944f1443-f33d-440c-964a-a600fd22b000" containerName="payload-processing" containerID="cri-o://df1afc219e21f8e30420e9356a6c060b2bc7a392eaa16b7b65a16ab5a609dd6c" gracePeriod=25
Nov 03 02:39:19.404387 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:19.404311    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz" podUID="c6acc642-397b-45d9-9d62-fae5f01cb830" containerName="payload-pre-processing" containerID="cri-o://4e5e1dc2dc9dfe56da59b0ec2760eb9b9cbd3a47f0e86e21c1ffa19d85aa0796" gracePeriod=25
Nov 03 02:39:25.363384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:25.363323    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz" podUID="c6acc642-397b-45d9-9d62-fae5f01cb830" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:39:25.363822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:25.363328    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l" podUID="944f1443-f33d-440c-964a-a600fd22b000" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:39:26.185227 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:26.185188    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:39:35.362636 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:35.362589    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz" podUID="c6acc642-397b-45d9-9d62-fae5f01cb830" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:39:35.363188 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:35.362593    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l" podUID="944f1443-f33d-440c-964a-a600fd22b000" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:39:36.961759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:36.961715    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"]
Nov 03 02:39:39.607304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.607272    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"
Nov 03 02:39:39.620689 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.620652    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944f1443-f33d-440c-964a-a600fd22b000-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944f1443-f33d-440c-964a-a600fd22b000-plugins-config-volume\") pod \"944f1443-f33d-440c-964a-a600fd22b000\" (UID: \"944f1443-f33d-440c-964a-a600fd22b000\") "
Nov 03 02:39:39.620895 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.620770    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944f1443-f33d-440c-964a-a600fd22b000-kube-api-access-cb49c\" (UniqueName: \"kubernetes.io/projected/944f1443-f33d-440c-964a-a600fd22b000-kube-api-access-cb49c\") pod \"944f1443-f33d-440c-964a-a600fd22b000\" (UID: \"944f1443-f33d-440c-964a-a600fd22b000\") "
Nov 03 02:39:39.621089 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.621062    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944f1443-f33d-440c-964a-a600fd22b000-plugins-config-volume" pod "944f1443-f33d-440c-964a-a600fd22b000" (UID: "944f1443-f33d-440c-964a-a600fd22b000"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:39:39.621943 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.621505    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/944f1443-f33d-440c-964a-a600fd22b000-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:39.623581 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.623535    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944f1443-f33d-440c-964a-a600fd22b000-kube-api-access-cb49c" pod "944f1443-f33d-440c-964a-a600fd22b000" (UID: "944f1443-f33d-440c-964a-a600fd22b000"). InnerVolumeSpecName "kube-api-access-cb49c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:39:39.678482 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.678451    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"
Nov 03 02:39:39.722897 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.722852    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6acc642-397b-45d9-9d62-fae5f01cb830-kube-api-access-cmtfs\" (UniqueName: \"kubernetes.io/projected/c6acc642-397b-45d9-9d62-fae5f01cb830-kube-api-access-cmtfs\") pod \"c6acc642-397b-45d9-9d62-fae5f01cb830\" (UID: \"c6acc642-397b-45d9-9d62-fae5f01cb830\") "
Nov 03 02:39:39.723125 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.722952    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6acc642-397b-45d9-9d62-fae5f01cb830-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6acc642-397b-45d9-9d62-fae5f01cb830-plugins-config-volume\") pod \"c6acc642-397b-45d9-9d62-fae5f01cb830\" (UID: \"c6acc642-397b-45d9-9d62-fae5f01cb830\") "
Nov 03 02:39:39.723387 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.723343    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb49c\" (UniqueName: \"kubernetes.io/projected/944f1443-f33d-440c-964a-a600fd22b000-kube-api-access-cb49c\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:39.723387 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.723344    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6acc642-397b-45d9-9d62-fae5f01cb830-plugins-config-volume" pod "c6acc642-397b-45d9-9d62-fae5f01cb830" (UID: "c6acc642-397b-45d9-9d62-fae5f01cb830"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:39:39.725472 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.725443    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6acc642-397b-45d9-9d62-fae5f01cb830-kube-api-access-cmtfs" pod "c6acc642-397b-45d9-9d62-fae5f01cb830" (UID: "c6acc642-397b-45d9-9d62-fae5f01cb830"). InnerVolumeSpecName "kube-api-access-cmtfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:39:39.824758 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.824665    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmtfs\" (UniqueName: \"kubernetes.io/projected/c6acc642-397b-45d9-9d62-fae5f01cb830-kube-api-access-cmtfs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:39.824758 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.824697    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6acc642-397b-45d9-9d62-fae5f01cb830-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:39.832228 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.832188    5508 generic.go:356] "Generic (PLEG): container finished" podID="c6acc642-397b-45d9-9d62-fae5f01cb830" containerID="4e5e1dc2dc9dfe56da59b0ec2760eb9b9cbd3a47f0e86e21c1ffa19d85aa0796" exitCode=0
Nov 03 02:39:39.832455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.832254    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"
Nov 03 02:39:39.832455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.832271    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz" event={"ID":"c6acc642-397b-45d9-9d62-fae5f01cb830","Type":"ContainerDied","Data":"4e5e1dc2dc9dfe56da59b0ec2760eb9b9cbd3a47f0e86e21c1ffa19d85aa0796"}
Nov 03 02:39:39.832455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.832329    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz" event={"ID":"c6acc642-397b-45d9-9d62-fae5f01cb830","Type":"ContainerDied","Data":"527013d4f6d882a73067041a7d98d7e5097884223a819158cc9ce7648ee31f1e"}
Nov 03 02:39:39.832455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.832387    5508 scope.go:122] "RemoveContainer" containerID="4e5e1dc2dc9dfe56da59b0ec2760eb9b9cbd3a47f0e86e21c1ffa19d85aa0796"
Nov 03 02:39:39.834047 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.834020    5508 generic.go:356] "Generic (PLEG): container finished" podID="944f1443-f33d-440c-964a-a600fd22b000" containerID="df1afc219e21f8e30420e9356a6c060b2bc7a392eaa16b7b65a16ab5a609dd6c" exitCode=0
Nov 03 02:39:39.834209 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.834069    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"
Nov 03 02:39:39.834209 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.834105    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l" event={"ID":"944f1443-f33d-440c-964a-a600fd22b000","Type":"ContainerDied","Data":"df1afc219e21f8e30420e9356a6c060b2bc7a392eaa16b7b65a16ab5a609dd6c"}
Nov 03 02:39:39.834209 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.834145    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l" event={"ID":"944f1443-f33d-440c-964a-a600fd22b000","Type":"ContainerDied","Data":"9f2dce96f8de5a5685802b9a36f8199211cc486d7d527af66da3ced6ceb11bfb"}
Nov 03 02:39:39.844953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.844921    5508 scope.go:122] "RemoveContainer" containerID="4e5e1dc2dc9dfe56da59b0ec2760eb9b9cbd3a47f0e86e21c1ffa19d85aa0796"
Nov 03 02:39:39.845420 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:39:39.845386    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e5e1dc2dc9dfe56da59b0ec2760eb9b9cbd3a47f0e86e21c1ffa19d85aa0796\": container with ID starting with 4e5e1dc2dc9dfe56da59b0ec2760eb9b9cbd3a47f0e86e21c1ffa19d85aa0796 not found: ID does not exist" containerID="4e5e1dc2dc9dfe56da59b0ec2760eb9b9cbd3a47f0e86e21c1ffa19d85aa0796"
Nov 03 02:39:39.845533 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.845431    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5e1dc2dc9dfe56da59b0ec2760eb9b9cbd3a47f0e86e21c1ffa19d85aa0796"} err="failed to get container status \"4e5e1dc2dc9dfe56da59b0ec2760eb9b9cbd3a47f0e86e21c1ffa19d85aa0796\": rpc error: code = NotFound desc = could not find container \"4e5e1dc2dc9dfe56da59b0ec2760eb9b9cbd3a47f0e86e21c1ffa19d85aa0796\": container with ID starting with 4e5e1dc2dc9dfe56da59b0ec2760eb9b9cbd3a47f0e86e21c1ffa19d85aa0796 not found: ID does not exist"
Nov 03 02:39:39.845533 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.845456    5508 scope.go:122] "RemoveContainer" containerID="df1afc219e21f8e30420e9356a6c060b2bc7a392eaa16b7b65a16ab5a609dd6c"
Nov 03 02:39:39.859126 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.859097    5508 scope.go:122] "RemoveContainer" containerID="df1afc219e21f8e30420e9356a6c060b2bc7a392eaa16b7b65a16ab5a609dd6c"
Nov 03 02:39:39.859594 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:39:39.859565    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1afc219e21f8e30420e9356a6c060b2bc7a392eaa16b7b65a16ab5a609dd6c\": container with ID starting with df1afc219e21f8e30420e9356a6c060b2bc7a392eaa16b7b65a16ab5a609dd6c not found: ID does not exist" containerID="df1afc219e21f8e30420e9356a6c060b2bc7a392eaa16b7b65a16ab5a609dd6c"
Nov 03 02:39:39.859726 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.859606    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1afc219e21f8e30420e9356a6c060b2bc7a392eaa16b7b65a16ab5a609dd6c"} err="failed to get container status \"df1afc219e21f8e30420e9356a6c060b2bc7a392eaa16b7b65a16ab5a609dd6c\": rpc error: code = NotFound desc = could not find container \"df1afc219e21f8e30420e9356a6c060b2bc7a392eaa16b7b65a16ab5a609dd6c\": container with ID starting with df1afc219e21f8e30420e9356a6c060b2bc7a392eaa16b7b65a16ab5a609dd6c not found: ID does not exist"
Nov 03 02:39:39.988791 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.988756    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"]
Nov 03 02:39:39.993453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:39.993418    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-434b6a-59f76dccc7-mzt6l"]
Nov 03 02:39:40.096386 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:40.096288    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"]
Nov 03 02:39:40.100968 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:40.100940    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-434b6a-7d6d5dfc8c-g7fwz"]
Nov 03 02:39:40.693859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:40.693823    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944f1443-f33d-440c-964a-a600fd22b000" path="/var/lib/kubelet/pods/944f1443-f33d-440c-964a-a600fd22b000/volumes"
Nov 03 02:39:40.694426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:40.694410    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6acc642-397b-45d9-9d62-fae5f01cb830" path="/var/lib/kubelet/pods/c6acc642-397b-45d9-9d62-fae5f01cb830/volumes"
Nov 03 02:39:41.962758 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:41.962695    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh" podUID="7fac1e0b-84e4-4edb-9abe-9537a002c2db" containerName="payload-pre-processing" containerID="cri-o://cb0592d2da7e3505b707ef7abc22a27b3535ca2e108daffac2fd1b6ddbad27e8" gracePeriod=25
Nov 03 02:39:42.710421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:42.710383    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"]
Nov 03 02:39:42.710701 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:42.710675    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" containerName="istio-proxy" containerID="cri-o://c225b96e741306753183bb09104726783a5dd3a2cb4e22741ff9f0c8f389e315" gracePeriod=30
Nov 03 02:39:43.818921 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:39:43.818867    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:39:45.315655 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:45.315611    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh" podUID="7fac1e0b-84e4-4edb-9abe-9537a002c2db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:39:47.909847 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:47.909792    5508 generic.go:356] "Generic (PLEG): container finished" podID="c98f1701-ad67-41e4-a528-23eb264203f8" containerID="c225b96e741306753183bb09104726783a5dd3a2cb4e22741ff9f0c8f389e315" exitCode=0
Nov 03 02:39:47.910270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:47.909843    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" event={"ID":"c98f1701-ad67-41e4-a528-23eb264203f8","Type":"ContainerDied","Data":"c225b96e741306753183bb09104726783a5dd3a2cb4e22741ff9f0c8f389e315"}
Nov 03 02:39:47.910270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:47.909914    5508 scope.go:122] "RemoveContainer" containerID="e08da6c9e680148b6d99ba863e2f1e4192dee750278c97bff05dc7522adfd450"
Nov 03 02:39:48.002472 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.002446    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:39:48.133988 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.133937    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-credential-socket\") pod \"c98f1701-ad67-41e4-a528-23eb264203f8\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") "
Nov 03 02:39:48.134184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.134015    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98f1701-ad67-41e4-a528-23eb264203f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98f1701-ad67-41e4-a528-23eb264203f8-istiod-ca-cert\") pod \"c98f1701-ad67-41e4-a528-23eb264203f8\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") "
Nov 03 02:39:48.134184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.134107    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c98f1701-ad67-41e4-a528-23eb264203f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98f1701-ad67-41e4-a528-23eb264203f8-istio-podinfo\") pod \"c98f1701-ad67-41e4-a528-23eb264203f8\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") "
Nov 03 02:39:48.134184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.134152    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-istio-token\" (UniqueName: \"kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-istio-token\") pod \"c98f1701-ad67-41e4-a528-23eb264203f8\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") "
Nov 03 02:39:48.134415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.134290    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-credential-socket" pod "c98f1701-ad67-41e4-a528-23eb264203f8" (UID: "c98f1701-ad67-41e4-a528-23eb264203f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:48.134482 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.134420    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-envoy\") pod \"c98f1701-ad67-41e4-a528-23eb264203f8\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") "
Nov 03 02:39:48.134540 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.134485    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-socket\") pod \"c98f1701-ad67-41e4-a528-23eb264203f8\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") "
Nov 03 02:39:48.134600 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.134542    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98f1701-ad67-41e4-a528-23eb264203f8-istiod-ca-cert" pod "c98f1701-ad67-41e4-a528-23eb264203f8" (UID: "c98f1701-ad67-41e4-a528-23eb264203f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:39:48.134600 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.134548    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-certs\") pod \"c98f1701-ad67-41e4-a528-23eb264203f8\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") "
Nov 03 02:39:48.134788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.134756    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-certs" pod "c98f1701-ad67-41e4-a528-23eb264203f8" (UID: "c98f1701-ad67-41e4-a528-23eb264203f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:48.134788 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.134771    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-socket" pod "c98f1701-ad67-41e4-a528-23eb264203f8" (UID: "c98f1701-ad67-41e4-a528-23eb264203f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:48.134998 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.134680    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-kube-api-access-89dvf\" (UniqueName: \"kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-kube-api-access-89dvf\") pod \"c98f1701-ad67-41e4-a528-23eb264203f8\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") "
Nov 03 02:39:48.134998 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.134866    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-data\") pod \"c98f1701-ad67-41e4-a528-23eb264203f8\" (UID: \"c98f1701-ad67-41e4-a528-23eb264203f8\") "
Nov 03 02:39:48.135147 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.135124    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-data" pod "c98f1701-ad67-41e4-a528-23eb264203f8" (UID: "c98f1701-ad67-41e4-a528-23eb264203f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:48.135372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.135334    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c98f1701-ad67-41e4-a528-23eb264203f8-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:48.135448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.135376    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:48.135448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.135391    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:48.135448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.135405    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:48.135448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.135419    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:48.136393 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.136363    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c98f1701-ad67-41e4-a528-23eb264203f8-istio-podinfo" pod "c98f1701-ad67-41e4-a528-23eb264203f8" (UID: "c98f1701-ad67-41e4-a528-23eb264203f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:39:48.136510 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.136467    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-envoy" pod "c98f1701-ad67-41e4-a528-23eb264203f8" (UID: "c98f1701-ad67-41e4-a528-23eb264203f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:39:48.136800 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.136785    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-istio-token" pod "c98f1701-ad67-41e4-a528-23eb264203f8" (UID: "c98f1701-ad67-41e4-a528-23eb264203f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:39:48.136890 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.136877    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-kube-api-access-89dvf" pod "c98f1701-ad67-41e4-a528-23eb264203f8" (UID: "c98f1701-ad67-41e4-a528-23eb264203f8"). InnerVolumeSpecName "kube-api-access-89dvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:39:48.236530 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.236435    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89dvf\" (UniqueName: \"kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-kube-api-access-89dvf\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:48.236530 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.236474    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c98f1701-ad67-41e4-a528-23eb264203f8-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:48.236530 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.236485    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c98f1701-ad67-41e4-a528-23eb264203f8-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:48.236530 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.236495    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c98f1701-ad67-41e4-a528-23eb264203f8-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:39:48.918484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.918457    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"
Nov 03 02:39:48.919006 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.918457    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf" event={"ID":"c98f1701-ad67-41e4-a528-23eb264203f8","Type":"ContainerDied","Data":"e58738f3a3e57f4a860c8428f5b7993050976ff609d15a9852825c8430d9c7bd"}
Nov 03 02:39:48.919006 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:48.918585    5508 scope.go:122] "RemoveContainer" containerID="c225b96e741306753183bb09104726783a5dd3a2cb4e22741ff9f0c8f389e315"
Nov 03 02:39:49.121417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:49.121304    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"]
Nov 03 02:39:49.169629 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:49.169582    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c34879-openshift-default-ccddfffff-8lghf"]
Nov 03 02:39:50.588531 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:50.588488    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"]
Nov 03 02:39:50.693115 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:50.693077    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" path="/var/lib/kubelet/pods/c98f1701-ad67-41e4-a528-23eb264203f8/volumes"
Nov 03 02:39:55.315083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:55.315047    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh" podUID="7fac1e0b-84e4-4edb-9abe-9537a002c2db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:39:55.589563 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:39:55.589467    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86" podUID="83bb0867-e955-4327-9790-2a78ab123f6e" containerName="payload-pre-processing" containerID="cri-o://a94929f9c4a585a45a7d2a79d6a03edc0127b946b5f76e1c38642b86915b5e1c" gracePeriod=25
Nov 03 02:40:01.078338 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:01.078296    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86" podUID="83bb0867-e955-4327-9790-2a78ab123f6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:40:02.252580 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:02.252539    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"
Nov 03 02:40:02.427311 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:02.427281    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fac1e0b-84e4-4edb-9abe-9537a002c2db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fac1e0b-84e4-4edb-9abe-9537a002c2db-plugins-config-volume\") pod \"7fac1e0b-84e4-4edb-9abe-9537a002c2db\" (UID: \"7fac1e0b-84e4-4edb-9abe-9537a002c2db\") "
Nov 03 02:40:02.427543 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:02.427509    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fac1e0b-84e4-4edb-9abe-9537a002c2db-kube-api-access-cr4bg\" (UniqueName: \"kubernetes.io/projected/7fac1e0b-84e4-4edb-9abe-9537a002c2db-kube-api-access-cr4bg\") pod \"7fac1e0b-84e4-4edb-9abe-9537a002c2db\" (UID: \"7fac1e0b-84e4-4edb-9abe-9537a002c2db\") "
Nov 03 02:40:02.427757 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:02.427735    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fac1e0b-84e4-4edb-9abe-9537a002c2db-plugins-config-volume" pod "7fac1e0b-84e4-4edb-9abe-9537a002c2db" (UID: "7fac1e0b-84e4-4edb-9abe-9537a002c2db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:40:02.428005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:02.427990    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fac1e0b-84e4-4edb-9abe-9537a002c2db-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:02.429637 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:02.429613    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fac1e0b-84e4-4edb-9abe-9537a002c2db-kube-api-access-cr4bg" pod "7fac1e0b-84e4-4edb-9abe-9537a002c2db" (UID: "7fac1e0b-84e4-4edb-9abe-9537a002c2db"). InnerVolumeSpecName "kube-api-access-cr4bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:40:02.529871 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:02.529785    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr4bg\" (UniqueName: \"kubernetes.io/projected/7fac1e0b-84e4-4edb-9abe-9537a002c2db-kube-api-access-cr4bg\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:03.028701 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:03.028663    5508 generic.go:356] "Generic (PLEG): container finished" podID="7fac1e0b-84e4-4edb-9abe-9537a002c2db" containerID="cb0592d2da7e3505b707ef7abc22a27b3535ca2e108daffac2fd1b6ddbad27e8" exitCode=0
Nov 03 02:40:03.028939 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:03.028741    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"
Nov 03 02:40:03.028939 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:03.028746    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh" event={"ID":"7fac1e0b-84e4-4edb-9abe-9537a002c2db","Type":"ContainerDied","Data":"cb0592d2da7e3505b707ef7abc22a27b3535ca2e108daffac2fd1b6ddbad27e8"}
Nov 03 02:40:03.028939 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:03.028811    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh" event={"ID":"7fac1e0b-84e4-4edb-9abe-9537a002c2db","Type":"ContainerDied","Data":"a5e07758da8f643275806796d1910949998641053473507a0ac74f85926b5005"}
Nov 03 02:40:03.028939 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:03.028841    5508 scope.go:122] "RemoveContainer" containerID="cb0592d2da7e3505b707ef7abc22a27b3535ca2e108daffac2fd1b6ddbad27e8"
Nov 03 02:40:03.039221 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:03.039198    5508 scope.go:122] "RemoveContainer" containerID="cb0592d2da7e3505b707ef7abc22a27b3535ca2e108daffac2fd1b6ddbad27e8"
Nov 03 02:40:03.039571 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:40:03.039544    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0592d2da7e3505b707ef7abc22a27b3535ca2e108daffac2fd1b6ddbad27e8\": container with ID starting with cb0592d2da7e3505b707ef7abc22a27b3535ca2e108daffac2fd1b6ddbad27e8 not found: ID does not exist" containerID="cb0592d2da7e3505b707ef7abc22a27b3535ca2e108daffac2fd1b6ddbad27e8"
Nov 03 02:40:03.039689 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:03.039582    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0592d2da7e3505b707ef7abc22a27b3535ca2e108daffac2fd1b6ddbad27e8"} err="failed to get container status \"cb0592d2da7e3505b707ef7abc22a27b3535ca2e108daffac2fd1b6ddbad27e8\": rpc error: code = NotFound desc = could not find container \"cb0592d2da7e3505b707ef7abc22a27b3535ca2e108daffac2fd1b6ddbad27e8\": container with ID starting with cb0592d2da7e3505b707ef7abc22a27b3535ca2e108daffac2fd1b6ddbad27e8 not found: ID does not exist"
Nov 03 02:40:03.185927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:03.185891    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"]
Nov 03 02:40:03.232105 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:03.232070    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-98a394-7cc8d55cd7-sxzzh"]
Nov 03 02:40:04.694121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:04.694082    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fac1e0b-84e4-4edb-9abe-9537a002c2db" path="/var/lib/kubelet/pods/7fac1e0b-84e4-4edb-9abe-9537a002c2db/volumes"
Nov 03 02:40:06.671270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:06.671232    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"]
Nov 03 02:40:06.671786 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:06.671541    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerName="istio-proxy" containerID="cri-o://6178c5ca607ab2310a61808adecee43bd36fb78c719f7f83d1e270902160ff02" gracePeriod=30
Nov 03 02:40:07.853316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853267    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"]
Nov 03 02:40:07.853885 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853809    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7eb719a8-4f64-4414-8cb4-e94c97c2c66e" containerName="revoke-keys"
Nov 03 02:40:07.853885 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853822    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb719a8-4f64-4414-8cb4-e94c97c2c66e" containerName="revoke-keys"
Nov 03 02:40:07.853885 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853835    5508 cpu_manager.go:414] "containerMap: removing container" podUID="eb6b9575-4487-4e4d-92c5-55a93baff2f4" containerName="maas-api"
Nov 03 02:40:07.853885 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853841    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb6b9575-4487-4e4d-92c5-55a93baff2f4" containerName="maas-api"
Nov 03 02:40:07.853885 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853851    5508 cpu_manager.go:414] "containerMap: removing container" podUID="acb627be-2d98-4b57-86e6-99ed6ff1e3b2" containerName="payload-pre-processing"
Nov 03 02:40:07.853885 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853858    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb627be-2d98-4b57-86e6-99ed6ff1e3b2" containerName="payload-pre-processing"
Nov 03 02:40:07.853885 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853868    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerName="istio-proxy"
Nov 03 02:40:07.853885 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853872    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerName="istio-proxy"
Nov 03 02:40:07.853885 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853881    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" containerName="istio-proxy"
Nov 03 02:40:07.853885 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853886    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" containerName="istio-proxy"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853894    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9fefda07-12d0-4158-a08d-4e0772428124" containerName="istio-proxy"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853902    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fefda07-12d0-4158-a08d-4e0772428124" containerName="istio-proxy"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853918    5508 cpu_manager.go:414] "containerMap: removing container" podUID="944f1443-f33d-440c-964a-a600fd22b000" containerName="payload-processing"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853924    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944f1443-f33d-440c-964a-a600fd22b000" containerName="payload-processing"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853935    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c6acc642-397b-45d9-9d62-fae5f01cb830" containerName="payload-pre-processing"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853941    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6acc642-397b-45d9-9d62-fae5f01cb830" containerName="payload-pre-processing"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853953    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" containerName="istio-proxy"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853959    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" containerName="istio-proxy"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853973    5508 cpu_manager.go:414] "containerMap: removing container" podUID="7fac1e0b-84e4-4edb-9abe-9537a002c2db" containerName="payload-pre-processing"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853980    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fac1e0b-84e4-4edb-9abe-9537a002c2db" containerName="payload-pre-processing"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.853997    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerName="istio-proxy"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.854004    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerName="istio-proxy"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.854139    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6acc642-397b-45d9-9d62-fae5f01cb830" containerName="payload-pre-processing"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.854154    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" containerName="istio-proxy"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.854164    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fac1e0b-84e4-4edb-9abe-9537a002c2db" containerName="payload-pre-processing"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.854175    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerName="istio-proxy"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.854184    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a199c50f-4280-402e-b6cd-9433e0639b7a" containerName="istio-proxy"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.854192    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb719a8-4f64-4414-8cb4-e94c97c2c66e" containerName="revoke-keys"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.854198    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb6b9575-4487-4e4d-92c5-55a93baff2f4" containerName="maas-api"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.854204    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb627be-2d98-4b57-86e6-99ed6ff1e3b2" containerName="payload-pre-processing"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.854210    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98f1701-ad67-41e4-a528-23eb264203f8" containerName="istio-proxy"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.854215    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fefda07-12d0-4158-a08d-4e0772428124" containerName="istio-proxy"
Nov 03 02:40:07.854439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.854224    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="944f1443-f33d-440c-964a-a600fd22b000" containerName="payload-processing"
Nov 03 02:40:07.859580 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.859555    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"
Nov 03 02:40:07.891081 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.891047    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hb4\" (UniqueName: \"kubernetes.io/projected/6b862333-9903-4c4c-b84a-a8519c7c0b14-kube-api-access-t5hb4\") pod \"maas-api-revoke-sub-ee878647-7lxrr\" (UID: \"6b862333-9903-4c4c-b84a-a8519c7c0b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"
Nov 03 02:40:07.891282 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.891118    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee878647-7lxrr\" (UID: \"6b862333-9903-4c4c-b84a-a8519c7c0b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"
Nov 03 02:40:07.891282 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.891237    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee878647-7lxrr\" (UID: \"6b862333-9903-4c4c-b84a-a8519c7c0b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"
Nov 03 02:40:07.908785 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.908749    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:40:07.911131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.911111    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:40:07.958333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.958296    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"]
Nov 03 02:40:07.993059 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.993013    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hb4\" (UniqueName: \"kubernetes.io/projected/6b862333-9903-4c4c-b84a-a8519c7c0b14-kube-api-access-t5hb4\") pod \"maas-api-revoke-sub-ee878647-7lxrr\" (UID: \"6b862333-9903-4c4c-b84a-a8519c7c0b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"
Nov 03 02:40:07.993293 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.993099    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee878647-7lxrr\" (UID: \"6b862333-9903-4c4c-b84a-a8519c7c0b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"
Nov 03 02:40:07.993293 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.993143    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee878647-7lxrr\" (UID: \"6b862333-9903-4c4c-b84a-a8519c7c0b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"
Nov 03 02:40:07.993956 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.993928    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee878647-7lxrr\" (UID: \"6b862333-9903-4c4c-b84a-a8519c7c0b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"
Nov 03 02:40:07.995697 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:07.995669    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee878647-7lxrr\" (UID: \"6b862333-9903-4c4c-b84a-a8519c7c0b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"
Nov 03 02:40:08.050605 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:08.050573    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hb4\" (UniqueName: \"kubernetes.io/projected/6b862333-9903-4c4c-b84a-a8519c7c0b14-kube-api-access-t5hb4\") pod \"maas-api-revoke-sub-ee878647-7lxrr\" (UID: \"6b862333-9903-4c4c-b84a-a8519c7c0b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"
Nov 03 02:40:08.177074 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:08.177035    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"
Nov 03 02:40:08.555504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:08.555423    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"]
Nov 03 02:40:08.563984 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:40:08.563948    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b862333_9903_4c4c_b84a_a8519c7c0b14.slice/crio-5aa3b944c33d69c4e56c2ddf1828abefc2dd9f9baa6a5c2246e0e6d1efb18457 WatchSource:0}: Error finding container 5aa3b944c33d69c4e56c2ddf1828abefc2dd9f9baa6a5c2246e0e6d1efb18457: Status 404 returned error can't find the container with id 5aa3b944c33d69c4e56c2ddf1828abefc2dd9f9baa6a5c2246e0e6d1efb18457
Nov 03 02:40:09.075947 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:09.075909    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr" event={"ID":"6b862333-9903-4c4c-b84a-a8519c7c0b14","Type":"ContainerStarted","Data":"8b51aa8791277bdaa0c811abcfbb2f545ce099bebb85833f33612de6826fef20"}
Nov 03 02:40:09.076389 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:09.075978    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr" event={"ID":"6b862333-9903-4c4c-b84a-a8519c7c0b14","Type":"ContainerStarted","Data":"5aa3b944c33d69c4e56c2ddf1828abefc2dd9f9baa6a5c2246e0e6d1efb18457"}
Nov 03 02:40:09.185096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:09.185040    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr" podStartSLOduration=2.185025361 podStartE2EDuration="2.185025361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:40:09.183243565 +0000 UTC m=+4888.608850030" watchObservedRunningTime="2026-11-03 02:40:09.185025361 +0000 UTC m=+4888.610631823"
Nov 03 02:40:09.572836 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:40:09.572795    5508 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b862333_9903_4c4c_b84a_a8519c7c0b14.slice/crio-8b51aa8791277bdaa0c811abcfbb2f545ce099bebb85833f33612de6826fef20.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 02:40:10.084766 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:10.084725    5508 generic.go:356] "Generic (PLEG): container finished" podID="6b862333-9903-4c4c-b84a-a8519c7c0b14" containerID="8b51aa8791277bdaa0c811abcfbb2f545ce099bebb85833f33612de6826fef20" exitCode=0
Nov 03 02:40:10.085185 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:10.084809    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr" event={"ID":"6b862333-9903-4c4c-b84a-a8519c7c0b14","Type":"ContainerDied","Data":"8b51aa8791277bdaa0c811abcfbb2f545ce099bebb85833f33612de6826fef20"}
Nov 03 02:40:11.078948 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:11.078901    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86" podUID="83bb0867-e955-4327-9790-2a78ab123f6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:40:11.238992 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:11.238966    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"
Nov 03 02:40:11.334836 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:11.334748    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-xks-ca\") pod \"6b862333-9903-4c4c-b84a-a8519c7c0b14\" (UID: \"6b862333-9903-4c4c-b84a-a8519c7c0b14\") "
Nov 03 02:40:11.335018 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:11.334878    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b862333-9903-4c4c-b84a-a8519c7c0b14-kube-api-access-t5hb4\" (UniqueName: \"kubernetes.io/projected/6b862333-9903-4c4c-b84a-a8519c7c0b14-kube-api-access-t5hb4\") pod \"6b862333-9903-4c4c-b84a-a8519c7c0b14\" (UID: \"6b862333-9903-4c4c-b84a-a8519c7c0b14\") "
Nov 03 02:40:11.335018 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:11.334979    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-openshift-ca\") pod \"6b862333-9903-4c4c-b84a-a8519c7c0b14\" (UID: \"6b862333-9903-4c4c-b84a-a8519c7c0b14\") "
Nov 03 02:40:11.335365 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:11.335322    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-openshift-ca" pod "6b862333-9903-4c4c-b84a-a8519c7c0b14" (UID: "6b862333-9903-4c4c-b84a-a8519c7c0b14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:40:11.335492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:11.335424    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:11.336959 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:11.336931    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-xks-ca" pod "6b862333-9903-4c4c-b84a-a8519c7c0b14" (UID: "6b862333-9903-4c4c-b84a-a8519c7c0b14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:40:11.336959 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:11.336947    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b862333-9903-4c4c-b84a-a8519c7c0b14-kube-api-access-t5hb4" pod "6b862333-9903-4c4c-b84a-a8519c7c0b14" (UID: "6b862333-9903-4c4c-b84a-a8519c7c0b14"). InnerVolumeSpecName "kube-api-access-t5hb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:40:11.436556 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:11.436520    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b862333-9903-4c4c-b84a-a8519c7c0b14-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:11.436556 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:11.436553    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5hb4\" (UniqueName: \"kubernetes.io/projected/6b862333-9903-4c4c-b84a-a8519c7c0b14-kube-api-access-t5hb4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:11.970274 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:11.970241    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:40:12.024594 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.024555    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:40:12.045808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.045770    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-kube-api-access-ptm8v\" (UniqueName: \"kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-kube-api-access-ptm8v\") pod \"2f380f36-d73c-4e52-afb0-ef1f167df815\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") "
Nov 03 02:40:12.045995 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.045876    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-data\") pod \"2f380f36-d73c-4e52-afb0-ef1f167df815\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") "
Nov 03 02:40:12.045995 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.045903    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-certs\") pod \"2f380f36-d73c-4e52-afb0-ef1f167df815\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") "
Nov 03 02:40:12.045995 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.045931    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-socket\") pod \"2f380f36-d73c-4e52-afb0-ef1f167df815\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") "
Nov 03 02:40:12.045995 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.045958    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-token\" (UniqueName: \"kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-token\") pod \"2f380f36-d73c-4e52-afb0-ef1f167df815\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") "
Nov 03 02:40:12.046248 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.046011    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f380f36-d73c-4e52-afb0-ef1f167df815-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f380f36-d73c-4e52-afb0-ef1f167df815-istiod-ca-cert\") pod \"2f380f36-d73c-4e52-afb0-ef1f167df815\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") "
Nov 03 02:40:12.046248 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.046060    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-envoy\") pod \"2f380f36-d73c-4e52-afb0-ef1f167df815\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") "
Nov 03 02:40:12.046248 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.046172    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-podinfo\") pod \"2f380f36-d73c-4e52-afb0-ef1f167df815\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") "
Nov 03 02:40:12.046248 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.046205    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-credential-socket\") pod \"2f380f36-d73c-4e52-afb0-ef1f167df815\" (UID: \"2f380f36-d73c-4e52-afb0-ef1f167df815\") "
Nov 03 02:40:12.046516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.046284    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-certs" pod "2f380f36-d73c-4e52-afb0-ef1f167df815" (UID: "2f380f36-d73c-4e52-afb0-ef1f167df815"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:40:12.046516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.046217    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-data" pod "2f380f36-d73c-4e52-afb0-ef1f167df815" (UID: "2f380f36-d73c-4e52-afb0-ef1f167df815"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:40:12.046639 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.046586    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-credential-socket" pod "2f380f36-d73c-4e52-afb0-ef1f167df815" (UID: "2f380f36-d73c-4e52-afb0-ef1f167df815"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:40:12.046696 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.046677    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-socket" pod "2f380f36-d73c-4e52-afb0-ef1f167df815" (UID: "2f380f36-d73c-4e52-afb0-ef1f167df815"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:40:12.046753 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.046725    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:12.046753 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.046742    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:12.046753 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.046743    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f380f36-d73c-4e52-afb0-ef1f167df815-istiod-ca-cert" pod "2f380f36-d73c-4e52-afb0-ef1f167df815" (UID: "2f380f36-d73c-4e52-afb0-ef1f167df815"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:40:12.046959 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.046758    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:12.048537 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.048509    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-envoy" pod "2f380f36-d73c-4e52-afb0-ef1f167df815" (UID: "2f380f36-d73c-4e52-afb0-ef1f167df815"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:40:12.048666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.048641    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-token" pod "2f380f36-d73c-4e52-afb0-ef1f167df815" (UID: "2f380f36-d73c-4e52-afb0-ef1f167df815"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:40:12.048666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.048649    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-podinfo" pod "2f380f36-d73c-4e52-afb0-ef1f167df815" (UID: "2f380f36-d73c-4e52-afb0-ef1f167df815"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:40:12.048791 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.048669    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-kube-api-access-ptm8v" pod "2f380f36-d73c-4e52-afb0-ef1f167df815" (UID: "2f380f36-d73c-4e52-afb0-ef1f167df815"). InnerVolumeSpecName "kube-api-access-ptm8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:40:12.104897 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.104861    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr" event={"ID":"6b862333-9903-4c4c-b84a-a8519c7c0b14","Type":"ContainerDied","Data":"5aa3b944c33d69c4e56c2ddf1828abefc2dd9f9baa6a5c2246e0e6d1efb18457"}
Nov 03 02:40:12.105103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.104926    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aa3b944c33d69c4e56c2ddf1828abefc2dd9f9baa6a5c2246e0e6d1efb18457"
Nov 03 02:40:12.105103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.104867    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"
Nov 03 02:40:12.107299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.107124    5508 generic.go:356] "Generic (PLEG): container finished" podID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerID="6178c5ca607ab2310a61808adecee43bd36fb78c719f7f83d1e270902160ff02" exitCode=0
Nov 03 02:40:12.107299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.107176    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"
Nov 03 02:40:12.107299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.107169    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" event={"ID":"2f380f36-d73c-4e52-afb0-ef1f167df815","Type":"ContainerDied","Data":"6178c5ca607ab2310a61808adecee43bd36fb78c719f7f83d1e270902160ff02"}
Nov 03 02:40:12.107601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.107296    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc" event={"ID":"2f380f36-d73c-4e52-afb0-ef1f167df815","Type":"ContainerDied","Data":"b6d6bb06936718ef3a0384fff4bfed3a6a0d49934779d971b6be3fe0efc96a29"}
Nov 03 02:40:12.107601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.107339    5508 scope.go:122] "RemoveContainer" containerID="6178c5ca607ab2310a61808adecee43bd36fb78c719f7f83d1e270902160ff02"
Nov 03 02:40:12.125233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.125036    5508 scope.go:122] "RemoveContainer" containerID="58cabfff660dc8b72cf33bc3d24f132b84591bffc01db2ddf6556770d16fce05"
Nov 03 02:40:12.136832 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.136808    5508 scope.go:122] "RemoveContainer" containerID="6178c5ca607ab2310a61808adecee43bd36fb78c719f7f83d1e270902160ff02"
Nov 03 02:40:12.137187 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:40:12.137162    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6178c5ca607ab2310a61808adecee43bd36fb78c719f7f83d1e270902160ff02\": container with ID starting with 6178c5ca607ab2310a61808adecee43bd36fb78c719f7f83d1e270902160ff02 not found: ID does not exist" containerID="6178c5ca607ab2310a61808adecee43bd36fb78c719f7f83d1e270902160ff02"
Nov 03 02:40:12.137329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.137197    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6178c5ca607ab2310a61808adecee43bd36fb78c719f7f83d1e270902160ff02"} err="failed to get container status \"6178c5ca607ab2310a61808adecee43bd36fb78c719f7f83d1e270902160ff02\": rpc error: code = NotFound desc = could not find container \"6178c5ca607ab2310a61808adecee43bd36fb78c719f7f83d1e270902160ff02\": container with ID starting with 6178c5ca607ab2310a61808adecee43bd36fb78c719f7f83d1e270902160ff02 not found: ID does not exist"
Nov 03 02:40:12.137329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.137219    5508 scope.go:122] "RemoveContainer" containerID="58cabfff660dc8b72cf33bc3d24f132b84591bffc01db2ddf6556770d16fce05"
Nov 03 02:40:12.137580 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:40:12.137557    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58cabfff660dc8b72cf33bc3d24f132b84591bffc01db2ddf6556770d16fce05\": container with ID starting with 58cabfff660dc8b72cf33bc3d24f132b84591bffc01db2ddf6556770d16fce05 not found: ID does not exist" containerID="58cabfff660dc8b72cf33bc3d24f132b84591bffc01db2ddf6556770d16fce05"
Nov 03 02:40:12.137622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.137588    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58cabfff660dc8b72cf33bc3d24f132b84591bffc01db2ddf6556770d16fce05"} err="failed to get container status \"58cabfff660dc8b72cf33bc3d24f132b84591bffc01db2ddf6556770d16fce05\": rpc error: code = NotFound desc = could not find container \"58cabfff660dc8b72cf33bc3d24f132b84591bffc01db2ddf6556770d16fce05\": container with ID starting with 58cabfff660dc8b72cf33bc3d24f132b84591bffc01db2ddf6556770d16fce05 not found: ID does not exist"
Nov 03 02:40:12.148320 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.148288    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:12.148320 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.148320    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:12.148645 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.148336    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f380f36-d73c-4e52-afb0-ef1f167df815-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:12.148645 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.148378    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:12.148645 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.148391    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f380f36-d73c-4e52-afb0-ef1f167df815-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:12.148645 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.148406    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptm8v\" (UniqueName: \"kubernetes.io/projected/2f380f36-d73c-4e52-afb0-ef1f167df815-kube-api-access-ptm8v\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:12.266037 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.266000    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"]
Nov 03 02:40:12.273124 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.273093    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-434b6a-openshift-default-77dc6bb55f-tcczc"]
Nov 03 02:40:12.691614 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:12.691584    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" path="/var/lib/kubelet/pods/2f380f36-d73c-4e52-afb0-ef1f167df815/volumes"
Nov 03 02:40:13.812461 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:40:13.812425    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:40:15.870273 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:15.870238    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"
Nov 03 02:40:15.992477 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:15.992439    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83bb0867-e955-4327-9790-2a78ab123f6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83bb0867-e955-4327-9790-2a78ab123f6e-plugins-config-volume\") pod \"83bb0867-e955-4327-9790-2a78ab123f6e\" (UID: \"83bb0867-e955-4327-9790-2a78ab123f6e\") "
Nov 03 02:40:15.992673 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:15.992632    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83bb0867-e955-4327-9790-2a78ab123f6e-kube-api-access-gq9fh\" (UniqueName: \"kubernetes.io/projected/83bb0867-e955-4327-9790-2a78ab123f6e-kube-api-access-gq9fh\") pod \"83bb0867-e955-4327-9790-2a78ab123f6e\" (UID: \"83bb0867-e955-4327-9790-2a78ab123f6e\") "
Nov 03 02:40:15.993095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:15.992905    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83bb0867-e955-4327-9790-2a78ab123f6e-plugins-config-volume" pod "83bb0867-e955-4327-9790-2a78ab123f6e" (UID: "83bb0867-e955-4327-9790-2a78ab123f6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:40:15.993095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:15.993066    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83bb0867-e955-4327-9790-2a78ab123f6e-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:15.995662 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:15.995630    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bb0867-e955-4327-9790-2a78ab123f6e-kube-api-access-gq9fh" pod "83bb0867-e955-4327-9790-2a78ab123f6e" (UID: "83bb0867-e955-4327-9790-2a78ab123f6e"). InnerVolumeSpecName "kube-api-access-gq9fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:40:16.093992 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:16.093946    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq9fh\" (UniqueName: \"kubernetes.io/projected/83bb0867-e955-4327-9790-2a78ab123f6e-kube-api-access-gq9fh\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:16.136972 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:16.136886    5508 generic.go:356] "Generic (PLEG): container finished" podID="83bb0867-e955-4327-9790-2a78ab123f6e" containerID="a94929f9c4a585a45a7d2a79d6a03edc0127b946b5f76e1c38642b86915b5e1c" exitCode=0
Nov 03 02:40:16.136972 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:16.136940    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"
Nov 03 02:40:16.137162 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:16.136962    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86" event={"ID":"83bb0867-e955-4327-9790-2a78ab123f6e","Type":"ContainerDied","Data":"a94929f9c4a585a45a7d2a79d6a03edc0127b946b5f76e1c38642b86915b5e1c"}
Nov 03 02:40:16.137162 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:16.137019    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86" event={"ID":"83bb0867-e955-4327-9790-2a78ab123f6e","Type":"ContainerDied","Data":"9cf180d74f1e7706a15242a77ed5aa0fda88c90c131c7abe24ec3a85dc128db0"}
Nov 03 02:40:16.137162 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:16.137041    5508 scope.go:122] "RemoveContainer" containerID="a94929f9c4a585a45a7d2a79d6a03edc0127b946b5f76e1c38642b86915b5e1c"
Nov 03 02:40:16.147121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:16.147100    5508 scope.go:122] "RemoveContainer" containerID="a94929f9c4a585a45a7d2a79d6a03edc0127b946b5f76e1c38642b86915b5e1c"
Nov 03 02:40:16.147428 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:40:16.147402    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a94929f9c4a585a45a7d2a79d6a03edc0127b946b5f76e1c38642b86915b5e1c\": container with ID starting with a94929f9c4a585a45a7d2a79d6a03edc0127b946b5f76e1c38642b86915b5e1c not found: ID does not exist" containerID="a94929f9c4a585a45a7d2a79d6a03edc0127b946b5f76e1c38642b86915b5e1c"
Nov 03 02:40:16.147483 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:16.147440    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a94929f9c4a585a45a7d2a79d6a03edc0127b946b5f76e1c38642b86915b5e1c"} err="failed to get container status \"a94929f9c4a585a45a7d2a79d6a03edc0127b946b5f76e1c38642b86915b5e1c\": rpc error: code = NotFound desc = could not find container \"a94929f9c4a585a45a7d2a79d6a03edc0127b946b5f76e1c38642b86915b5e1c\": container with ID starting with a94929f9c4a585a45a7d2a79d6a03edc0127b946b5f76e1c38642b86915b5e1c not found: ID does not exist"
Nov 03 02:40:16.291508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:16.291467    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"]
Nov 03 02:40:16.297077 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:16.297048    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9576a-7b67d65c8c-qjt86"]
Nov 03 02:40:16.698238 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:16.698202    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83bb0867-e955-4327-9790-2a78ab123f6e" path="/var/lib/kubelet/pods/83bb0867-e955-4327-9790-2a78ab123f6e/volumes"
Nov 03 02:40:22.045058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:22.045011    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"]
Nov 03 02:40:22.093009 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:22.092970    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41bc39c3-jj6kg"]
Nov 03 02:40:22.692478 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:22.692442    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31fa90d8-9ef4-41bd-b11d-c903b734e8f2" path="/var/lib/kubelet/pods/31fa90d8-9ef4-41bd-b11d-c903b734e8f2/volumes"
Nov 03 02:40:23.900674 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.900633    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"]
Nov 03 02:40:23.901333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.901311    5508 cpu_manager.go:414] "containerMap: removing container" podUID="6b862333-9903-4c4c-b84a-a8519c7c0b14" containerName="revoke-keys"
Nov 03 02:40:23.901473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.901334    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b862333-9903-4c4c-b84a-a8519c7c0b14" containerName="revoke-keys"
Nov 03 02:40:23.901473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.901398    5508 cpu_manager.go:414] "containerMap: removing container" podUID="83bb0867-e955-4327-9790-2a78ab123f6e" containerName="payload-pre-processing"
Nov 03 02:40:23.901473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.901407    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bb0867-e955-4327-9790-2a78ab123f6e" containerName="payload-pre-processing"
Nov 03 02:40:23.901473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.901430    5508 cpu_manager.go:414] "containerMap: removing container" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerName="istio-proxy"
Nov 03 02:40:23.901473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.901437    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerName="istio-proxy"
Nov 03 02:40:23.901473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.901451    5508 cpu_manager.go:414] "containerMap: removing container" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerName="istio-proxy"
Nov 03 02:40:23.901473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.901458    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerName="istio-proxy"
Nov 03 02:40:23.901856 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.901590    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b83f3c5-8dbb-4187-ae4d-37b745bde034" containerName="istio-proxy"
Nov 03 02:40:23.901856 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.901605    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bb0867-e955-4327-9790-2a78ab123f6e" containerName="payload-pre-processing"
Nov 03 02:40:23.901856 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.901619    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerName="istio-proxy"
Nov 03 02:40:23.901856 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.901628    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f380f36-d73c-4e52-afb0-ef1f167df815" containerName="istio-proxy"
Nov 03 02:40:23.901856 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.901641    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b862333-9903-4c4c-b84a-a8519c7c0b14" containerName="revoke-keys"
Nov 03 02:40:23.905499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.905474    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"
Nov 03 02:40:23.954242 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.954212    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:40:23.993859 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.993814    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9962ecc5-fp448\" (UID: \"ce6081b1-f32a-4206-9041-4adb4147dcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"
Nov 03 02:40:23.994087 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.993894    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9962ecc5-fp448\" (UID: \"ce6081b1-f32a-4206-9041-4adb4147dcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"
Nov 03 02:40:23.994087 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:23.994057    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qs6v\" (UniqueName: \"kubernetes.io/projected/ce6081b1-f32a-4206-9041-4adb4147dcdf-kube-api-access-5qs6v\") pod \"maas-api-revoke-sub-9962ecc5-fp448\" (UID: \"ce6081b1-f32a-4206-9041-4adb4147dcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"
Nov 03 02:40:24.000298 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:24.000268    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:40:24.005137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:24.005105    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"]
Nov 03 02:40:24.094976 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:24.094937    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9962ecc5-fp448\" (UID: \"ce6081b1-f32a-4206-9041-4adb4147dcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"
Nov 03 02:40:24.095182 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:24.095016    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9962ecc5-fp448\" (UID: \"ce6081b1-f32a-4206-9041-4adb4147dcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"
Nov 03 02:40:24.095182 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:24.095125    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qs6v\" (UniqueName: \"kubernetes.io/projected/ce6081b1-f32a-4206-9041-4adb4147dcdf-kube-api-access-5qs6v\") pod \"maas-api-revoke-sub-9962ecc5-fp448\" (UID: \"ce6081b1-f32a-4206-9041-4adb4147dcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"
Nov 03 02:40:24.095672 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:24.095654    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9962ecc5-fp448\" (UID: \"ce6081b1-f32a-4206-9041-4adb4147dcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"
Nov 03 02:40:24.097589 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:24.097558    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9962ecc5-fp448\" (UID: \"ce6081b1-f32a-4206-9041-4adb4147dcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"
Nov 03 02:40:24.152923 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:24.152852    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qs6v\" (UniqueName: \"kubernetes.io/projected/ce6081b1-f32a-4206-9041-4adb4147dcdf-kube-api-access-5qs6v\") pod \"maas-api-revoke-sub-9962ecc5-fp448\" (UID: \"ce6081b1-f32a-4206-9041-4adb4147dcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"
Nov 03 02:40:24.218876 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:24.218844    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"
Nov 03 02:40:24.592072 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:24.591992    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"]
Nov 03 02:40:24.598732 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:40:24.598698    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6081b1_f32a_4206_9041_4adb4147dcdf.slice/crio-e8157495708437e033220aa6c6e7f92b4904447a390808379706d8b7a96ccb0f WatchSource:0}: Error finding container e8157495708437e033220aa6c6e7f92b4904447a390808379706d8b7a96ccb0f: Status 404 returned error can't find the container with id e8157495708437e033220aa6c6e7f92b4904447a390808379706d8b7a96ccb0f
Nov 03 02:40:25.128458 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:25.128364    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"]
Nov 03 02:40:25.209770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:25.209727    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448" event={"ID":"ce6081b1-f32a-4206-9041-4adb4147dcdf","Type":"ContainerStarted","Data":"5032afd5cc12934bf8e54c5c73cd25b0470de117fa2ff433e23f010c706627a4"}
Nov 03 02:40:25.210081 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:25.209791    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448" event={"ID":"ce6081b1-f32a-4206-9041-4adb4147dcdf","Type":"ContainerStarted","Data":"e8157495708437e033220aa6c6e7f92b4904447a390808379706d8b7a96ccb0f"}
Nov 03 02:40:25.228389 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:25.228358    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"]
Nov 03 02:40:25.347392 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:25.347320    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448" podStartSLOduration=2.347304384 podStartE2EDuration="2.347304384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:40:25.344526021 +0000 UTC m=+4904.770132486" watchObservedRunningTime="2026-11-03 02:40:25.347304384 +0000 UTC m=+4904.772910888"
Nov 03 02:40:26.229731 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:26.229692    5508 generic.go:356] "Generic (PLEG): container finished" podID="ce6081b1-f32a-4206-9041-4adb4147dcdf" containerID="5032afd5cc12934bf8e54c5c73cd25b0470de117fa2ff433e23f010c706627a4" exitCode=0
Nov 03 02:40:26.232391 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:26.230590    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448" event={"ID":"ce6081b1-f32a-4206-9041-4adb4147dcdf","Type":"ContainerDied","Data":"5032afd5cc12934bf8e54c5c73cd25b0470de117fa2ff433e23f010c706627a4"}
Nov 03 02:40:26.848517 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:26.848481    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"]
Nov 03 02:40:27.244378 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.243587    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t" podUID="bed779ff-4b38-4116-a5bd-1e07b2e28441" containerName="maas-api" containerID="cri-o://8495fbe805ca8b15c61bc0c510295eeae47c4301570620754afa887ae34a9fa6" gracePeriod=30
Nov 03 02:40:27.414404 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.414373    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"
Nov 03 02:40:27.536380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.536301    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:40:27.542812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.542781    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce6081b1-f32a-4206-9041-4adb4147dcdf-kube-api-access-5qs6v\" (UniqueName: \"kubernetes.io/projected/ce6081b1-f32a-4206-9041-4adb4147dcdf-kube-api-access-5qs6v\") pod \"ce6081b1-f32a-4206-9041-4adb4147dcdf\" (UID: \"ce6081b1-f32a-4206-9041-4adb4147dcdf\") "
Nov 03 02:40:27.542974 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.542870    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-xks-ca\") pod \"ce6081b1-f32a-4206-9041-4adb4147dcdf\" (UID: \"ce6081b1-f32a-4206-9041-4adb4147dcdf\") "
Nov 03 02:40:27.542974 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.542910    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-openshift-ca\") pod \"ce6081b1-f32a-4206-9041-4adb4147dcdf\" (UID: \"ce6081b1-f32a-4206-9041-4adb4147dcdf\") "
Nov 03 02:40:27.543604 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.543566    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-openshift-ca" pod "ce6081b1-f32a-4206-9041-4adb4147dcdf" (UID: "ce6081b1-f32a-4206-9041-4adb4147dcdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:40:27.545754 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.545727    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6081b1-f32a-4206-9041-4adb4147dcdf-kube-api-access-5qs6v" pod "ce6081b1-f32a-4206-9041-4adb4147dcdf" (UID: "ce6081b1-f32a-4206-9041-4adb4147dcdf"). InnerVolumeSpecName "kube-api-access-5qs6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:40:27.545993 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.545972    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-xks-ca" pod "ce6081b1-f32a-4206-9041-4adb4147dcdf" (UID: "ce6081b1-f32a-4206-9041-4adb4147dcdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:40:27.644086 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.644052    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed779ff-4b38-4116-a5bd-1e07b2e28441-kube-api-access-8mwbw\" (UniqueName: \"kubernetes.io/projected/bed779ff-4b38-4116-a5bd-1e07b2e28441-kube-api-access-8mwbw\") pod \"bed779ff-4b38-4116-a5bd-1e07b2e28441\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") "
Nov 03 02:40:27.644310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.644150    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-maas-api-tls\") pod \"bed779ff-4b38-4116-a5bd-1e07b2e28441\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") "
Nov 03 02:40:27.644310 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.644251    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-metrics-certs\") pod \"bed779ff-4b38-4116-a5bd-1e07b2e28441\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") "
Nov 03 02:40:27.644451 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.644395    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed779ff-4b38-4116-a5bd-1e07b2e28441-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bed779ff-4b38-4116-a5bd-1e07b2e28441-openshift-service-ca\") pod \"bed779ff-4b38-4116-a5bd-1e07b2e28441\" (UID: \"bed779ff-4b38-4116-a5bd-1e07b2e28441\") "
Nov 03 02:40:27.644876 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.644849    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qs6v\" (UniqueName: \"kubernetes.io/projected/ce6081b1-f32a-4206-9041-4adb4147dcdf-kube-api-access-5qs6v\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:27.645037 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.644879    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:27.645037 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.644895    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce6081b1-f32a-4206-9041-4adb4147dcdf-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:27.645169 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.645038    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed779ff-4b38-4116-a5bd-1e07b2e28441-openshift-service-ca" pod "bed779ff-4b38-4116-a5bd-1e07b2e28441" (UID: "bed779ff-4b38-4116-a5bd-1e07b2e28441"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:40:27.646639 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.646612    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-metrics-certs" pod "bed779ff-4b38-4116-a5bd-1e07b2e28441" (UID: "bed779ff-4b38-4116-a5bd-1e07b2e28441"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:40:27.646948 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.646922    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-maas-api-tls" pod "bed779ff-4b38-4116-a5bd-1e07b2e28441" (UID: "bed779ff-4b38-4116-a5bd-1e07b2e28441"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:40:27.647062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.647014    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed779ff-4b38-4116-a5bd-1e07b2e28441-kube-api-access-8mwbw" pod "bed779ff-4b38-4116-a5bd-1e07b2e28441" (UID: "bed779ff-4b38-4116-a5bd-1e07b2e28441"). InnerVolumeSpecName "kube-api-access-8mwbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:40:27.746251 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.746157    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mwbw\" (UniqueName: \"kubernetes.io/projected/bed779ff-4b38-4116-a5bd-1e07b2e28441-kube-api-access-8mwbw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:27.746251 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.746201    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:27.746251 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.746215    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bed779ff-4b38-4116-a5bd-1e07b2e28441-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:27.746251 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:27.746231    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bed779ff-4b38-4116-a5bd-1e07b2e28441-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:28.251116 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:28.251079    5508 generic.go:356] "Generic (PLEG): container finished" podID="bed779ff-4b38-4116-a5bd-1e07b2e28441" containerID="8495fbe805ca8b15c61bc0c510295eeae47c4301570620754afa887ae34a9fa6" exitCode=0
Nov 03 02:40:28.251671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:28.251146    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"
Nov 03 02:40:28.251671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:28.251170    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t" event={"ID":"bed779ff-4b38-4116-a5bd-1e07b2e28441","Type":"ContainerDied","Data":"8495fbe805ca8b15c61bc0c510295eeae47c4301570620754afa887ae34a9fa6"}
Nov 03 02:40:28.251671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:28.251224    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t" event={"ID":"bed779ff-4b38-4116-a5bd-1e07b2e28441","Type":"ContainerDied","Data":"ac5486cab1a6c1426ce61651cd2d0dcc69cf0f414d7510cb8884ba94ae466539"}
Nov 03 02:40:28.251671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:28.251246    5508 scope.go:122] "RemoveContainer" containerID="8495fbe805ca8b15c61bc0c510295eeae47c4301570620754afa887ae34a9fa6"
Nov 03 02:40:28.253105 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:28.252798    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"
Nov 03 02:40:28.253105 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:28.252814    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448" event={"ID":"ce6081b1-f32a-4206-9041-4adb4147dcdf","Type":"ContainerDied","Data":"e8157495708437e033220aa6c6e7f92b4904447a390808379706d8b7a96ccb0f"}
Nov 03 02:40:28.253105 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:28.252856    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8157495708437e033220aa6c6e7f92b4904447a390808379706d8b7a96ccb0f"
Nov 03 02:40:28.263603 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:28.263572    5508 scope.go:122] "RemoveContainer" containerID="8495fbe805ca8b15c61bc0c510295eeae47c4301570620754afa887ae34a9fa6"
Nov 03 02:40:28.263902 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:40:28.263882    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8495fbe805ca8b15c61bc0c510295eeae47c4301570620754afa887ae34a9fa6\": container with ID starting with 8495fbe805ca8b15c61bc0c510295eeae47c4301570620754afa887ae34a9fa6 not found: ID does not exist" containerID="8495fbe805ca8b15c61bc0c510295eeae47c4301570620754afa887ae34a9fa6"
Nov 03 02:40:28.263982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:28.263915    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8495fbe805ca8b15c61bc0c510295eeae47c4301570620754afa887ae34a9fa6"} err="failed to get container status \"8495fbe805ca8b15c61bc0c510295eeae47c4301570620754afa887ae34a9fa6\": rpc error: code = NotFound desc = could not find container \"8495fbe805ca8b15c61bc0c510295eeae47c4301570620754afa887ae34a9fa6\": container with ID starting with 8495fbe805ca8b15c61bc0c510295eeae47c4301570620754afa887ae34a9fa6 not found: ID does not exist"
Nov 03 02:40:28.454554 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:28.454513    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"]
Nov 03 02:40:28.500640 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:28.500599    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9576a-768cbd74df-gmv6t"]
Nov 03 02:40:28.694280 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:28.694238    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed779ff-4b38-4116-a5bd-1e07b2e28441" path="/var/lib/kubelet/pods/bed779ff-4b38-4116-a5bd-1e07b2e28441/volumes"
Nov 03 02:40:30.129137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:30.129094    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7" podUID="3588e919-ccba-4fe9-8e34-30efb454c3b2" containerName="payload-processing" containerID="cri-o://1d689822fb8f00ace1a046f1759b407c2c7f9b9935b17a50d739d43868d6617c" gracePeriod=25
Nov 03 02:40:30.228656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:30.228611    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4" podUID="3b2deab8-243a-4592-a30c-c521fe20db90" containerName="payload-pre-processing" containerID="cri-o://0871fe04083f43fa50625bb8921e0bf570683638867d4b1bb6fc95bc8fd1db04" gracePeriod=25
Nov 03 02:40:34.537117 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:34.537072    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4" podUID="3b2deab8-243a-4592-a30c-c521fe20db90" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:40:34.537695 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:34.537090    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7" podUID="3588e919-ccba-4fe9-8e34-30efb454c3b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:40:35.046463 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:35.046417    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"]
Nov 03 02:40:35.096391 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:35.096336    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a524fe7a-z2629"]
Nov 03 02:40:36.693198 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:36.693162    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0411036c-38ea-4b9e-be73-475c2b9491c7" path="/var/lib/kubelet/pods/0411036c-38ea-4b9e-be73-475c2b9491c7/volumes"
Nov 03 02:40:37.888828 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:37.888781    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:40:40.364829 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.364788    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"]
Nov 03 02:40:40.365547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.365523    5508 cpu_manager.go:414] "containerMap: removing container" podUID="bed779ff-4b38-4116-a5bd-1e07b2e28441" containerName="maas-api"
Nov 03 02:40:40.365656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.365550    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed779ff-4b38-4116-a5bd-1e07b2e28441" containerName="maas-api"
Nov 03 02:40:40.365656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.365611    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ce6081b1-f32a-4206-9041-4adb4147dcdf" containerName="revoke-keys"
Nov 03 02:40:40.365656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.365620    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6081b1-f32a-4206-9041-4adb4147dcdf" containerName="revoke-keys"
Nov 03 02:40:40.365825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.365750    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed779ff-4b38-4116-a5bd-1e07b2e28441" containerName="maas-api"
Nov 03 02:40:40.365825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.365773    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6081b1-f32a-4206-9041-4adb4147dcdf" containerName="revoke-keys"
Nov 03 02:40:40.370955 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.370930    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"
Nov 03 02:40:40.424875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.424831    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"]
Nov 03 02:40:40.507102 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.507063    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:40:40.507102 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.507083    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:40:40.533075 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.533038    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c63fcec7-wwmrg\" (UID: \"312d031e-0dcd-4f2d-b648-7037d07e75a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"
Nov 03 02:40:40.533265 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.533098    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzrmz\" (UniqueName: \"kubernetes.io/projected/312d031e-0dcd-4f2d-b648-7037d07e75a8-kube-api-access-qzrmz\") pod \"maas-api-revoke-sub-c63fcec7-wwmrg\" (UID: \"312d031e-0dcd-4f2d-b648-7037d07e75a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"
Nov 03 02:40:40.533265 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.533189    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c63fcec7-wwmrg\" (UID: \"312d031e-0dcd-4f2d-b648-7037d07e75a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"
Nov 03 02:40:40.634503 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.634412    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c63fcec7-wwmrg\" (UID: \"312d031e-0dcd-4f2d-b648-7037d07e75a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"
Nov 03 02:40:40.634681 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.634580    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrmz\" (UniqueName: \"kubernetes.io/projected/312d031e-0dcd-4f2d-b648-7037d07e75a8-kube-api-access-qzrmz\") pod \"maas-api-revoke-sub-c63fcec7-wwmrg\" (UID: \"312d031e-0dcd-4f2d-b648-7037d07e75a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"
Nov 03 02:40:40.634681 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.634627    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c63fcec7-wwmrg\" (UID: \"312d031e-0dcd-4f2d-b648-7037d07e75a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"
Nov 03 02:40:40.635180 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.635145    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c63fcec7-wwmrg\" (UID: \"312d031e-0dcd-4f2d-b648-7037d07e75a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"
Nov 03 02:40:40.637117 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.637089    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c63fcec7-wwmrg\" (UID: \"312d031e-0dcd-4f2d-b648-7037d07e75a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"
Nov 03 02:40:40.696005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:40.695973    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzrmz\" (UniqueName: \"kubernetes.io/projected/312d031e-0dcd-4f2d-b648-7037d07e75a8-kube-api-access-qzrmz\") pod \"maas-api-revoke-sub-c63fcec7-wwmrg\" (UID: \"312d031e-0dcd-4f2d-b648-7037d07e75a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"
Nov 03 02:40:41.033291 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:41.033250    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:40:41.035237 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:41.035208    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"
Nov 03 02:40:41.416127 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:41.416095    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"]
Nov 03 02:40:41.422554 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:40:41.422517    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312d031e_0dcd_4f2d_b648_7037d07e75a8.slice/crio-3930c9a47b14996bc9a80def9e284920341ac296003ca69ddaa94f76485af6c1 WatchSource:0}: Error finding container 3930c9a47b14996bc9a80def9e284920341ac296003ca69ddaa94f76485af6c1: Status 404 returned error can't find the container with id 3930c9a47b14996bc9a80def9e284920341ac296003ca69ddaa94f76485af6c1
Nov 03 02:40:42.369662 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:42.369629    5508 generic.go:356] "Generic (PLEG): container finished" podID="312d031e-0dcd-4f2d-b648-7037d07e75a8" containerID="addbfdbbe253cece2819ec244287542f844396881ce7e169d021dedd04e8f3eb" exitCode=0
Nov 03 02:40:42.369849 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:42.369708    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg" event={"ID":"312d031e-0dcd-4f2d-b648-7037d07e75a8","Type":"ContainerDied","Data":"addbfdbbe253cece2819ec244287542f844396881ce7e169d021dedd04e8f3eb"}
Nov 03 02:40:42.369849 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:42.369764    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg" event={"ID":"312d031e-0dcd-4f2d-b648-7037d07e75a8","Type":"ContainerStarted","Data":"3930c9a47b14996bc9a80def9e284920341ac296003ca69ddaa94f76485af6c1"}
Nov 03 02:40:43.512572 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:43.512531    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"
Nov 03 02:40:43.669287 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:43.669250    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-xks-ca\") pod \"312d031e-0dcd-4f2d-b648-7037d07e75a8\" (UID: \"312d031e-0dcd-4f2d-b648-7037d07e75a8\") "
Nov 03 02:40:43.669530 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:43.669323    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/312d031e-0dcd-4f2d-b648-7037d07e75a8-kube-api-access-qzrmz\" (UniqueName: \"kubernetes.io/projected/312d031e-0dcd-4f2d-b648-7037d07e75a8-kube-api-access-qzrmz\") pod \"312d031e-0dcd-4f2d-b648-7037d07e75a8\" (UID: \"312d031e-0dcd-4f2d-b648-7037d07e75a8\") "
Nov 03 02:40:43.669530 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:43.669501    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-openshift-ca\") pod \"312d031e-0dcd-4f2d-b648-7037d07e75a8\" (UID: \"312d031e-0dcd-4f2d-b648-7037d07e75a8\") "
Nov 03 02:40:43.669864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:43.669840    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-openshift-ca" pod "312d031e-0dcd-4f2d-b648-7037d07e75a8" (UID: "312d031e-0dcd-4f2d-b648-7037d07e75a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:40:43.671395 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:43.671367    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-xks-ca" pod "312d031e-0dcd-4f2d-b648-7037d07e75a8" (UID: "312d031e-0dcd-4f2d-b648-7037d07e75a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:40:43.671511 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:43.671467    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312d031e-0dcd-4f2d-b648-7037d07e75a8-kube-api-access-qzrmz" pod "312d031e-0dcd-4f2d-b648-7037d07e75a8" (UID: "312d031e-0dcd-4f2d-b648-7037d07e75a8"). InnerVolumeSpecName "kube-api-access-qzrmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:40:43.770962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:43.770921    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:43.770962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:43.770954    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/312d031e-0dcd-4f2d-b648-7037d07e75a8-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:43.770962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:43.770964    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzrmz\" (UniqueName: \"kubernetes.io/projected/312d031e-0dcd-4f2d-b648-7037d07e75a8-kube-api-access-qzrmz\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:43.814680 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:40:43.814638    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:40:44.384875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:44.384841    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"
Nov 03 02:40:44.385107 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:44.384877    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg" event={"ID":"312d031e-0dcd-4f2d-b648-7037d07e75a8","Type":"ContainerDied","Data":"3930c9a47b14996bc9a80def9e284920341ac296003ca69ddaa94f76485af6c1"}
Nov 03 02:40:44.385107 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:44.384938    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3930c9a47b14996bc9a80def9e284920341ac296003ca69ddaa94f76485af6c1"
Nov 03 02:40:44.536821 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:44.536786    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7" podUID="3588e919-ccba-4fe9-8e34-30efb454c3b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:40:44.536821 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:44.536813    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4" podUID="3b2deab8-243a-4592-a30c-c521fe20db90" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:40:44.636734 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:44.636633    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"]
Nov 03 02:40:50.426520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.423102    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"
Nov 03 02:40:50.429741 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.429707    5508 generic.go:356] "Generic (PLEG): container finished" podID="3588e919-ccba-4fe9-8e34-30efb454c3b2" containerID="1d689822fb8f00ace1a046f1759b407c2c7f9b9935b17a50d739d43868d6617c" exitCode=0
Nov 03 02:40:50.429905 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.429761    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"
Nov 03 02:40:50.429905 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.429786    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7" event={"ID":"3588e919-ccba-4fe9-8e34-30efb454c3b2","Type":"ContainerDied","Data":"1d689822fb8f00ace1a046f1759b407c2c7f9b9935b17a50d739d43868d6617c"}
Nov 03 02:40:50.429905 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.429849    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7" event={"ID":"3588e919-ccba-4fe9-8e34-30efb454c3b2","Type":"ContainerDied","Data":"ce36d5f3f8913cb7fadc48d9248dc39a4dc376b365f75eb1fd9b803a4104b63a"}
Nov 03 02:40:50.430058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.429877    5508 scope.go:122] "RemoveContainer" containerID="1d689822fb8f00ace1a046f1759b407c2c7f9b9935b17a50d739d43868d6617c"
Nov 03 02:40:50.432031 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.431848    5508 generic.go:356] "Generic (PLEG): container finished" podID="3b2deab8-243a-4592-a30c-c521fe20db90" containerID="0871fe04083f43fa50625bb8921e0bf570683638867d4b1bb6fc95bc8fd1db04" exitCode=0
Nov 03 02:40:50.432031 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.431937    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4" event={"ID":"3b2deab8-243a-4592-a30c-c521fe20db90","Type":"ContainerDied","Data":"0871fe04083f43fa50625bb8921e0bf570683638867d4b1bb6fc95bc8fd1db04"}
Nov 03 02:40:50.441729 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.441705    5508 scope.go:122] "RemoveContainer" containerID="1d689822fb8f00ace1a046f1759b407c2c7f9b9935b17a50d739d43868d6617c"
Nov 03 02:40:50.442120 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:40:50.442100    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d689822fb8f00ace1a046f1759b407c2c7f9b9935b17a50d739d43868d6617c\": container with ID starting with 1d689822fb8f00ace1a046f1759b407c2c7f9b9935b17a50d739d43868d6617c not found: ID does not exist" containerID="1d689822fb8f00ace1a046f1759b407c2c7f9b9935b17a50d739d43868d6617c"
Nov 03 02:40:50.442188 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.442135    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d689822fb8f00ace1a046f1759b407c2c7f9b9935b17a50d739d43868d6617c"} err="failed to get container status \"1d689822fb8f00ace1a046f1759b407c2c7f9b9935b17a50d739d43868d6617c\": rpc error: code = NotFound desc = could not find container \"1d689822fb8f00ace1a046f1759b407c2c7f9b9935b17a50d739d43868d6617c\": container with ID starting with 1d689822fb8f00ace1a046f1759b407c2c7f9b9935b17a50d739d43868d6617c not found: ID does not exist"
Nov 03 02:40:50.449194 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.449145    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3588e919-ccba-4fe9-8e34-30efb454c3b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3588e919-ccba-4fe9-8e34-30efb454c3b2-plugins-config-volume\") pod \"3588e919-ccba-4fe9-8e34-30efb454c3b2\" (UID: \"3588e919-ccba-4fe9-8e34-30efb454c3b2\") "
Nov 03 02:40:50.449340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.449308    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3588e919-ccba-4fe9-8e34-30efb454c3b2-kube-api-access-9xmtg\" (UniqueName: \"kubernetes.io/projected/3588e919-ccba-4fe9-8e34-30efb454c3b2-kube-api-access-9xmtg\") pod \"3588e919-ccba-4fe9-8e34-30efb454c3b2\" (UID: \"3588e919-ccba-4fe9-8e34-30efb454c3b2\") "
Nov 03 02:40:50.450180 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.450148    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3588e919-ccba-4fe9-8e34-30efb454c3b2-plugins-config-volume" pod "3588e919-ccba-4fe9-8e34-30efb454c3b2" (UID: "3588e919-ccba-4fe9-8e34-30efb454c3b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:40:50.451695 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.451644    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3588e919-ccba-4fe9-8e34-30efb454c3b2-kube-api-access-9xmtg" pod "3588e919-ccba-4fe9-8e34-30efb454c3b2" (UID: "3588e919-ccba-4fe9-8e34-30efb454c3b2"). InnerVolumeSpecName "kube-api-access-9xmtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:40:50.512854 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.512826    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"
Nov 03 02:40:50.550253 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.550212    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b2deab8-243a-4592-a30c-c521fe20db90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2deab8-243a-4592-a30c-c521fe20db90-plugins-config-volume\") pod \"3b2deab8-243a-4592-a30c-c521fe20db90\" (UID: \"3b2deab8-243a-4592-a30c-c521fe20db90\") "
Nov 03 02:40:50.550440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.550379    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b2deab8-243a-4592-a30c-c521fe20db90-kube-api-access-ngq4l\" (UniqueName: \"kubernetes.io/projected/3b2deab8-243a-4592-a30c-c521fe20db90-kube-api-access-ngq4l\") pod \"3b2deab8-243a-4592-a30c-c521fe20db90\" (UID: \"3b2deab8-243a-4592-a30c-c521fe20db90\") "
Nov 03 02:40:50.551008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.550663    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b2deab8-243a-4592-a30c-c521fe20db90-plugins-config-volume" pod "3b2deab8-243a-4592-a30c-c521fe20db90" (UID: "3b2deab8-243a-4592-a30c-c521fe20db90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:40:50.551008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.550947    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3588e919-ccba-4fe9-8e34-30efb454c3b2-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:50.551008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.550968    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b2deab8-243a-4592-a30c-c521fe20db90-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:50.551008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.550983    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xmtg\" (UniqueName: \"kubernetes.io/projected/3588e919-ccba-4fe9-8e34-30efb454c3b2-kube-api-access-9xmtg\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:50.552685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.552660    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2deab8-243a-4592-a30c-c521fe20db90-kube-api-access-ngq4l" pod "3b2deab8-243a-4592-a30c-c521fe20db90" (UID: "3b2deab8-243a-4592-a30c-c521fe20db90"). InnerVolumeSpecName "kube-api-access-ngq4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:40:50.652049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.651966    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngq4l\" (UniqueName: \"kubernetes.io/projected/3b2deab8-243a-4592-a30c-c521fe20db90-kube-api-access-ngq4l\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:50.848601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.848557    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"]
Nov 03 02:40:50.851441 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:50.851412    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-41aa5f-84c4859f45-7vdp7"]
Nov 03 02:40:51.441514 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:51.441415    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4" event={"ID":"3b2deab8-243a-4592-a30c-c521fe20db90","Type":"ContainerDied","Data":"24bfdce41df33b857fc7436fdcbd3e49da786ad7eb7c64f94a2b88e6c3a8f74c"}
Nov 03 02:40:51.441514 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:51.441486    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"
Nov 03 02:40:51.441972 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:51.441523    5508 scope.go:122] "RemoveContainer" containerID="0871fe04083f43fa50625bb8921e0bf570683638867d4b1bb6fc95bc8fd1db04"
Nov 03 02:40:51.599026 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:51.598988    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"]
Nov 03 02:40:51.644867 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:51.644823    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-41aa5f-9c9957f-mvhv4"]
Nov 03 02:40:52.692255 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.692220    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3588e919-ccba-4fe9-8e34-30efb454c3b2" path="/var/lib/kubelet/pods/3588e919-ccba-4fe9-8e34-30efb454c3b2/volumes"
Nov 03 02:40:52.692701 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.692685    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b2deab8-243a-4592-a30c-c521fe20db90" path="/var/lib/kubelet/pods/3b2deab8-243a-4592-a30c-c521fe20db90/volumes"
Nov 03 02:40:52.750014 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.749983    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"]
Nov 03 02:40:52.750472 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.750455    5508 cpu_manager.go:414] "containerMap: removing container" podUID="312d031e-0dcd-4f2d-b648-7037d07e75a8" containerName="revoke-keys"
Nov 03 02:40:52.750520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.750474    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="312d031e-0dcd-4f2d-b648-7037d07e75a8" containerName="revoke-keys"
Nov 03 02:40:52.750520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.750490    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3b2deab8-243a-4592-a30c-c521fe20db90" containerName="payload-pre-processing"
Nov 03 02:40:52.750520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.750496    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b2deab8-243a-4592-a30c-c521fe20db90" containerName="payload-pre-processing"
Nov 03 02:40:52.750626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.750530    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3588e919-ccba-4fe9-8e34-30efb454c3b2" containerName="payload-processing"
Nov 03 02:40:52.750626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.750538    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3588e919-ccba-4fe9-8e34-30efb454c3b2" containerName="payload-processing"
Nov 03 02:40:52.750626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.750609    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="312d031e-0dcd-4f2d-b648-7037d07e75a8" containerName="revoke-keys"
Nov 03 02:40:52.750626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.750619    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b2deab8-243a-4592-a30c-c521fe20db90" containerName="payload-pre-processing"
Nov 03 02:40:52.750763 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.750630    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3588e919-ccba-4fe9-8e34-30efb454c3b2" containerName="payload-processing"
Nov 03 02:40:52.755274 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.755252    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"
Nov 03 02:40:52.816833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.816800    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:40:52.854222 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.854190    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:40:52.867469 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.867439    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"]
Nov 03 02:40:52.877650 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.877621    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78804323-a28a-441e-a1c7-506a502e0469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab02f30b-jh2ps\" (UID: \"78804323-a28a-441e-a1c7-506a502e0469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"
Nov 03 02:40:52.877833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.877697    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28gnr\" (UniqueName: \"kubernetes.io/projected/78804323-a28a-441e-a1c7-506a502e0469-kube-api-access-28gnr\") pod \"maas-api-revoke-sub-ab02f30b-jh2ps\" (UID: \"78804323-a28a-441e-a1c7-506a502e0469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"
Nov 03 02:40:52.877935 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.877870    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78804323-a28a-441e-a1c7-506a502e0469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab02f30b-jh2ps\" (UID: \"78804323-a28a-441e-a1c7-506a502e0469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"
Nov 03 02:40:52.978646 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.978547    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78804323-a28a-441e-a1c7-506a502e0469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab02f30b-jh2ps\" (UID: \"78804323-a28a-441e-a1c7-506a502e0469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"
Nov 03 02:40:52.978646 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.978615    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78804323-a28a-441e-a1c7-506a502e0469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab02f30b-jh2ps\" (UID: \"78804323-a28a-441e-a1c7-506a502e0469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"
Nov 03 02:40:52.979028 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.978678    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28gnr\" (UniqueName: \"kubernetes.io/projected/78804323-a28a-441e-a1c7-506a502e0469-kube-api-access-28gnr\") pod \"maas-api-revoke-sub-ab02f30b-jh2ps\" (UID: \"78804323-a28a-441e-a1c7-506a502e0469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"
Nov 03 02:40:52.979407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.979383    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78804323-a28a-441e-a1c7-506a502e0469-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab02f30b-jh2ps\" (UID: \"78804323-a28a-441e-a1c7-506a502e0469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"
Nov 03 02:40:52.981006 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:52.980982    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78804323-a28a-441e-a1c7-506a502e0469-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab02f30b-jh2ps\" (UID: \"78804323-a28a-441e-a1c7-506a502e0469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"
Nov 03 02:40:53.036101 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:53.036061    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28gnr\" (UniqueName: \"kubernetes.io/projected/78804323-a28a-441e-a1c7-506a502e0469-kube-api-access-28gnr\") pod \"maas-api-revoke-sub-ab02f30b-jh2ps\" (UID: \"78804323-a28a-441e-a1c7-506a502e0469\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"
Nov 03 02:40:53.068251 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:53.068223    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"
Nov 03 02:40:53.447953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:53.447919    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"]
Nov 03 02:40:53.454427 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:40:53.454400    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78804323_a28a_441e_a1c7_506a502e0469.slice/crio-a2215906cb801a6c8d1a182270ff60e8f2cc85a715f2099333d2004d3eeb5586 WatchSource:0}: Error finding container a2215906cb801a6c8d1a182270ff60e8f2cc85a715f2099333d2004d3eeb5586: Status 404 returned error can't find the container with id a2215906cb801a6c8d1a182270ff60e8f2cc85a715f2099333d2004d3eeb5586
Nov 03 02:40:54.293439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:54.293405    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"]
Nov 03 02:40:54.463638 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:54.463604    5508 generic.go:356] "Generic (PLEG): container finished" podID="78804323-a28a-441e-a1c7-506a502e0469" containerID="d692eafaa37cd0d37620109274d05f440f86ddb2b5974b9c3318997efe2840d0" exitCode=0
Nov 03 02:40:54.463816 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:54.463692    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps" event={"ID":"78804323-a28a-441e-a1c7-506a502e0469","Type":"ContainerDied","Data":"d692eafaa37cd0d37620109274d05f440f86ddb2b5974b9c3318997efe2840d0"}
Nov 03 02:40:54.463816 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:54.463745    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps" event={"ID":"78804323-a28a-441e-a1c7-506a502e0469","Type":"ContainerStarted","Data":"a2215906cb801a6c8d1a182270ff60e8f2cc85a715f2099333d2004d3eeb5586"}
Nov 03 02:40:55.633141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:55.632325    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"
Nov 03 02:40:55.716281 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:55.716247    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78804323-a28a-441e-a1c7-506a502e0469-kube-api-access-28gnr\" (UniqueName: \"kubernetes.io/projected/78804323-a28a-441e-a1c7-506a502e0469-kube-api-access-28gnr\") pod \"78804323-a28a-441e-a1c7-506a502e0469\" (UID: \"78804323-a28a-441e-a1c7-506a502e0469\") "
Nov 03 02:40:55.716520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:55.716305    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78804323-a28a-441e-a1c7-506a502e0469-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78804323-a28a-441e-a1c7-506a502e0469-maas-api-xks-ca\") pod \"78804323-a28a-441e-a1c7-506a502e0469\" (UID: \"78804323-a28a-441e-a1c7-506a502e0469\") "
Nov 03 02:40:55.716520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:55.716326    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78804323-a28a-441e-a1c7-506a502e0469-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78804323-a28a-441e-a1c7-506a502e0469-maas-api-openshift-ca\") pod \"78804323-a28a-441e-a1c7-506a502e0469\" (UID: \"78804323-a28a-441e-a1c7-506a502e0469\") "
Nov 03 02:40:55.716703 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:55.716677    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78804323-a28a-441e-a1c7-506a502e0469-maas-api-openshift-ca" pod "78804323-a28a-441e-a1c7-506a502e0469" (UID: "78804323-a28a-441e-a1c7-506a502e0469"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:40:55.718468 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:55.718448    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78804323-a28a-441e-a1c7-506a502e0469-maas-api-xks-ca" pod "78804323-a28a-441e-a1c7-506a502e0469" (UID: "78804323-a28a-441e-a1c7-506a502e0469"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:40:55.718567 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:55.718551    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78804323-a28a-441e-a1c7-506a502e0469-kube-api-access-28gnr" pod "78804323-a28a-441e-a1c7-506a502e0469" (UID: "78804323-a28a-441e-a1c7-506a502e0469"). InnerVolumeSpecName "kube-api-access-28gnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:40:55.830306 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:55.830259    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28gnr\" (UniqueName: \"kubernetes.io/projected/78804323-a28a-441e-a1c7-506a502e0469-kube-api-access-28gnr\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:55.830623 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:55.830596    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78804323-a28a-441e-a1c7-506a502e0469-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:55.830790 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:55.830769    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78804323-a28a-441e-a1c7-506a502e0469-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:40:56.477645 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:56.477613    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"
Nov 03 02:40:56.477844 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:56.477665    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps" event={"ID":"78804323-a28a-441e-a1c7-506a502e0469","Type":"ContainerDied","Data":"a2215906cb801a6c8d1a182270ff60e8f2cc85a715f2099333d2004d3eeb5586"}
Nov 03 02:40:56.477844 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:56.477724    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2215906cb801a6c8d1a182270ff60e8f2cc85a715f2099333d2004d3eeb5586"
Nov 03 02:40:59.293970 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:59.293918    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j" podUID="6684afe1-182b-4c8b-b2ed-d9491cfd68a8" containerName="payload-processing" containerID="cri-o://20aa4477277de3f3f8fbd2bdfbb88aeab2dae84247695eed5d5a1111cebf6e92" gracePeriod=25
Nov 03 02:40:59.664187 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:59.664146    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main" containerID="cri-o://e6ab55afcee4122d6c7698fd1dc1819aeb2d2192b90c28e49281dba99f5bc2d0" gracePeriod=45
Nov 03 02:40:59.939084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:40:59.939052    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 02:41:00.099336 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.099294    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-model-cache\") pod \"5a4243b3-d50b-4995-b51d-29174542a75d\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") "
Nov 03 02:41:00.099525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.099502    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-tmp-dir\") pod \"5a4243b3-d50b-4995-b51d-29174542a75d\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") "
Nov 03 02:41:00.099595 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.099552    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-home\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-home\") pod \"5a4243b3-d50b-4995-b51d-29174542a75d\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") "
Nov 03 02:41:00.099653 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.099586    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-model-cache" pod "5a4243b3-d50b-4995-b51d-29174542a75d" (UID: "5a4243b3-d50b-4995-b51d-29174542a75d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:00.099775 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.099755    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a4243b3-d50b-4995-b51d-29174542a75d-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4243b3-d50b-4995-b51d-29174542a75d-tls-certs\") pod \"5a4243b3-d50b-4995-b51d-29174542a75d\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") "
Nov 03 02:41:00.099775 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.099752    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-tmp-dir" pod "5a4243b3-d50b-4995-b51d-29174542a75d" (UID: "5a4243b3-d50b-4995-b51d-29174542a75d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:00.099978 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.099788    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4243b3-d50b-4995-b51d-29174542a75d-kube-api-access-ftxd4\" (UniqueName: \"kubernetes.io/projected/5a4243b3-d50b-4995-b51d-29174542a75d-kube-api-access-ftxd4\") pod \"5a4243b3-d50b-4995-b51d-29174542a75d\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") "
Nov 03 02:41:00.099978 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.099816    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-dshm\") pod \"5a4243b3-d50b-4995-b51d-29174542a75d\" (UID: \"5a4243b3-d50b-4995-b51d-29174542a75d\") "
Nov 03 02:41:00.099978 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.099813    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-home" pod "5a4243b3-d50b-4995-b51d-29174542a75d" (UID: "5a4243b3-d50b-4995-b51d-29174542a75d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:00.100178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.100161    5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-model-cache\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:00.100229 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.100186    5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-tmp-dir\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:00.100229 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.100202    5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-home\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:00.102028 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.102000    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-dshm" pod "5a4243b3-d50b-4995-b51d-29174542a75d" (UID: "5a4243b3-d50b-4995-b51d-29174542a75d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:00.103540 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.103506    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4243b3-d50b-4995-b51d-29174542a75d-kube-api-access-ftxd4" pod "5a4243b3-d50b-4995-b51d-29174542a75d" (UID: "5a4243b3-d50b-4995-b51d-29174542a75d"). InnerVolumeSpecName "kube-api-access-ftxd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:41:00.103668 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.103585    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4243b3-d50b-4995-b51d-29174542a75d-tls-certs" pod "5a4243b3-d50b-4995-b51d-29174542a75d" (UID: "5a4243b3-d50b-4995-b51d-29174542a75d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:41:00.200954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.200879    5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4243b3-d50b-4995-b51d-29174542a75d-tls-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:00.200954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.200906    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftxd4\" (UniqueName: \"kubernetes.io/projected/5a4243b3-d50b-4995-b51d-29174542a75d-kube-api-access-ftxd4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:00.200954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.200917    5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4243b3-d50b-4995-b51d-29174542a75d-dshm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:00.502506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.502418    5508 generic.go:356] "Generic (PLEG): container finished" podID="5a4243b3-d50b-4995-b51d-29174542a75d" containerID="e6ab55afcee4122d6c7698fd1dc1819aeb2d2192b90c28e49281dba99f5bc2d0" exitCode=0
Nov 03 02:41:00.502506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.502463    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"
Nov 03 02:41:00.502506 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.502495    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" event={"ID":"5a4243b3-d50b-4995-b51d-29174542a75d","Type":"ContainerDied","Data":"e6ab55afcee4122d6c7698fd1dc1819aeb2d2192b90c28e49281dba99f5bc2d0"}
Nov 03 02:41:00.503033 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.502528    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws" event={"ID":"5a4243b3-d50b-4995-b51d-29174542a75d","Type":"ContainerDied","Data":"8d1a088dd6c2a1d757d9a60d6858dfc30229d64db737750c101c751de03d9da1"}
Nov 03 02:41:00.503033 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.502546    5508 scope.go:122] "RemoveContainer" containerID="e6ab55afcee4122d6c7698fd1dc1819aeb2d2192b90c28e49281dba99f5bc2d0"
Nov 03 02:41:00.512619 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.512596    5508 scope.go:122] "RemoveContainer" containerID="e6ab55afcee4122d6c7698fd1dc1819aeb2d2192b90c28e49281dba99f5bc2d0"
Nov 03 02:41:00.512928 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:41:00.512899    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ab55afcee4122d6c7698fd1dc1819aeb2d2192b90c28e49281dba99f5bc2d0\": container with ID starting with e6ab55afcee4122d6c7698fd1dc1819aeb2d2192b90c28e49281dba99f5bc2d0 not found: ID does not exist" containerID="e6ab55afcee4122d6c7698fd1dc1819aeb2d2192b90c28e49281dba99f5bc2d0"
Nov 03 02:41:00.513027 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.512928    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ab55afcee4122d6c7698fd1dc1819aeb2d2192b90c28e49281dba99f5bc2d0"} err="failed to get container status \"e6ab55afcee4122d6c7698fd1dc1819aeb2d2192b90c28e49281dba99f5bc2d0\": rpc error: code = NotFound desc = could not find container \"e6ab55afcee4122d6c7698fd1dc1819aeb2d2192b90c28e49281dba99f5bc2d0\": container with ID starting with e6ab55afcee4122d6c7698fd1dc1819aeb2d2192b90c28e49281dba99f5bc2d0 not found: ID does not exist"
Nov 03 02:41:00.691569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.691541    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"]
Nov 03 02:41:00.731109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:00.731077    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0bf34a/distinct-2-w4-0bf34a-kserve-678f464dbd-cwmws"]
Nov 03 02:41:01.577285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:01.577235    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j" podUID="6684afe1-182b-4c8b-b2ed-d9491cfd68a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:41:02.691706 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:02.691672    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" path="/var/lib/kubelet/pods/5a4243b3-d50b-4995-b51d-29174542a75d/volumes"
Nov 03 02:41:07.205019 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:07.204977    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:41:11.578368 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:11.578306    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j" podUID="6684afe1-182b-4c8b-b2ed-d9491cfd68a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:41:13.833184 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:41:13.833134    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:41:19.566991 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.566922    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"
Nov 03 02:41:19.654644 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.654608    5508 generic.go:356] "Generic (PLEG): container finished" podID="6684afe1-182b-4c8b-b2ed-d9491cfd68a8" containerID="20aa4477277de3f3f8fbd2bdfbb88aeab2dae84247695eed5d5a1111cebf6e92" exitCode=0
Nov 03 02:41:19.654838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.654656    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"
Nov 03 02:41:19.654838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.654667    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j" event={"ID":"6684afe1-182b-4c8b-b2ed-d9491cfd68a8","Type":"ContainerDied","Data":"20aa4477277de3f3f8fbd2bdfbb88aeab2dae84247695eed5d5a1111cebf6e92"}
Nov 03 02:41:19.654838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.654711    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j" event={"ID":"6684afe1-182b-4c8b-b2ed-d9491cfd68a8","Type":"ContainerDied","Data":"3abbbdb3f0809d35f9ff446902e9758ef4b23d182bfa9a0109194ea8aa2492c7"}
Nov 03 02:41:19.654838 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.654735    5508 scope.go:122] "RemoveContainer" containerID="20aa4477277de3f3f8fbd2bdfbb88aeab2dae84247695eed5d5a1111cebf6e92"
Nov 03 02:41:19.661180 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.661153    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-plugins-config-volume\") pod \"6684afe1-182b-4c8b-b2ed-d9491cfd68a8\" (UID: \"6684afe1-182b-4c8b-b2ed-d9491cfd68a8\") "
Nov 03 02:41:19.661328 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.661256    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-kube-api-access-crvt6\" (UniqueName: \"kubernetes.io/projected/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-kube-api-access-crvt6\") pod \"6684afe1-182b-4c8b-b2ed-d9491cfd68a8\" (UID: \"6684afe1-182b-4c8b-b2ed-d9491cfd68a8\") "
Nov 03 02:41:19.661573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.661552    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-plugins-config-volume" pod "6684afe1-182b-4c8b-b2ed-d9491cfd68a8" (UID: "6684afe1-182b-4c8b-b2ed-d9491cfd68a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:41:19.661758 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.661742    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:19.663709 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.663682    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-kube-api-access-crvt6" pod "6684afe1-182b-4c8b-b2ed-d9491cfd68a8" (UID: "6684afe1-182b-4c8b-b2ed-d9491cfd68a8"). InnerVolumeSpecName "kube-api-access-crvt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:41:19.665180 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.665159    5508 scope.go:122] "RemoveContainer" containerID="20aa4477277de3f3f8fbd2bdfbb88aeab2dae84247695eed5d5a1111cebf6e92"
Nov 03 02:41:19.665556 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:41:19.665533    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20aa4477277de3f3f8fbd2bdfbb88aeab2dae84247695eed5d5a1111cebf6e92\": container with ID starting with 20aa4477277de3f3f8fbd2bdfbb88aeab2dae84247695eed5d5a1111cebf6e92 not found: ID does not exist" containerID="20aa4477277de3f3f8fbd2bdfbb88aeab2dae84247695eed5d5a1111cebf6e92"
Nov 03 02:41:19.665621 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.665567    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20aa4477277de3f3f8fbd2bdfbb88aeab2dae84247695eed5d5a1111cebf6e92"} err="failed to get container status \"20aa4477277de3f3f8fbd2bdfbb88aeab2dae84247695eed5d5a1111cebf6e92\": rpc error: code = NotFound desc = could not find container \"20aa4477277de3f3f8fbd2bdfbb88aeab2dae84247695eed5d5a1111cebf6e92\": container with ID starting with 20aa4477277de3f3f8fbd2bdfbb88aeab2dae84247695eed5d5a1111cebf6e92 not found: ID does not exist"
Nov 03 02:41:19.763039 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:19.763004    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crvt6\" (UniqueName: \"kubernetes.io/projected/6684afe1-182b-4c8b-b2ed-d9491cfd68a8-kube-api-access-crvt6\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:20.070780 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:20.070740    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"]
Nov 03 02:41:20.116817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:20.116781    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-0bf34a-5659d86f7f-qn52j"]
Nov 03 02:41:20.690695 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:20.690657    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6684afe1-182b-4c8b-b2ed-d9491cfd68a8" path="/var/lib/kubelet/pods/6684afe1-182b-4c8b-b2ed-d9491cfd68a8/volumes"
Nov 03 02:41:21.238155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:21.238117    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"]
Nov 03 02:41:21.284540 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:21.284504    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"]
Nov 03 02:41:21.287237 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:21.287206    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"]
Nov 03 02:41:36.258730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.258688    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main" containerID="cri-o://33a8dbc71ed34f025fc6e8b2a20928178815895a0722d8d269634f69158deb17" gracePeriod=45
Nov 03 02:41:36.314769 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.314711    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" containerID="cri-o://acd4fd150c0d041dd62ce2741b856cb315878e487105ed126f56d91b769e1363" gracePeriod=45
Nov 03 02:41:36.315771 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.315742    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main" containerID="cri-o://e338c94661370d4f774e35063c5b6d0967fc1a5fadca152d02e00fdb12715b80" gracePeriod=45
Nov 03 02:41:36.322555 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.322517    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": dial tcp 10.133.0.62:8000: connect: connection refused"
Nov 03 02:41:36.597748 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.597711    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 02:41:36.632942 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.632917    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 02:41:36.636094 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.636070    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 02:41:36.666958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.666924    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-tmp-dir\") pod \"2d613494-eedc-4a88-86c3-d076b634c9e1\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") "
Nov 03 02:41:36.667167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.666981    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-dshm\") pod \"2d613494-eedc-4a88-86c3-d076b634c9e1\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") "
Nov 03 02:41:36.667167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.667012    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-model-cache\") pod \"2d613494-eedc-4a88-86c3-d076b634c9e1\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") "
Nov 03 02:41:36.667167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.667035    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d613494-eedc-4a88-86c3-d076b634c9e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d613494-eedc-4a88-86c3-d076b634c9e1-tls-certs\") pod \"2d613494-eedc-4a88-86c3-d076b634c9e1\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") "
Nov 03 02:41:36.667167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.667121    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-home\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-home\") pod \"2d613494-eedc-4a88-86c3-d076b634c9e1\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") "
Nov 03 02:41:36.667167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.667142    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d613494-eedc-4a88-86c3-d076b634c9e1-kube-api-access-qcf59\" (UniqueName: \"kubernetes.io/projected/2d613494-eedc-4a88-86c3-d076b634c9e1-kube-api-access-qcf59\") pod \"2d613494-eedc-4a88-86c3-d076b634c9e1\" (UID: \"2d613494-eedc-4a88-86c3-d076b634c9e1\") "
Nov 03 02:41:36.667482 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.667270    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-tmp-dir" pod "2d613494-eedc-4a88-86c3-d076b634c9e1" (UID: "2d613494-eedc-4a88-86c3-d076b634c9e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:36.667551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.667537    5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-tmp-dir\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.667611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.667557    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-model-cache" pod "2d613494-eedc-4a88-86c3-d076b634c9e1" (UID: "2d613494-eedc-4a88-86c3-d076b634c9e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:36.667611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.667566    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-home" pod "2d613494-eedc-4a88-86c3-d076b634c9e1" (UID: "2d613494-eedc-4a88-86c3-d076b634c9e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:36.669752 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.669727    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-dshm" pod "2d613494-eedc-4a88-86c3-d076b634c9e1" (UID: "2d613494-eedc-4a88-86c3-d076b634c9e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:36.669887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.669764    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d613494-eedc-4a88-86c3-d076b634c9e1-tls-certs" pod "2d613494-eedc-4a88-86c3-d076b634c9e1" (UID: "2d613494-eedc-4a88-86c3-d076b634c9e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:41:36.669887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.669772    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d613494-eedc-4a88-86c3-d076b634c9e1-kube-api-access-qcf59" pod "2d613494-eedc-4a88-86c3-d076b634c9e1" (UID: "2d613494-eedc-4a88-86c3-d076b634c9e1"). InnerVolumeSpecName "kube-api-access-qcf59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:41:36.767950 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.767912    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tls-certs\") pod \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") "
Nov 03 02:41:36.768156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768119    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-model-cache\") pod \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") "
Nov 03 02:41:36.768156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768143    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-dshm\") pod \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") "
Nov 03 02:41:36.768301 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768166    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tls-certs\" (UniqueName: \"kubernetes.io/secret/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tls-certs\") pod \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") "
Nov 03 02:41:36.768301 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768187    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-home\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-home\") pod \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") "
Nov 03 02:41:36.768301 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768217    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-kube-api-access-8rw2j\" (UniqueName: \"kubernetes.io/projected/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-kube-api-access-8rw2j\") pod \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") "
Nov 03 02:41:36.768301 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768245    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-model-cache\") pod \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") "
Nov 03 02:41:36.768569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768510    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-dshm\") pod \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") "
Nov 03 02:41:36.768569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768537    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-home" pod "9e6d6d46-c350-4e74-9bad-964e2fd33dbd" (UID: "9e6d6d46-c350-4e74-9bad-964e2fd33dbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:36.768569 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768557    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-model-cache" pod "9e6d6d46-c350-4e74-9bad-964e2fd33dbd" (UID: "9e6d6d46-c350-4e74-9bad-964e2fd33dbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:36.768759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768595    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tmp-dir\") pod \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\" (UID: \"9e6d6d46-c350-4e74-9bad-964e2fd33dbd\") "
Nov 03 02:41:36.768759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768635    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-kube-api-access-wx5gk\" (UniqueName: \"kubernetes.io/projected/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-kube-api-access-wx5gk\") pod \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") "
Nov 03 02:41:36.768759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768668    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-home\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-home\") pod \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") "
Nov 03 02:41:36.768759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768707    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tmp-dir\") pod \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\" (UID: \"32669dc7-6c3a-4e8d-805b-651dbc1ab37f\") "
Nov 03 02:41:36.768983 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768772    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-model-cache" pod "32669dc7-6c3a-4e8d-805b-651dbc1ab37f" (UID: "32669dc7-6c3a-4e8d-805b-651dbc1ab37f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:36.768983 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.768911    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tmp-dir" pod "9e6d6d46-c350-4e74-9bad-964e2fd33dbd" (UID: "9e6d6d46-c350-4e74-9bad-964e2fd33dbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:36.769132 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.769112    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-home" pod "32669dc7-6c3a-4e8d-805b-651dbc1ab37f" (UID: "32669dc7-6c3a-4e8d-805b-651dbc1ab37f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:36.769330 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.769302    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tmp-dir" pod "32669dc7-6c3a-4e8d-805b-651dbc1ab37f" (UID: "32669dc7-6c3a-4e8d-805b-651dbc1ab37f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:36.769436 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.769370    5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-model-cache\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.769436 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.769394    5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-home\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.769436 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.769411    5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-model-cache\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.769436 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.769426    5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-dshm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.769716 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.769440    5508 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-model-cache\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.769716 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.769455    5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d613494-eedc-4a88-86c3-d076b634c9e1-tls-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.769716 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.769469    5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tmp-dir\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.769716 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.769482    5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-home\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.769716 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.769494    5508 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d613494-eedc-4a88-86c3-d076b634c9e1-home\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.769716 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.769510    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcf59\" (UniqueName: \"kubernetes.io/projected/2d613494-eedc-4a88-86c3-d076b634c9e1-kube-api-access-qcf59\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.770909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.770882    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tls-certs" pod "32669dc7-6c3a-4e8d-805b-651dbc1ab37f" (UID: "32669dc7-6c3a-4e8d-805b-651dbc1ab37f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:41:36.771030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.770938    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-dshm" pod "9e6d6d46-c350-4e74-9bad-964e2fd33dbd" (UID: "9e6d6d46-c350-4e74-9bad-964e2fd33dbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:36.771030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.770942    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tls-certs" pod "9e6d6d46-c350-4e74-9bad-964e2fd33dbd" (UID: "9e6d6d46-c350-4e74-9bad-964e2fd33dbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:41:36.771166 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.771149    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-kube-api-access-wx5gk" pod "32669dc7-6c3a-4e8d-805b-651dbc1ab37f" (UID: "32669dc7-6c3a-4e8d-805b-651dbc1ab37f"). InnerVolumeSpecName "kube-api-access-wx5gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:41:36.771402 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.771387    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-kube-api-access-8rw2j" pod "9e6d6d46-c350-4e74-9bad-964e2fd33dbd" (UID: "9e6d6d46-c350-4e74-9bad-964e2fd33dbd"). InnerVolumeSpecName "kube-api-access-8rw2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:41:36.772007 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.771991    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-dshm" pod "32669dc7-6c3a-4e8d-805b-651dbc1ab37f" (UID: "32669dc7-6c3a-4e8d-805b-651dbc1ab37f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:41:36.787591 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.787558    5508 generic.go:356] "Generic (PLEG): container finished" podID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerID="acd4fd150c0d041dd62ce2741b856cb315878e487105ed126f56d91b769e1363" exitCode=0
Nov 03 02:41:36.787764 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.787618    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"
Nov 03 02:41:36.787764 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.787639    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" event={"ID":"9e6d6d46-c350-4e74-9bad-964e2fd33dbd","Type":"ContainerDied","Data":"acd4fd150c0d041dd62ce2741b856cb315878e487105ed126f56d91b769e1363"}
Nov 03 02:41:36.787764 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.787686    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz" event={"ID":"9e6d6d46-c350-4e74-9bad-964e2fd33dbd","Type":"ContainerDied","Data":"e9c682dc818159b4b2e9942d9e9c4e4573d9983587a3d3d1f37be0468bafb03b"}
Nov 03 02:41:36.787764 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.787704    5508 scope.go:122] "RemoveContainer" containerID="acd4fd150c0d041dd62ce2741b856cb315878e487105ed126f56d91b769e1363"
Nov 03 02:41:36.789154 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.789132    5508 generic.go:356] "Generic (PLEG): container finished" podID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerID="e338c94661370d4f774e35063c5b6d0967fc1a5fadca152d02e00fdb12715b80" exitCode=0
Nov 03 02:41:36.789292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.789165    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" event={"ID":"32669dc7-6c3a-4e8d-805b-651dbc1ab37f","Type":"ContainerDied","Data":"e338c94661370d4f774e35063c5b6d0967fc1a5fadca152d02e00fdb12715b80"}
Nov 03 02:41:36.789292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.789202    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs" event={"ID":"32669dc7-6c3a-4e8d-805b-651dbc1ab37f","Type":"ContainerDied","Data":"459b271145ebf4e07784e9481206db33fd37f322ecf15c35f8bd9df18bab2d89"}
Nov 03 02:41:36.789292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.789202    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"
Nov 03 02:41:36.790706 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.790684    5508 generic.go:356] "Generic (PLEG): container finished" podID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerID="33a8dbc71ed34f025fc6e8b2a20928178815895a0722d8d269634f69158deb17" exitCode=0
Nov 03 02:41:36.790825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.790746    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" event={"ID":"2d613494-eedc-4a88-86c3-d076b634c9e1","Type":"ContainerDied","Data":"33a8dbc71ed34f025fc6e8b2a20928178815895a0722d8d269634f69158deb17"}
Nov 03 02:41:36.790825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.790771    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb" event={"ID":"2d613494-eedc-4a88-86c3-d076b634c9e1","Type":"ContainerDied","Data":"a467bf38ee3b62f222fe133e6b9c84a0f57907b822f1ba56ae650e024ca01362"}
Nov 03 02:41:36.790825 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.790749    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"
Nov 03 02:41:36.798342 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.798319    5508 scope.go:122] "RemoveContainer" containerID="acd4fd150c0d041dd62ce2741b856cb315878e487105ed126f56d91b769e1363"
Nov 03 02:41:36.798809 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:41:36.798672    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd4fd150c0d041dd62ce2741b856cb315878e487105ed126f56d91b769e1363\": container with ID starting with acd4fd150c0d041dd62ce2741b856cb315878e487105ed126f56d91b769e1363 not found: ID does not exist" containerID="acd4fd150c0d041dd62ce2741b856cb315878e487105ed126f56d91b769e1363"
Nov 03 02:41:36.798809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.798699    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd4fd150c0d041dd62ce2741b856cb315878e487105ed126f56d91b769e1363"} err="failed to get container status \"acd4fd150c0d041dd62ce2741b856cb315878e487105ed126f56d91b769e1363\": rpc error: code = NotFound desc = could not find container \"acd4fd150c0d041dd62ce2741b856cb315878e487105ed126f56d91b769e1363\": container with ID starting with acd4fd150c0d041dd62ce2741b856cb315878e487105ed126f56d91b769e1363 not found: ID does not exist"
Nov 03 02:41:36.798809 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.798715    5508 scope.go:122] "RemoveContainer" containerID="e338c94661370d4f774e35063c5b6d0967fc1a5fadca152d02e00fdb12715b80"
Nov 03 02:41:36.808030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.808006    5508 scope.go:122] "RemoveContainer" containerID="e338c94661370d4f774e35063c5b6d0967fc1a5fadca152d02e00fdb12715b80"
Nov 03 02:41:36.808273 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:41:36.808251    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e338c94661370d4f774e35063c5b6d0967fc1a5fadca152d02e00fdb12715b80\": container with ID starting with e338c94661370d4f774e35063c5b6d0967fc1a5fadca152d02e00fdb12715b80 not found: ID does not exist" containerID="e338c94661370d4f774e35063c5b6d0967fc1a5fadca152d02e00fdb12715b80"
Nov 03 02:41:36.808365 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.808285    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e338c94661370d4f774e35063c5b6d0967fc1a5fadca152d02e00fdb12715b80"} err="failed to get container status \"e338c94661370d4f774e35063c5b6d0967fc1a5fadca152d02e00fdb12715b80\": rpc error: code = NotFound desc = could not find container \"e338c94661370d4f774e35063c5b6d0967fc1a5fadca152d02e00fdb12715b80\": container with ID starting with e338c94661370d4f774e35063c5b6d0967fc1a5fadca152d02e00fdb12715b80 not found: ID does not exist"
Nov 03 02:41:36.808365 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.808312    5508 scope.go:122] "RemoveContainer" containerID="33a8dbc71ed34f025fc6e8b2a20928178815895a0722d8d269634f69158deb17"
Nov 03 02:41:36.817212 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.817132    5508 scope.go:122] "RemoveContainer" containerID="33a8dbc71ed34f025fc6e8b2a20928178815895a0722d8d269634f69158deb17"
Nov 03 02:41:36.818180 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:41:36.817783    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a8dbc71ed34f025fc6e8b2a20928178815895a0722d8d269634f69158deb17\": container with ID starting with 33a8dbc71ed34f025fc6e8b2a20928178815895a0722d8d269634f69158deb17 not found: ID does not exist" containerID="33a8dbc71ed34f025fc6e8b2a20928178815895a0722d8d269634f69158deb17"
Nov 03 02:41:36.818180 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.818123    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a8dbc71ed34f025fc6e8b2a20928178815895a0722d8d269634f69158deb17"} err="failed to get container status \"33a8dbc71ed34f025fc6e8b2a20928178815895a0722d8d269634f69158deb17\": rpc error: code = NotFound desc = could not find container \"33a8dbc71ed34f025fc6e8b2a20928178815895a0722d8d269634f69158deb17\": container with ID starting with 33a8dbc71ed34f025fc6e8b2a20928178815895a0722d8d269634f69158deb17 not found: ID does not exist"
Nov 03 02:41:36.870822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.870780    5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-dshm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.870822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.870811    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx5gk\" (UniqueName: \"kubernetes.io/projected/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-kube-api-access-wx5gk\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.870822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.870823    5508 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tmp-dir\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.871159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.870835    5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-tls-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.871159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.870844    5508 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-dshm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.871159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.870852    5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32669dc7-6c3a-4e8d-805b-651dbc1ab37f-tls-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:36.871159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:36.870861    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rw2j\" (UniqueName: \"kubernetes.io/projected/9e6d6d46-c350-4e74-9bad-964e2fd33dbd-kube-api-access-8rw2j\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:41:37.036637 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:37.036539    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"]
Nov 03 02:41:37.040908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:37.040869    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/premium-w2-9da431-kserve-6d58665668-s2ztz"]
Nov 03 02:41:37.144437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:37.144401    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"]
Nov 03 02:41:37.151015 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:37.150979    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/simulator-w2-9da431-kserve-8df84bcb9-dhmcs"]
Nov 03 02:41:37.254150 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:37.254109    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"]
Nov 03 02:41:37.259767 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:37.259733    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9da431/distinct-2-w2-9da431-kserve-6f94d9c99b-7b7wb"]
Nov 03 02:41:38.694307 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:38.694273    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" path="/var/lib/kubelet/pods/2d613494-eedc-4a88-86c3-d076b634c9e1/volumes"
Nov 03 02:41:38.694817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:38.694747    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" path="/var/lib/kubelet/pods/32669dc7-6c3a-4e8d-805b-651dbc1ab37f/volumes"
Nov 03 02:41:38.695127 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:38.695112    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" path="/var/lib/kubelet/pods/9e6d6d46-c350-4e74-9bad-964e2fd33dbd/volumes"
Nov 03 02:41:39.319814 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:39.319775    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:41:43.816501 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:41:43.816454    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:41:46.627377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:46.625695    5508 scope.go:122] "RemoveContainer" containerID="7ffe44d3d678cff4583cd845df0da434dc511dd5d51985328fa580f0505e7a2e"
Nov 03 02:41:46.644836 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:41:46.644707    5508 scope.go:122] "RemoveContainer" containerID="abd36e1ab769149fc3764b0d42733d7f35b3a421c84932d237a9ca0e4ba35828"
Nov 03 02:42:02.475748 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:02.475703    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"]
Nov 03 02:42:02.478092 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:02.476013    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh" podUID="da498724-5787-4ae8-82ed-0e3355fb496e" containerName="istio-proxy" containerID="cri-o://810759daef347d85f4b6c31860e46a758e989e472539e20c14341d963680dba6" gracePeriod=30
Nov 03 02:42:03.075834 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:03.075795    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:42:07.752804 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.752736    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:42:07.854161 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.854118    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-socket\") pod \"da498724-5787-4ae8-82ed-0e3355fb496e\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") "
Nov 03 02:42:07.854385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.854186    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-credential-socket\") pod \"da498724-5787-4ae8-82ed-0e3355fb496e\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") "
Nov 03 02:42:07.854385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.854244    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-certs\") pod \"da498724-5787-4ae8-82ed-0e3355fb496e\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") "
Nov 03 02:42:07.854385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.854314    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-kube-api-access-8ss8f\" (UniqueName: \"kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-kube-api-access-8ss8f\") pod \"da498724-5787-4ae8-82ed-0e3355fb496e\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") "
Nov 03 02:42:07.854385 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.854371    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-envoy\") pod \"da498724-5787-4ae8-82ed-0e3355fb496e\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") "
Nov 03 02:42:07.854635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.854434    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da498724-5787-4ae8-82ed-0e3355fb496e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da498724-5787-4ae8-82ed-0e3355fb496e-istiod-ca-cert\") pod \"da498724-5787-4ae8-82ed-0e3355fb496e\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") "
Nov 03 02:42:07.854635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.854472    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-certs" pod "da498724-5787-4ae8-82ed-0e3355fb496e" (UID: "da498724-5787-4ae8-82ed-0e3355fb496e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:42:07.854635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.854482    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da498724-5787-4ae8-82ed-0e3355fb496e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da498724-5787-4ae8-82ed-0e3355fb496e-istio-podinfo\") pod \"da498724-5787-4ae8-82ed-0e3355fb496e\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") "
Nov 03 02:42:07.854635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.854507    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-socket" pod "da498724-5787-4ae8-82ed-0e3355fb496e" (UID: "da498724-5787-4ae8-82ed-0e3355fb496e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:42:07.854635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.854536    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-credential-socket" pod "da498724-5787-4ae8-82ed-0e3355fb496e" (UID: "da498724-5787-4ae8-82ed-0e3355fb496e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:42:07.854635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.854588    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-istio-token\" (UniqueName: \"kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-istio-token\") pod \"da498724-5787-4ae8-82ed-0e3355fb496e\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") "
Nov 03 02:42:07.854635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.854632    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-data\") pod \"da498724-5787-4ae8-82ed-0e3355fb496e\" (UID: \"da498724-5787-4ae8-82ed-0e3355fb496e\") "
Nov 03 02:42:07.855049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.854832    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da498724-5787-4ae8-82ed-0e3355fb496e-istiod-ca-cert" pod "da498724-5787-4ae8-82ed-0e3355fb496e" (UID: "da498724-5787-4ae8-82ed-0e3355fb496e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:42:07.855049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.855039    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-data" pod "da498724-5787-4ae8-82ed-0e3355fb496e" (UID: "da498724-5787-4ae8-82ed-0e3355fb496e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:42:07.855178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.855160    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:07.855241 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.855183    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:07.855241 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.855199    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:07.855241 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.855213    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:07.855241 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.855229    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da498724-5787-4ae8-82ed-0e3355fb496e-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:07.856734 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.856708    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da498724-5787-4ae8-82ed-0e3355fb496e-istio-podinfo" pod "da498724-5787-4ae8-82ed-0e3355fb496e" (UID: "da498724-5787-4ae8-82ed-0e3355fb496e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:42:07.856734 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.856720    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-envoy" pod "da498724-5787-4ae8-82ed-0e3355fb496e" (UID: "da498724-5787-4ae8-82ed-0e3355fb496e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:42:07.856920 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.856804    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-kube-api-access-8ss8f" pod "da498724-5787-4ae8-82ed-0e3355fb496e" (UID: "da498724-5787-4ae8-82ed-0e3355fb496e"). InnerVolumeSpecName "kube-api-access-8ss8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:42:07.857084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.857071    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-istio-token" pod "da498724-5787-4ae8-82ed-0e3355fb496e" (UID: "da498724-5787-4ae8-82ed-0e3355fb496e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:42:07.956178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.956136    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ss8f\" (UniqueName: \"kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-kube-api-access-8ss8f\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:07.956178 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.956178    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da498724-5787-4ae8-82ed-0e3355fb496e-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:07.956421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.956194    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da498724-5787-4ae8-82ed-0e3355fb496e-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:07.956421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.956207    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da498724-5787-4ae8-82ed-0e3355fb496e-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:07.996337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.996302    5508 generic.go:356] "Generic (PLEG): container finished" podID="da498724-5787-4ae8-82ed-0e3355fb496e" containerID="810759daef347d85f4b6c31860e46a758e989e472539e20c14341d963680dba6" exitCode=0
Nov 03 02:42:07.996571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.996374    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"
Nov 03 02:42:07.996571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.996395    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh" event={"ID":"da498724-5787-4ae8-82ed-0e3355fb496e","Type":"ContainerDied","Data":"810759daef347d85f4b6c31860e46a758e989e472539e20c14341d963680dba6"}
Nov 03 02:42:07.996571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.996460    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh" event={"ID":"da498724-5787-4ae8-82ed-0e3355fb496e","Type":"ContainerDied","Data":"d152736f520e17e2e78ffafaedaf87ed7b0ce09fc02be82213e7fe545ea9b97c"}
Nov 03 02:42:07.996571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:07.996493    5508 scope.go:122] "RemoveContainer" containerID="810759daef347d85f4b6c31860e46a758e989e472539e20c14341d963680dba6"
Nov 03 02:42:08.006748 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:08.006729    5508 scope.go:122] "RemoveContainer" containerID="810759daef347d85f4b6c31860e46a758e989e472539e20c14341d963680dba6"
Nov 03 02:42:08.006994 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:42:08.006978    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"810759daef347d85f4b6c31860e46a758e989e472539e20c14341d963680dba6\": container with ID starting with 810759daef347d85f4b6c31860e46a758e989e472539e20c14341d963680dba6 not found: ID does not exist" containerID="810759daef347d85f4b6c31860e46a758e989e472539e20c14341d963680dba6"
Nov 03 02:42:08.007058 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:08.007005    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"810759daef347d85f4b6c31860e46a758e989e472539e20c14341d963680dba6"} err="failed to get container status \"810759daef347d85f4b6c31860e46a758e989e472539e20c14341d963680dba6\": rpc error: code = NotFound desc = could not find container \"810759daef347d85f4b6c31860e46a758e989e472539e20c14341d963680dba6\": container with ID starting with 810759daef347d85f4b6c31860e46a758e989e472539e20c14341d963680dba6 not found: ID does not exist"
Nov 03 02:42:08.152384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:08.152330    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"]
Nov 03 02:42:08.158396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:08.158363    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9da431-openshift-default-65c799fd77-hp8wh"]
Nov 03 02:42:08.691922 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:08.691893    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da498724-5787-4ae8-82ed-0e3355fb496e" path="/var/lib/kubelet/pods/da498724-5787-4ae8-82ed-0e3355fb496e/volumes"
Nov 03 02:42:13.811579 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:42:13.811533    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:42:23.028325 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028288    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"]
Nov 03 02:42:23.028772 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028736    5508 cpu_manager.go:414] "containerMap: removing container" podUID="78804323-a28a-441e-a1c7-506a502e0469" containerName="revoke-keys"
Nov 03 02:42:23.028772 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028749    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78804323-a28a-441e-a1c7-506a502e0469" containerName="revoke-keys"
Nov 03 02:42:23.028878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028776    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main"
Nov 03 02:42:23.028878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028782    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main"
Nov 03 02:42:23.028878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028795    5508 cpu_manager.go:414] "containerMap: removing container" podUID="6684afe1-182b-4c8b-b2ed-d9491cfd68a8" containerName="payload-processing"
Nov 03 02:42:23.028878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028800    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6684afe1-182b-4c8b-b2ed-d9491cfd68a8" containerName="payload-processing"
Nov 03 02:42:23.028878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028809    5508 cpu_manager.go:414] "containerMap: removing container" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main"
Nov 03 02:42:23.028878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028814    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main"
Nov 03 02:42:23.028878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028824    5508 cpu_manager.go:414] "containerMap: removing container" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main"
Nov 03 02:42:23.028878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028829    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main"
Nov 03 02:42:23.028878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028835    5508 cpu_manager.go:414] "containerMap: removing container" podUID="da498724-5787-4ae8-82ed-0e3355fb496e" containerName="istio-proxy"
Nov 03 02:42:23.028878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028839    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da498724-5787-4ae8-82ed-0e3355fb496e" containerName="istio-proxy"
Nov 03 02:42:23.028878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028847    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main"
Nov 03 02:42:23.028878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028853    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main"
Nov 03 02:42:23.029308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028906    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="32669dc7-6c3a-4e8d-805b-651dbc1ab37f" containerName="main"
Nov 03 02:42:23.029308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028915    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d613494-eedc-4a88-86c3-d076b634c9e1" containerName="main"
Nov 03 02:42:23.029308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028922    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="da498724-5787-4ae8-82ed-0e3355fb496e" containerName="istio-proxy"
Nov 03 02:42:23.029308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028928    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4243b3-d50b-4995-b51d-29174542a75d" containerName="main"
Nov 03 02:42:23.029308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028935    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="78804323-a28a-441e-a1c7-506a502e0469" containerName="revoke-keys"
Nov 03 02:42:23.029308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028943    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6d6d46-c350-4e74-9bad-964e2fd33dbd" containerName="main"
Nov 03 02:42:23.029308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.028950    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6684afe1-182b-4c8b-b2ed-d9491cfd68a8" containerName="payload-processing"
Nov 03 02:42:23.032282 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.032261    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"
Nov 03 02:42:23.081164 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.081129    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:42:23.123963 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.123912    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkpdn\" (UniqueName: \"kubernetes.io/projected/f671fa22-98b0-4dc3-bb0b-fa278afd2991-kube-api-access-kkpdn\") pod \"maas-api-revoke-sub-04223fd8-8t2p2\" (UID: \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"
Nov 03 02:42:23.124142 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.124078    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04223fd8-8t2p2\" (UID: \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"
Nov 03 02:42:23.124186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.124136    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04223fd8-8t2p2\" (UID: \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"
Nov 03 02:42:23.127452 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.127429    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:42:23.130069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.130046    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"]
Nov 03 02:42:23.225095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.225051    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkpdn\" (UniqueName: \"kubernetes.io/projected/f671fa22-98b0-4dc3-bb0b-fa278afd2991-kube-api-access-kkpdn\") pod \"maas-api-revoke-sub-04223fd8-8t2p2\" (UID: \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"
Nov 03 02:42:23.225292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.225116    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04223fd8-8t2p2\" (UID: \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"
Nov 03 02:42:23.225292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.225167    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04223fd8-8t2p2\" (UID: \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"
Nov 03 02:42:23.225873 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.225840    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04223fd8-8t2p2\" (UID: \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"
Nov 03 02:42:23.227634 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.227603    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04223fd8-8t2p2\" (UID: \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"
Nov 03 02:42:23.282238 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.282154    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkpdn\" (UniqueName: \"kubernetes.io/projected/f671fa22-98b0-4dc3-bb0b-fa278afd2991-kube-api-access-kkpdn\") pod \"maas-api-revoke-sub-04223fd8-8t2p2\" (UID: \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"
Nov 03 02:42:23.345304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.345255    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"
Nov 03 02:42:23.719333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:23.719300    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"]
Nov 03 02:42:23.727018 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:42:23.726981    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf671fa22_98b0_4dc3_bb0b_fa278afd2991.slice/crio-916ad2898f010044c219de056557f22d3a6a99ee20a965b900cad177a49d98ed WatchSource:0}: Error finding container 916ad2898f010044c219de056557f22d3a6a99ee20a965b900cad177a49d98ed: Status 404 returned error can't find the container with id 916ad2898f010044c219de056557f22d3a6a99ee20a965b900cad177a49d98ed
Nov 03 02:42:24.097912 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:24.097877    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2" event={"ID":"f671fa22-98b0-4dc3-bb0b-fa278afd2991","Type":"ContainerStarted","Data":"256b385ea4124d3bb6f079575c1314fc888eddc8a597441acddfa68c5d7369ee"}
Nov 03 02:42:24.098317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:24.097933    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2" event={"ID":"f671fa22-98b0-4dc3-bb0b-fa278afd2991","Type":"ContainerStarted","Data":"916ad2898f010044c219de056557f22d3a6a99ee20a965b900cad177a49d98ed"}
Nov 03 02:42:24.207115 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:24.207061    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2" podStartSLOduration=2.207044457 podStartE2EDuration="2.207044457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:42:24.205666272 +0000 UTC m=+5023.631272739" watchObservedRunningTime="2026-11-03 02:42:24.207044457 +0000 UTC m=+5023.632650916"
Nov 03 02:42:25.105627 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:25.105589    5508 generic.go:356] "Generic (PLEG): container finished" podID="f671fa22-98b0-4dc3-bb0b-fa278afd2991" containerID="256b385ea4124d3bb6f079575c1314fc888eddc8a597441acddfa68c5d7369ee" exitCode=0
Nov 03 02:42:25.106119 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:25.105683    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2" event={"ID":"f671fa22-98b0-4dc3-bb0b-fa278afd2991","Type":"ContainerDied","Data":"256b385ea4124d3bb6f079575c1314fc888eddc8a597441acddfa68c5d7369ee"}
Nov 03 02:42:26.046270 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.046237    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"]
Nov 03 02:42:26.066609 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.066578    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18bf9aaa-bc54d"]
Nov 03 02:42:26.253578 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.253549    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"
Nov 03 02:42:26.357893 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.357794    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f671fa22-98b0-4dc3-bb0b-fa278afd2991-kube-api-access-kkpdn\" (UniqueName: \"kubernetes.io/projected/f671fa22-98b0-4dc3-bb0b-fa278afd2991-kube-api-access-kkpdn\") pod \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\" (UID: \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\") "
Nov 03 02:42:26.357893 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.357868    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-xks-ca\") pod \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\" (UID: \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\") "
Nov 03 02:42:26.358173 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.358026    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-openshift-ca\") pod \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\" (UID: \"f671fa22-98b0-4dc3-bb0b-fa278afd2991\") "
Nov 03 02:42:26.358498 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.358456    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-openshift-ca" pod "f671fa22-98b0-4dc3-bb0b-fa278afd2991" (UID: "f671fa22-98b0-4dc3-bb0b-fa278afd2991"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:42:26.360170 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.360139    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f671fa22-98b0-4dc3-bb0b-fa278afd2991-kube-api-access-kkpdn" pod "f671fa22-98b0-4dc3-bb0b-fa278afd2991" (UID: "f671fa22-98b0-4dc3-bb0b-fa278afd2991"). InnerVolumeSpecName "kube-api-access-kkpdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:42:26.360285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.360186    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-xks-ca" pod "f671fa22-98b0-4dc3-bb0b-fa278afd2991" (UID: "f671fa22-98b0-4dc3-bb0b-fa278afd2991"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:42:26.459484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.459437    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkpdn\" (UniqueName: \"kubernetes.io/projected/f671fa22-98b0-4dc3-bb0b-fa278afd2991-kube-api-access-kkpdn\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:26.459484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.459470    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:26.459484 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.459481    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f671fa22-98b0-4dc3-bb0b-fa278afd2991-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:26.578913 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.578878    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:42:26.691173 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:26.691141    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f58757-8386-475c-a39a-623194abcef7" path="/var/lib/kubelet/pods/59f58757-8386-475c-a39a-623194abcef7/volumes"
Nov 03 02:42:27.119090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:27.119008    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"
Nov 03 02:42:27.119090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:27.119036    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2" event={"ID":"f671fa22-98b0-4dc3-bb0b-fa278afd2991","Type":"ContainerDied","Data":"916ad2898f010044c219de056557f22d3a6a99ee20a965b900cad177a49d98ed"}
Nov 03 02:42:27.119090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:27.119084    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="916ad2898f010044c219de056557f22d3a6a99ee20a965b900cad177a49d98ed"
Nov 03 02:42:33.837343 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:33.837304    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"]
Nov 03 02:42:33.837863 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:33.837744    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f671fa22-98b0-4dc3-bb0b-fa278afd2991" containerName="revoke-keys"
Nov 03 02:42:33.837863 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:33.837755    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f671fa22-98b0-4dc3-bb0b-fa278afd2991" containerName="revoke-keys"
Nov 03 02:42:33.837863 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:33.837826    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f671fa22-98b0-4dc3-bb0b-fa278afd2991" containerName="revoke-keys"
Nov 03 02:42:33.841990 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:33.841967    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"
Nov 03 02:42:33.890800 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:33.890767    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:42:33.936907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:33.936877    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:42:33.939543 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:33.939505    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"]
Nov 03 02:42:33.939958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:33.939931    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4747842-59tfl\" (UID: \"9ce2b589-5743-4821-9b2c-0e46087fba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"
Nov 03 02:42:33.940044 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:33.940028    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82czq\" (UniqueName: \"kubernetes.io/projected/9ce2b589-5743-4821-9b2c-0e46087fba80-kube-api-access-82czq\") pod \"maas-api-revoke-sub-c4747842-59tfl\" (UID: \"9ce2b589-5743-4821-9b2c-0e46087fba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"
Nov 03 02:42:33.940086 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:33.940073    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4747842-59tfl\" (UID: \"9ce2b589-5743-4821-9b2c-0e46087fba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"
Nov 03 02:42:34.041122 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:34.041089    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4747842-59tfl\" (UID: \"9ce2b589-5743-4821-9b2c-0e46087fba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"
Nov 03 02:42:34.041333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:34.041144    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82czq\" (UniqueName: \"kubernetes.io/projected/9ce2b589-5743-4821-9b2c-0e46087fba80-kube-api-access-82czq\") pod \"maas-api-revoke-sub-c4747842-59tfl\" (UID: \"9ce2b589-5743-4821-9b2c-0e46087fba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"
Nov 03 02:42:34.041333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:34.041176    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4747842-59tfl\" (UID: \"9ce2b589-5743-4821-9b2c-0e46087fba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"
Nov 03 02:42:34.041807 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:34.041782    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4747842-59tfl\" (UID: \"9ce2b589-5743-4821-9b2c-0e46087fba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"
Nov 03 02:42:34.044076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:34.044054    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4747842-59tfl\" (UID: \"9ce2b589-5743-4821-9b2c-0e46087fba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"
Nov 03 02:42:34.099113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:34.099032    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82czq\" (UniqueName: \"kubernetes.io/projected/9ce2b589-5743-4821-9b2c-0e46087fba80-kube-api-access-82czq\") pod \"maas-api-revoke-sub-c4747842-59tfl\" (UID: \"9ce2b589-5743-4821-9b2c-0e46087fba80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"
Nov 03 02:42:34.154380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:34.154337    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"
Nov 03 02:42:34.534454 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:34.534417    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"]
Nov 03 02:42:34.541978 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:42:34.541944    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce2b589_5743_4821_9b2c_0e46087fba80.slice/crio-2ad944a0bbd6d93fa37d9d1badd9a09b9c7b4edf66fbba94611ae78d58117dbf WatchSource:0}: Error finding container 2ad944a0bbd6d93fa37d9d1badd9a09b9c7b4edf66fbba94611ae78d58117dbf: Status 404 returned error can't find the container with id 2ad944a0bbd6d93fa37d9d1badd9a09b9c7b4edf66fbba94611ae78d58117dbf
Nov 03 02:42:35.164977 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:35.164942    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl" event={"ID":"9ce2b589-5743-4821-9b2c-0e46087fba80","Type":"ContainerStarted","Data":"9ac121b40ff174f911ba753a59d23df7756c050de014ac20aecf5852ec29fb7d"}
Nov 03 02:42:35.165417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:35.164997    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl" event={"ID":"9ce2b589-5743-4821-9b2c-0e46087fba80","Type":"ContainerStarted","Data":"2ad944a0bbd6d93fa37d9d1badd9a09b9c7b4edf66fbba94611ae78d58117dbf"}
Nov 03 02:42:35.273815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:35.273749    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl" podStartSLOduration=2.273733127 podStartE2EDuration="2.273733127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:42:35.272504148 +0000 UTC m=+5034.698110611" watchObservedRunningTime="2026-11-03 02:42:35.273733127 +0000 UTC m=+5034.699339590"
Nov 03 02:42:36.172163 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:36.172130    5508 generic.go:356] "Generic (PLEG): container finished" podID="9ce2b589-5743-4821-9b2c-0e46087fba80" containerID="9ac121b40ff174f911ba753a59d23df7756c050de014ac20aecf5852ec29fb7d" exitCode=0
Nov 03 02:42:36.172712 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:36.172184    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl" event={"ID":"9ce2b589-5743-4821-9b2c-0e46087fba80","Type":"ContainerDied","Data":"9ac121b40ff174f911ba753a59d23df7756c050de014ac20aecf5852ec29fb7d"}
Nov 03 02:42:37.058026 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:37.057989    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"]
Nov 03 02:42:37.060518 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:37.060492    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bc4a76-r82bj"]
Nov 03 02:42:37.318305 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:37.318245    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"
Nov 03 02:42:37.477867 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:37.477832    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce2b589-5743-4821-9b2c-0e46087fba80-kube-api-access-82czq\" (UniqueName: \"kubernetes.io/projected/9ce2b589-5743-4821-9b2c-0e46087fba80-kube-api-access-82czq\") pod \"9ce2b589-5743-4821-9b2c-0e46087fba80\" (UID: \"9ce2b589-5743-4821-9b2c-0e46087fba80\") "
Nov 03 02:42:37.478062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:37.477892    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-xks-ca\") pod \"9ce2b589-5743-4821-9b2c-0e46087fba80\" (UID: \"9ce2b589-5743-4821-9b2c-0e46087fba80\") "
Nov 03 02:42:37.478062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:37.477920    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-openshift-ca\") pod \"9ce2b589-5743-4821-9b2c-0e46087fba80\" (UID: \"9ce2b589-5743-4821-9b2c-0e46087fba80\") "
Nov 03 02:42:37.478397 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:37.478341    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-openshift-ca" pod "9ce2b589-5743-4821-9b2c-0e46087fba80" (UID: "9ce2b589-5743-4821-9b2c-0e46087fba80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:42:37.480004 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:37.479982    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-xks-ca" pod "9ce2b589-5743-4821-9b2c-0e46087fba80" (UID: "9ce2b589-5743-4821-9b2c-0e46087fba80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:42:37.480113 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:37.480028    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce2b589-5743-4821-9b2c-0e46087fba80-kube-api-access-82czq" pod "9ce2b589-5743-4821-9b2c-0e46087fba80" (UID: "9ce2b589-5743-4821-9b2c-0e46087fba80"). InnerVolumeSpecName "kube-api-access-82czq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:42:37.579034 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:37.578938    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82czq\" (UniqueName: \"kubernetes.io/projected/9ce2b589-5743-4821-9b2c-0e46087fba80-kube-api-access-82czq\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:37.579034 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:37.578979    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:37.579034 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:37.578991    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce2b589-5743-4821-9b2c-0e46087fba80-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:38.187548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:38.187515    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl" event={"ID":"9ce2b589-5743-4821-9b2c-0e46087fba80","Type":"ContainerDied","Data":"2ad944a0bbd6d93fa37d9d1badd9a09b9c7b4edf66fbba94611ae78d58117dbf"}
Nov 03 02:42:38.187548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:38.187545    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"
Nov 03 02:42:38.187789 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:38.187571    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ad944a0bbd6d93fa37d9d1badd9a09b9c7b4edf66fbba94611ae78d58117dbf"
Nov 03 02:42:38.693295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:38.693241    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba64a7a8-9c90-4c77-ac37-57a6904c5549" path="/var/lib/kubelet/pods/ba64a7a8-9c90-4c77-ac37-57a6904c5549/volumes"
Nov 03 02:42:43.554589 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:43.554548    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"]
Nov 03 02:42:43.555057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:43.554770    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" podUID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" containerName="istio-proxy" containerID="cri-o://77a64726ad3c51be97e98419c6cd9527f3e402f227bda1a5d2d25400ed628b3b" gracePeriod=30
Nov 03 02:42:43.810803 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:42:43.810717    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:42:44.303508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:44.303469    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:42:45.142525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.142490    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-c4d57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:42:45.142945 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.142559    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" podUID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:42:45.471299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.471265    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"]
Nov 03 02:42:45.471745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.471732    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9ce2b589-5743-4821-9b2c-0e46087fba80" containerName="revoke-keys"
Nov 03 02:42:45.471789 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.471747    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce2b589-5743-4821-9b2c-0e46087fba80" containerName="revoke-keys"
Nov 03 02:42:45.471841 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.471832    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce2b589-5743-4821-9b2c-0e46087fba80" containerName="revoke-keys"
Nov 03 02:42:45.476192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.476171    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"
Nov 03 02:42:45.525930 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.525889    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:42:45.525930 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.525907    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:42:45.575043 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.575007    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"]
Nov 03 02:42:45.666826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.666785    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f02f7cbd-ktplc\" (UID: \"57696021-91bd-41a6-b396-7cbc018dcf9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"
Nov 03 02:42:45.667033 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.666915    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f02f7cbd-ktplc\" (UID: \"57696021-91bd-41a6-b396-7cbc018dcf9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"
Nov 03 02:42:45.667033 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.666991    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4d9n\" (UniqueName: \"kubernetes.io/projected/57696021-91bd-41a6-b396-7cbc018dcf9a-kube-api-access-d4d9n\") pod \"maas-api-revoke-sub-f02f7cbd-ktplc\" (UID: \"57696021-91bd-41a6-b396-7cbc018dcf9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"
Nov 03 02:42:45.768522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.768422    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4d9n\" (UniqueName: \"kubernetes.io/projected/57696021-91bd-41a6-b396-7cbc018dcf9a-kube-api-access-d4d9n\") pod \"maas-api-revoke-sub-f02f7cbd-ktplc\" (UID: \"57696021-91bd-41a6-b396-7cbc018dcf9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"
Nov 03 02:42:45.768522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.768510    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f02f7cbd-ktplc\" (UID: \"57696021-91bd-41a6-b396-7cbc018dcf9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"
Nov 03 02:42:45.768784 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.768555    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f02f7cbd-ktplc\" (UID: \"57696021-91bd-41a6-b396-7cbc018dcf9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"
Nov 03 02:42:45.769147 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.769125    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f02f7cbd-ktplc\" (UID: \"57696021-91bd-41a6-b396-7cbc018dcf9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"
Nov 03 02:42:45.770882 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.770859    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f02f7cbd-ktplc\" (UID: \"57696021-91bd-41a6-b396-7cbc018dcf9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"
Nov 03 02:42:45.826517 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:45.826481    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4d9n\" (UniqueName: \"kubernetes.io/projected/57696021-91bd-41a6-b396-7cbc018dcf9a-kube-api-access-d4d9n\") pod \"maas-api-revoke-sub-f02f7cbd-ktplc\" (UID: \"57696021-91bd-41a6-b396-7cbc018dcf9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"
Nov 03 02:42:46.089195 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:46.089076    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"
Nov 03 02:42:46.462339 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:46.462302    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"]
Nov 03 02:42:46.470291 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:42:46.470255    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57696021_91bd_41a6_b396_7cbc018dcf9a.slice/crio-7b084cfe8204d004bffad8c17af3973be8ef380e3a55ab99038ba043e21a5c8b WatchSource:0}: Error finding container 7b084cfe8204d004bffad8c17af3973be8ef380e3a55ab99038ba043e21a5c8b: Status 404 returned error can't find the container with id 7b084cfe8204d004bffad8c17af3973be8ef380e3a55ab99038ba043e21a5c8b
Nov 03 02:42:47.244559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:47.244515    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc" event={"ID":"57696021-91bd-41a6-b396-7cbc018dcf9a","Type":"ContainerStarted","Data":"ecc36e5ce76cb5f003d6685d7b7a923033a8b4bfabb0d8313e366a007da6573f"}
Nov 03 02:42:47.244759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:47.244581    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc" event={"ID":"57696021-91bd-41a6-b396-7cbc018dcf9a","Type":"ContainerStarted","Data":"7b084cfe8204d004bffad8c17af3973be8ef380e3a55ab99038ba043e21a5c8b"}
Nov 03 02:42:47.360564 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:47.360502    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc" podStartSLOduration=2.360482362 podStartE2EDuration="2.360482362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:42:47.357877615 +0000 UTC m=+5046.783484071" watchObservedRunningTime="2026-11-03 02:42:47.360482362 +0000 UTC m=+5046.786088825"
Nov 03 02:42:48.251466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.251429    5508 generic.go:356] "Generic (PLEG): container finished" podID="57696021-91bd-41a6-b396-7cbc018dcf9a" containerID="ecc36e5ce76cb5f003d6685d7b7a923033a8b4bfabb0d8313e366a007da6573f" exitCode=0
Nov 03 02:42:48.251952 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.251520    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc" event={"ID":"57696021-91bd-41a6-b396-7cbc018dcf9a","Type":"ContainerDied","Data":"ecc36e5ce76cb5f003d6685d7b7a923033a8b4bfabb0d8313e366a007da6573f"}
Nov 03 02:42:48.812982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.812958    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:42:48.901371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.901315    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-podinfo\") pod \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") "
Nov 03 02:42:48.901576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.901381    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-envoy\") pod \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") "
Nov 03 02:42:48.901576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.901403    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istiod-ca-cert\") pod \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") "
Nov 03 02:42:48.901576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.901439    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-certs\") pod \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") "
Nov 03 02:42:48.901576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.901467    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-socket\") pod \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") "
Nov 03 02:42:48.901576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.901493    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-token\" (UniqueName: \"kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-token\") pod \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") "
Nov 03 02:42:48.901576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.901547    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-data\") pod \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") "
Nov 03 02:42:48.901576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.901565    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-credential-socket\") pod \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") "
Nov 03 02:42:48.901962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.901588    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-kube-api-access-vsz72\" (UniqueName: \"kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-kube-api-access-vsz72\") pod \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\" (UID: \"a4194050-7f6a-49eb-ade8-fdd65ce6603c\") "
Nov 03 02:42:48.901962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.901760    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-certs" pod "a4194050-7f6a-49eb-ade8-fdd65ce6603c" (UID: "a4194050-7f6a-49eb-ade8-fdd65ce6603c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:42:48.901962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.901823    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-socket" pod "a4194050-7f6a-49eb-ade8-fdd65ce6603c" (UID: "a4194050-7f6a-49eb-ade8-fdd65ce6603c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:42:48.901962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.901890    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-data" pod "a4194050-7f6a-49eb-ade8-fdd65ce6603c" (UID: "a4194050-7f6a-49eb-ade8-fdd65ce6603c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:42:48.901962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.901943    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istiod-ca-cert" pod "a4194050-7f6a-49eb-ade8-fdd65ce6603c" (UID: "a4194050-7f6a-49eb-ade8-fdd65ce6603c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:42:48.902218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.902031    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-credential-socket" pod "a4194050-7f6a-49eb-ade8-fdd65ce6603c" (UID: "a4194050-7f6a-49eb-ade8-fdd65ce6603c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:42:48.902218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.902167    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:48.902218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.902185    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:48.902218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.902198    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:48.902218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.902213    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:48.903646 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.903622    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-podinfo" pod "a4194050-7f6a-49eb-ade8-fdd65ce6603c" (UID: "a4194050-7f6a-49eb-ade8-fdd65ce6603c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:42:48.903769 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.903745    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-kube-api-access-vsz72" pod "a4194050-7f6a-49eb-ade8-fdd65ce6603c" (UID: "a4194050-7f6a-49eb-ade8-fdd65ce6603c"). InnerVolumeSpecName "kube-api-access-vsz72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:42:48.904046 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.904023    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-token" pod "a4194050-7f6a-49eb-ade8-fdd65ce6603c" (UID: "a4194050-7f6a-49eb-ade8-fdd65ce6603c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:42:48.904046 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:48.904027    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-envoy" pod "a4194050-7f6a-49eb-ade8-fdd65ce6603c" (UID: "a4194050-7f6a-49eb-ade8-fdd65ce6603c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:42:49.002909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.002874    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:49.002909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.002903    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4194050-7f6a-49eb-ade8-fdd65ce6603c-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:49.002909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.002916    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:49.003174 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.002926    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsz72\" (UniqueName: \"kubernetes.io/projected/a4194050-7f6a-49eb-ade8-fdd65ce6603c-kube-api-access-vsz72\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:49.003174 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.002937    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4194050-7f6a-49eb-ade8-fdd65ce6603c-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:49.258689 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.258602    5508 generic.go:356] "Generic (PLEG): container finished" podID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" containerID="77a64726ad3c51be97e98419c6cd9527f3e402f227bda1a5d2d25400ed628b3b" exitCode=0
Nov 03 02:42:49.258689 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.258652    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"
Nov 03 02:42:49.259225 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.258691    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" event={"ID":"a4194050-7f6a-49eb-ade8-fdd65ce6603c","Type":"ContainerDied","Data":"77a64726ad3c51be97e98419c6cd9527f3e402f227bda1a5d2d25400ed628b3b"}
Nov 03 02:42:49.259225 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.258756    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57" event={"ID":"a4194050-7f6a-49eb-ade8-fdd65ce6603c","Type":"ContainerDied","Data":"6ac693e038e6750a33f86e5aebd41336061629ee46802d1cbd83e663ea587a9c"}
Nov 03 02:42:49.259225 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.258787    5508 scope.go:122] "RemoveContainer" containerID="77a64726ad3c51be97e98419c6cd9527f3e402f227bda1a5d2d25400ed628b3b"
Nov 03 02:42:49.269637 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.269617    5508 scope.go:122] "RemoveContainer" containerID="77a64726ad3c51be97e98419c6cd9527f3e402f227bda1a5d2d25400ed628b3b"
Nov 03 02:42:49.269956 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:42:49.269938    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a64726ad3c51be97e98419c6cd9527f3e402f227bda1a5d2d25400ed628b3b\": container with ID starting with 77a64726ad3c51be97e98419c6cd9527f3e402f227bda1a5d2d25400ed628b3b not found: ID does not exist" containerID="77a64726ad3c51be97e98419c6cd9527f3e402f227bda1a5d2d25400ed628b3b"
Nov 03 02:42:49.270008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.269967    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a64726ad3c51be97e98419c6cd9527f3e402f227bda1a5d2d25400ed628b3b"} err="failed to get container status \"77a64726ad3c51be97e98419c6cd9527f3e402f227bda1a5d2d25400ed628b3b\": rpc error: code = NotFound desc = could not find container \"77a64726ad3c51be97e98419c6cd9527f3e402f227bda1a5d2d25400ed628b3b\": container with ID starting with 77a64726ad3c51be97e98419c6cd9527f3e402f227bda1a5d2d25400ed628b3b not found: ID does not exist"
Nov 03 02:42:49.388551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.388523    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"
Nov 03 02:42:49.406487 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.406450    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-openshift-ca\") pod \"57696021-91bd-41a6-b396-7cbc018dcf9a\" (UID: \"57696021-91bd-41a6-b396-7cbc018dcf9a\") "
Nov 03 02:42:49.406678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.406602    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57696021-91bd-41a6-b396-7cbc018dcf9a-kube-api-access-d4d9n\" (UniqueName: \"kubernetes.io/projected/57696021-91bd-41a6-b396-7cbc018dcf9a-kube-api-access-d4d9n\") pod \"57696021-91bd-41a6-b396-7cbc018dcf9a\" (UID: \"57696021-91bd-41a6-b396-7cbc018dcf9a\") "
Nov 03 02:42:49.406678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.406654    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-xks-ca\") pod \"57696021-91bd-41a6-b396-7cbc018dcf9a\" (UID: \"57696021-91bd-41a6-b396-7cbc018dcf9a\") "
Nov 03 02:42:49.406875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.406852    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-openshift-ca" pod "57696021-91bd-41a6-b396-7cbc018dcf9a" (UID: "57696021-91bd-41a6-b396-7cbc018dcf9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:42:49.407011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.406993    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:49.408664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.408637    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-xks-ca" pod "57696021-91bd-41a6-b396-7cbc018dcf9a" (UID: "57696021-91bd-41a6-b396-7cbc018dcf9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:42:49.408766 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.408726    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57696021-91bd-41a6-b396-7cbc018dcf9a-kube-api-access-d4d9n" pod "57696021-91bd-41a6-b396-7cbc018dcf9a" (UID: "57696021-91bd-41a6-b396-7cbc018dcf9a"). InnerVolumeSpecName "kube-api-access-d4d9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:42:49.414530 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.414491    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"]
Nov 03 02:42:49.459528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.459492    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-c4d57"]
Nov 03 02:42:49.507800 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.507763    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4d9n\" (UniqueName: \"kubernetes.io/projected/57696021-91bd-41a6-b396-7cbc018dcf9a-kube-api-access-d4d9n\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:49.507800 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:49.507792    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57696021-91bd-41a6-b396-7cbc018dcf9a-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:50.266982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:50.266949    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"
Nov 03 02:42:50.266982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:50.266970    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc" event={"ID":"57696021-91bd-41a6-b396-7cbc018dcf9a","Type":"ContainerDied","Data":"7b084cfe8204d004bffad8c17af3973be8ef380e3a55ab99038ba043e21a5c8b"}
Nov 03 02:42:50.267531 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:50.267016    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b084cfe8204d004bffad8c17af3973be8ef380e3a55ab99038ba043e21a5c8b"
Nov 03 02:42:50.692902 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:50.692872    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" path="/var/lib/kubelet/pods/a4194050-7f6a-49eb-ade8-fdd65ce6603c/volumes"
Nov 03 02:42:56.424914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.424875    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"]
Nov 03 02:42:56.425382 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.425326    5508 cpu_manager.go:414] "containerMap: removing container" podUID="57696021-91bd-41a6-b396-7cbc018dcf9a" containerName="revoke-keys"
Nov 03 02:42:56.425382 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.425337    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57696021-91bd-41a6-b396-7cbc018dcf9a" containerName="revoke-keys"
Nov 03 02:42:56.425382 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.425359    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" containerName="istio-proxy"
Nov 03 02:42:56.425382 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.425365    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" containerName="istio-proxy"
Nov 03 02:42:56.425553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.425431    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="57696021-91bd-41a6-b396-7cbc018dcf9a" containerName="revoke-keys"
Nov 03 02:42:56.425553 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.425439    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4194050-7f6a-49eb-ade8-fdd65ce6603c" containerName="istio-proxy"
Nov 03 02:42:56.429168 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.429145    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"
Nov 03 02:42:56.477969 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.477934    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:42:56.480480 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.480451    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da6d111d-sdc5s\" (UID: \"2d143031-d33e-4e37-815c-f0b6f3a12835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"
Nov 03 02:42:56.480607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.480524    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm2k5\" (UniqueName: \"kubernetes.io/projected/2d143031-d33e-4e37-815c-f0b6f3a12835-kube-api-access-hm2k5\") pod \"maas-api-revoke-sub-da6d111d-sdc5s\" (UID: \"2d143031-d33e-4e37-815c-f0b6f3a12835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"
Nov 03 02:42:56.480681 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.480631    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da6d111d-sdc5s\" (UID: \"2d143031-d33e-4e37-815c-f0b6f3a12835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"
Nov 03 02:42:56.524153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.524123    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:42:56.526953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.526926    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"]
Nov 03 02:42:56.581251 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.581216    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da6d111d-sdc5s\" (UID: \"2d143031-d33e-4e37-815c-f0b6f3a12835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"
Nov 03 02:42:56.581467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.581282    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm2k5\" (UniqueName: \"kubernetes.io/projected/2d143031-d33e-4e37-815c-f0b6f3a12835-kube-api-access-hm2k5\") pod \"maas-api-revoke-sub-da6d111d-sdc5s\" (UID: \"2d143031-d33e-4e37-815c-f0b6f3a12835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"
Nov 03 02:42:56.581467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.581318    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da6d111d-sdc5s\" (UID: \"2d143031-d33e-4e37-815c-f0b6f3a12835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"
Nov 03 02:42:56.582015 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.581990    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da6d111d-sdc5s\" (UID: \"2d143031-d33e-4e37-815c-f0b6f3a12835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"
Nov 03 02:42:56.583618 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.583598    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da6d111d-sdc5s\" (UID: \"2d143031-d33e-4e37-815c-f0b6f3a12835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"
Nov 03 02:42:56.639433 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.639400    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm2k5\" (UniqueName: \"kubernetes.io/projected/2d143031-d33e-4e37-815c-f0b6f3a12835-kube-api-access-hm2k5\") pod \"maas-api-revoke-sub-da6d111d-sdc5s\" (UID: \"2d143031-d33e-4e37-815c-f0b6f3a12835\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"
Nov 03 02:42:56.741905 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:56.741819    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"
Nov 03 02:42:57.116633 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:57.116552    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"]
Nov 03 02:42:57.123995 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:42:57.123956    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d143031_d33e_4e37_815c_f0b6f3a12835.slice/crio-f9b6a2f57c4e90f2d9fc9e4113825c0bc6cd6ecf7692ff08994ca46e397249e8 WatchSource:0}: Error finding container f9b6a2f57c4e90f2d9fc9e4113825c0bc6cd6ecf7692ff08994ca46e397249e8: Status 404 returned error can't find the container with id f9b6a2f57c4e90f2d9fc9e4113825c0bc6cd6ecf7692ff08994ca46e397249e8
Nov 03 02:42:57.314162 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:57.314117    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s" event={"ID":"2d143031-d33e-4e37-815c-f0b6f3a12835","Type":"ContainerStarted","Data":"f9b6a2f57c4e90f2d9fc9e4113825c0bc6cd6ecf7692ff08994ca46e397249e8"}
Nov 03 02:42:58.322314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:58.322226    5508 generic.go:356] "Generic (PLEG): container finished" podID="2d143031-d33e-4e37-815c-f0b6f3a12835" containerID="e2d22024a69772fe778cd265f5935e2350d9c922a90e6d0cb3a1d165d991873d" exitCode=0
Nov 03 02:42:58.322723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:58.322313    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s" event={"ID":"2d143031-d33e-4e37-815c-f0b6f3a12835","Type":"ContainerDied","Data":"e2d22024a69772fe778cd265f5935e2350d9c922a90e6d0cb3a1d165d991873d"}
Nov 03 02:42:59.469573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:59.469539    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"
Nov 03 02:42:59.510725 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:59.510688    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-xks-ca\") pod \"2d143031-d33e-4e37-815c-f0b6f3a12835\" (UID: \"2d143031-d33e-4e37-815c-f0b6f3a12835\") "
Nov 03 02:42:59.510910 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:59.510794    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d143031-d33e-4e37-815c-f0b6f3a12835-kube-api-access-hm2k5\" (UniqueName: \"kubernetes.io/projected/2d143031-d33e-4e37-815c-f0b6f3a12835-kube-api-access-hm2k5\") pod \"2d143031-d33e-4e37-815c-f0b6f3a12835\" (UID: \"2d143031-d33e-4e37-815c-f0b6f3a12835\") "
Nov 03 02:42:59.510910 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:59.510822    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-openshift-ca\") pod \"2d143031-d33e-4e37-815c-f0b6f3a12835\" (UID: \"2d143031-d33e-4e37-815c-f0b6f3a12835\") "
Nov 03 02:42:59.511195 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:59.511161    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-openshift-ca" pod "2d143031-d33e-4e37-815c-f0b6f3a12835" (UID: "2d143031-d33e-4e37-815c-f0b6f3a12835"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:42:59.512870 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:59.512840    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-xks-ca" pod "2d143031-d33e-4e37-815c-f0b6f3a12835" (UID: "2d143031-d33e-4e37-815c-f0b6f3a12835"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:42:59.512999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:59.512946    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d143031-d33e-4e37-815c-f0b6f3a12835-kube-api-access-hm2k5" pod "2d143031-d33e-4e37-815c-f0b6f3a12835" (UID: "2d143031-d33e-4e37-815c-f0b6f3a12835"). InnerVolumeSpecName "kube-api-access-hm2k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:42:59.612627 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:59.612535    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:59.612627 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:59.612568    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm2k5\" (UniqueName: \"kubernetes.io/projected/2d143031-d33e-4e37-815c-f0b6f3a12835-kube-api-access-hm2k5\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:42:59.612627 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:42:59.612582    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d143031-d33e-4e37-815c-f0b6f3a12835-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:43:00.335236 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:00.335208    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"
Nov 03 02:43:00.335447 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:00.335204    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s" event={"ID":"2d143031-d33e-4e37-815c-f0b6f3a12835","Type":"ContainerDied","Data":"f9b6a2f57c4e90f2d9fc9e4113825c0bc6cd6ecf7692ff08994ca46e397249e8"}
Nov 03 02:43:00.335447 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:00.335325    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9b6a2f57c4e90f2d9fc9e4113825c0bc6cd6ecf7692ff08994ca46e397249e8"
Nov 03 02:43:00.769542 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:00.769508    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:43:10.045669 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:10.045630    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"]
Nov 03 02:43:10.091679 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:10.091645    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52a12add-g4t86"]
Nov 03 02:43:10.691892 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:10.691857    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3" path="/var/lib/kubelet/pods/ddb3932d-77a4-4cd8-9d06-0de4dcc0ddc3/volumes"
Nov 03 02:43:13.813552 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:43:13.813511    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:43:21.053972 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:21.053934    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"]
Nov 03 02:43:21.099776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:21.099740    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59d64720-gd84b"]
Nov 03 02:43:22.691067 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:22.691033    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eb719a8-4f64-4414-8cb4-e94c97c2c66e" path="/var/lib/kubelet/pods/7eb719a8-4f64-4414-8cb4-e94c97c2c66e/volumes"
Nov 03 02:43:22.803202 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:22.803166    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:43:41.663467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:41.663440    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:43:41.793387 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:41.793358    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:43:43.811036 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:43:43.810990    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:43:46.726660 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:46.726625    5508 scope.go:122] "RemoveContainer" containerID="f24122ad335596d4d1669318203b7bb91202128bb3bcb90b2fe060770c688150"
Nov 03 02:43:46.736273 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:43:46.736250    5508 scope.go:122] "RemoveContainer" containerID="cbaa63ccc742c056245d9bf3012ec3c6970bbfd690b88c49cbe4242078f3a570"
Nov 03 02:44:06.640492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:06.640453    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:44:13.814364 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:44:13.814319    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:44:16.729666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.729586    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"]
Nov 03 02:44:16.730076 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.730061    5508 cpu_manager.go:414] "containerMap: removing container" podUID="2d143031-d33e-4e37-815c-f0b6f3a12835" containerName="revoke-keys"
Nov 03 02:44:16.730129 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.730078    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d143031-d33e-4e37-815c-f0b6f3a12835" containerName="revoke-keys"
Nov 03 02:44:16.730166 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.730133    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d143031-d33e-4e37-815c-f0b6f3a12835" containerName="revoke-keys"
Nov 03 02:44:16.733774 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.733748    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"
Nov 03 02:44:16.782383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.782322    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:44:16.822055 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.822022    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnpzh\" (UniqueName: \"kubernetes.io/projected/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-kube-api-access-bnpzh\") pod \"maas-api-revoke-sub-e83852c7-782hp\" (UID: \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"
Nov 03 02:44:16.822275 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.822062    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e83852c7-782hp\" (UID: \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"
Nov 03 02:44:16.822275 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.822241    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e83852c7-782hp\" (UID: \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"
Nov 03 02:44:16.828643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.828608    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:44:16.831440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.831418    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"]
Nov 03 02:44:16.923327 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.923284    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnpzh\" (UniqueName: \"kubernetes.io/projected/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-kube-api-access-bnpzh\") pod \"maas-api-revoke-sub-e83852c7-782hp\" (UID: \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"
Nov 03 02:44:16.923571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.923342    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e83852c7-782hp\" (UID: \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"
Nov 03 02:44:16.923571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.923445    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e83852c7-782hp\" (UID: \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"
Nov 03 02:44:16.924052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.924023    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e83852c7-782hp\" (UID: \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"
Nov 03 02:44:16.925727 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.925707    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e83852c7-782hp\" (UID: \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"
Nov 03 02:44:16.980571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:16.980493    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnpzh\" (UniqueName: \"kubernetes.io/projected/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-kube-api-access-bnpzh\") pod \"maas-api-revoke-sub-e83852c7-782hp\" (UID: \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"
Nov 03 02:44:17.046504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:17.046466    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"
Nov 03 02:44:17.420283 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:17.420252    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"]
Nov 03 02:44:17.427610 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:44:17.427569    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5429f1b5_6d1b_4891_9c5d_80876ebd15b2.slice/crio-2559c0c08440089ea418710bcb3cd92f899a77dd72704ac7e042537ceb067ddb WatchSource:0}: Error finding container 2559c0c08440089ea418710bcb3cd92f899a77dd72704ac7e042537ceb067ddb: Status 404 returned error can't find the container with id 2559c0c08440089ea418710bcb3cd92f899a77dd72704ac7e042537ceb067ddb
Nov 03 02:44:17.813075 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:17.813036    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp" event={"ID":"5429f1b5-6d1b-4891-9c5d-80876ebd15b2","Type":"ContainerStarted","Data":"6abb34e55f916dea8b12bf14c663136a9ecd1e6f60d5816e057f47bdf7857164"}
Nov 03 02:44:17.813525 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:17.813099    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp" event={"ID":"5429f1b5-6d1b-4891-9c5d-80876ebd15b2","Type":"ContainerStarted","Data":"2559c0c08440089ea418710bcb3cd92f899a77dd72704ac7e042537ceb067ddb"}
Nov 03 02:44:17.923183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:17.923128    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp" podStartSLOduration=1.923114623 podStartE2EDuration="1.923114623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:44:17.921997894 +0000 UTC m=+5137.347604379" watchObservedRunningTime="2026-11-03 02:44:17.923114623 +0000 UTC m=+5137.348721087"
Nov 03 02:44:18.820483 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:18.820448    5508 generic.go:356] "Generic (PLEG): container finished" podID="5429f1b5-6d1b-4891-9c5d-80876ebd15b2" containerID="6abb34e55f916dea8b12bf14c663136a9ecd1e6f60d5816e057f47bdf7857164" exitCode=0
Nov 03 02:44:18.820909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:18.820545    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp" event={"ID":"5429f1b5-6d1b-4891-9c5d-80876ebd15b2","Type":"ContainerDied","Data":"6abb34e55f916dea8b12bf14c663136a9ecd1e6f60d5816e057f47bdf7857164"}
Nov 03 02:44:19.967901 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:19.967867    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"
Nov 03 02:44:20.056511 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:20.056474    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-kube-api-access-bnpzh\" (UniqueName: \"kubernetes.io/projected/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-kube-api-access-bnpzh\") pod \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\" (UID: \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\") "
Nov 03 02:44:20.056718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:20.056594    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-xks-ca\") pod \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\" (UID: \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\") "
Nov 03 02:44:20.056718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:20.056629    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-openshift-ca\") pod \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\" (UID: \"5429f1b5-6d1b-4891-9c5d-80876ebd15b2\") "
Nov 03 02:44:20.057102 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:20.057071    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-openshift-ca" pod "5429f1b5-6d1b-4891-9c5d-80876ebd15b2" (UID: "5429f1b5-6d1b-4891-9c5d-80876ebd15b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:44:20.058800 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:20.058771    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-kube-api-access-bnpzh" pod "5429f1b5-6d1b-4891-9c5d-80876ebd15b2" (UID: "5429f1b5-6d1b-4891-9c5d-80876ebd15b2"). InnerVolumeSpecName "kube-api-access-bnpzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:44:20.058800 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:20.058784    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-xks-ca" pod "5429f1b5-6d1b-4891-9c5d-80876ebd15b2" (UID: "5429f1b5-6d1b-4891-9c5d-80876ebd15b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:44:20.157758 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:20.157715    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnpzh\" (UniqueName: \"kubernetes.io/projected/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-kube-api-access-bnpzh\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:20.157758 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:20.157753    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:20.158032 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:20.157769    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5429f1b5-6d1b-4891-9c5d-80876ebd15b2-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:20.835189 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:20.835157    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"
Nov 03 02:44:20.835402 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:20.835191    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp" event={"ID":"5429f1b5-6d1b-4891-9c5d-80876ebd15b2","Type":"ContainerDied","Data":"2559c0c08440089ea418710bcb3cd92f899a77dd72704ac7e042537ceb067ddb"}
Nov 03 02:44:20.835402 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:20.835247    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2559c0c08440089ea418710bcb3cd92f899a77dd72704ac7e042537ceb067ddb"
Nov 03 02:44:27.536601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.536563    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"]
Nov 03 02:44:27.537094 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.537028    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5429f1b5-6d1b-4891-9c5d-80876ebd15b2" containerName="revoke-keys"
Nov 03 02:44:27.537094 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.537041    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5429f1b5-6d1b-4891-9c5d-80876ebd15b2" containerName="revoke-keys"
Nov 03 02:44:27.537223 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.537111    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5429f1b5-6d1b-4891-9c5d-80876ebd15b2" containerName="revoke-keys"
Nov 03 02:44:27.540722 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.540700    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"
Nov 03 02:44:27.589704 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.589674    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:44:27.635935 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.635887    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:44:27.638338 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.638306    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5efaa27-wtp5k\" (UID: \"eb156e1e-6a24-42d9-a471-64f097e7f548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"
Nov 03 02:44:27.638528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.638477    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5efaa27-wtp5k\" (UID: \"eb156e1e-6a24-42d9-a471-64f097e7f548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"
Nov 03 02:44:27.638528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.638509    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sltpp\" (UniqueName: \"kubernetes.io/projected/eb156e1e-6a24-42d9-a471-64f097e7f548-kube-api-access-sltpp\") pod \"maas-api-revoke-sub-e5efaa27-wtp5k\" (UID: \"eb156e1e-6a24-42d9-a471-64f097e7f548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"
Nov 03 02:44:27.638779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.638753    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"]
Nov 03 02:44:27.739916 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.739873    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5efaa27-wtp5k\" (UID: \"eb156e1e-6a24-42d9-a471-64f097e7f548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"
Nov 03 02:44:27.740110 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.739934    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sltpp\" (UniqueName: \"kubernetes.io/projected/eb156e1e-6a24-42d9-a471-64f097e7f548-kube-api-access-sltpp\") pod \"maas-api-revoke-sub-e5efaa27-wtp5k\" (UID: \"eb156e1e-6a24-42d9-a471-64f097e7f548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"
Nov 03 02:44:27.740110 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.739996    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5efaa27-wtp5k\" (UID: \"eb156e1e-6a24-42d9-a471-64f097e7f548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"
Nov 03 02:44:27.740590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.740561    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5efaa27-wtp5k\" (UID: \"eb156e1e-6a24-42d9-a471-64f097e7f548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"
Nov 03 02:44:27.742383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.742363    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5efaa27-wtp5k\" (UID: \"eb156e1e-6a24-42d9-a471-64f097e7f548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"
Nov 03 02:44:27.798095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.798011    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sltpp\" (UniqueName: \"kubernetes.io/projected/eb156e1e-6a24-42d9-a471-64f097e7f548-kube-api-access-sltpp\") pod \"maas-api-revoke-sub-e5efaa27-wtp5k\" (UID: \"eb156e1e-6a24-42d9-a471-64f097e7f548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"
Nov 03 02:44:27.852998 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:27.852952    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"
Nov 03 02:44:28.235662 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:28.235616    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"]
Nov 03 02:44:28.243126 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:44:28.243099    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb156e1e_6a24_42d9_a471_64f097e7f548.slice/crio-36b4c6f3fa411135c7eb207c143c17a208c28e3d4d7673d3a1e7090d154578fd WatchSource:0}: Error finding container 36b4c6f3fa411135c7eb207c143c17a208c28e3d4d7673d3a1e7090d154578fd: Status 404 returned error can't find the container with id 36b4c6f3fa411135c7eb207c143c17a208c28e3d4d7673d3a1e7090d154578fd
Nov 03 02:44:28.892491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:28.892396    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k" event={"ID":"eb156e1e-6a24-42d9-a471-64f097e7f548","Type":"ContainerStarted","Data":"293b370692b10639c8c882daac8fd58be5e45eae25a9ff0e6d5019f898b86a8d"}
Nov 03 02:44:28.892491 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:28.892478    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k" event={"ID":"eb156e1e-6a24-42d9-a471-64f097e7f548","Type":"ContainerStarted","Data":"36b4c6f3fa411135c7eb207c143c17a208c28e3d4d7673d3a1e7090d154578fd"}
Nov 03 02:44:29.001582 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:29.001447    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k" podStartSLOduration=2.001428373 podStartE2EDuration="2.001428373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:44:29.001115468 +0000 UTC m=+5148.426721950" watchObservedRunningTime="2026-11-03 02:44:29.001428373 +0000 UTC m=+5148.427034838"
Nov 03 02:44:29.899744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:29.899707    5508 generic.go:356] "Generic (PLEG): container finished" podID="eb156e1e-6a24-42d9-a471-64f097e7f548" containerID="293b370692b10639c8c882daac8fd58be5e45eae25a9ff0e6d5019f898b86a8d" exitCode=0
Nov 03 02:44:29.900168 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:29.899796    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k" event={"ID":"eb156e1e-6a24-42d9-a471-64f097e7f548","Type":"ContainerDied","Data":"293b370692b10639c8c882daac8fd58be5e45eae25a9ff0e6d5019f898b86a8d"}
Nov 03 02:44:31.044598 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:31.044571    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"
Nov 03 02:44:31.179080 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:31.179048    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-openshift-ca\") pod \"eb156e1e-6a24-42d9-a471-64f097e7f548\" (UID: \"eb156e1e-6a24-42d9-a471-64f097e7f548\") "
Nov 03 02:44:31.179295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:31.179089    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb156e1e-6a24-42d9-a471-64f097e7f548-kube-api-access-sltpp\" (UniqueName: \"kubernetes.io/projected/eb156e1e-6a24-42d9-a471-64f097e7f548-kube-api-access-sltpp\") pod \"eb156e1e-6a24-42d9-a471-64f097e7f548\" (UID: \"eb156e1e-6a24-42d9-a471-64f097e7f548\") "
Nov 03 02:44:31.179295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:31.179123    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-xks-ca\") pod \"eb156e1e-6a24-42d9-a471-64f097e7f548\" (UID: \"eb156e1e-6a24-42d9-a471-64f097e7f548\") "
Nov 03 02:44:31.179463 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:31.179441    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-openshift-ca" pod "eb156e1e-6a24-42d9-a471-64f097e7f548" (UID: "eb156e1e-6a24-42d9-a471-64f097e7f548"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:44:31.181194 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:31.181169    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb156e1e-6a24-42d9-a471-64f097e7f548-kube-api-access-sltpp" pod "eb156e1e-6a24-42d9-a471-64f097e7f548" (UID: "eb156e1e-6a24-42d9-a471-64f097e7f548"). InnerVolumeSpecName "kube-api-access-sltpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:44:31.181321 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:31.181243    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-xks-ca" pod "eb156e1e-6a24-42d9-a471-64f097e7f548" (UID: "eb156e1e-6a24-42d9-a471-64f097e7f548"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:44:31.280415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:31.280315    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:31.280415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:31.280367    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sltpp\" (UniqueName: \"kubernetes.io/projected/eb156e1e-6a24-42d9-a471-64f097e7f548-kube-api-access-sltpp\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:31.280415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:31.280383    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb156e1e-6a24-42d9-a471-64f097e7f548-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:31.913330 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:31.913299    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"
Nov 03 02:44:31.913330 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:31.913324    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k" event={"ID":"eb156e1e-6a24-42d9-a471-64f097e7f548","Type":"ContainerDied","Data":"36b4c6f3fa411135c7eb207c143c17a208c28e3d4d7673d3a1e7090d154578fd"}
Nov 03 02:44:31.913629 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:31.913383    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36b4c6f3fa411135c7eb207c143c17a208c28e3d4d7673d3a1e7090d154578fd"
Nov 03 02:44:33.174705 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:33.174671    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:44:39.124193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.124156    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"]
Nov 03 02:44:39.124676 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.124659    5508 cpu_manager.go:414] "containerMap: removing container" podUID="eb156e1e-6a24-42d9-a471-64f097e7f548" containerName="revoke-keys"
Nov 03 02:44:39.124718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.124677    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb156e1e-6a24-42d9-a471-64f097e7f548" containerName="revoke-keys"
Nov 03 02:44:39.124775 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.124763    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb156e1e-6a24-42d9-a471-64f097e7f548" containerName="revoke-keys"
Nov 03 02:44:39.128356 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.128326    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"
Nov 03 02:44:39.152986 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.152955    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47aa225f-zz72q\" (UID: \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"
Nov 03 02:44:39.153155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.152992    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47aa225f-zz72q\" (UID: \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"
Nov 03 02:44:39.153155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.153014    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qt6z\" (UniqueName: \"kubernetes.io/projected/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-kube-api-access-7qt6z\") pod \"maas-api-revoke-sub-47aa225f-zz72q\" (UID: \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"
Nov 03 02:44:39.223379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.223306    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:44:39.223379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.223324    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:44:39.226040 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.226012    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"]
Nov 03 02:44:39.254432 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.254393    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47aa225f-zz72q\" (UID: \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"
Nov 03 02:44:39.254640 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.254441    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47aa225f-zz72q\" (UID: \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"
Nov 03 02:44:39.254640 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.254472    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qt6z\" (UniqueName: \"kubernetes.io/projected/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-kube-api-access-7qt6z\") pod \"maas-api-revoke-sub-47aa225f-zz72q\" (UID: \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"
Nov 03 02:44:39.255016 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.254993    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47aa225f-zz72q\" (UID: \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"
Nov 03 02:44:39.256793 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.256770    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47aa225f-zz72q\" (UID: \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"
Nov 03 02:44:39.312219 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.312182    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qt6z\" (UniqueName: \"kubernetes.io/projected/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-kube-api-access-7qt6z\") pod \"maas-api-revoke-sub-47aa225f-zz72q\" (UID: \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"
Nov 03 02:44:39.440699 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.440662    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"
Nov 03 02:44:39.816308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.816225    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"]
Nov 03 02:44:39.823088 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:44:39.823056    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b5d5f3_b3e3_46a6_938b_fe7dc75add83.slice/crio-d9021277a9fd348d1e54e333a74e3e517e95be38265106421c9484c01a2dc46a WatchSource:0}: Error finding container d9021277a9fd348d1e54e333a74e3e517e95be38265106421c9484c01a2dc46a: Status 404 returned error can't find the container with id d9021277a9fd348d1e54e333a74e3e517e95be38265106421c9484c01a2dc46a
Nov 03 02:44:39.961306 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:39.961276    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q" event={"ID":"65b5d5f3-b3e3-46a6-938b-fe7dc75add83","Type":"ContainerStarted","Data":"d9021277a9fd348d1e54e333a74e3e517e95be38265106421c9484c01a2dc46a"}
Nov 03 02:44:40.970118 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:40.970027    5508 generic.go:356] "Generic (PLEG): container finished" podID="65b5d5f3-b3e3-46a6-938b-fe7dc75add83" containerID="861e315b6cfc9c79977d20853c2334d60c10d17539541a1e9941d1d5a4721792" exitCode=0
Nov 03 02:44:40.970532 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:40.970121    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q" event={"ID":"65b5d5f3-b3e3-46a6-938b-fe7dc75add83","Type":"ContainerDied","Data":"861e315b6cfc9c79977d20853c2334d60c10d17539541a1e9941d1d5a4721792"}
Nov 03 02:44:42.124936 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:42.124901    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"
Nov 03 02:44:42.187084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:42.187041    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-xks-ca\") pod \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\" (UID: \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\") "
Nov 03 02:44:42.187287 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:42.187155    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-kube-api-access-7qt6z\" (UniqueName: \"kubernetes.io/projected/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-kube-api-access-7qt6z\") pod \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\" (UID: \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\") "
Nov 03 02:44:42.187287 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:42.187277    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-openshift-ca\") pod \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\" (UID: \"65b5d5f3-b3e3-46a6-938b-fe7dc75add83\") "
Nov 03 02:44:42.187701 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:42.187670    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-openshift-ca" pod "65b5d5f3-b3e3-46a6-938b-fe7dc75add83" (UID: "65b5d5f3-b3e3-46a6-938b-fe7dc75add83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:44:42.189375 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:42.189324    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-xks-ca" pod "65b5d5f3-b3e3-46a6-938b-fe7dc75add83" (UID: "65b5d5f3-b3e3-46a6-938b-fe7dc75add83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:44:42.189479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:42.189422    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-kube-api-access-7qt6z" pod "65b5d5f3-b3e3-46a6-938b-fe7dc75add83" (UID: "65b5d5f3-b3e3-46a6-938b-fe7dc75add83"). InnerVolumeSpecName "kube-api-access-7qt6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:44:42.303425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:42.293013    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:42.303425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:42.293058    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:42.303425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:42.293071    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qt6z\" (UniqueName: \"kubernetes.io/projected/65b5d5f3-b3e3-46a6-938b-fe7dc75add83-kube-api-access-7qt6z\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:42.984426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:42.984314    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q" event={"ID":"65b5d5f3-b3e3-46a6-938b-fe7dc75add83","Type":"ContainerDied","Data":"d9021277a9fd348d1e54e333a74e3e517e95be38265106421c9484c01a2dc46a"}
Nov 03 02:44:42.984426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:42.984340    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"
Nov 03 02:44:42.984426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:42.984386    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9021277a9fd348d1e54e333a74e3e517e95be38265106421c9484c01a2dc46a"
Nov 03 02:44:43.812139 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:44:43.812106    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:44:46.767983 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:46.767950    5508 scope.go:122] "RemoveContainer" containerID="e9ef46d4da057c34a85d2f568bed8c49d3d40eff75501f4faba5de6cfb62ef3a"
Nov 03 02:44:46.785268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:46.785239    5508 scope.go:122] "RemoveContainer" containerID="ed3745b78c86e7893d4418adeac297b632b4635665ac3611202e0c6bd915b690"
Nov 03 02:44:50.078590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.078545    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"]
Nov 03 02:44:50.079186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.079162    5508 cpu_manager.go:414] "containerMap: removing container" podUID="65b5d5f3-b3e3-46a6-938b-fe7dc75add83" containerName="revoke-keys"
Nov 03 02:44:50.079286 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.079188    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b5d5f3-b3e3-46a6-938b-fe7dc75add83" containerName="revoke-keys"
Nov 03 02:44:50.079373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.079338    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b5d5f3-b3e3-46a6-938b-fe7dc75add83" containerName="revoke-keys"
Nov 03 02:44:50.084141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.084118    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"
Nov 03 02:44:50.133314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.133275    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:44:50.176693 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.176651    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5faf4d8d-lfh9w\" (UID: \"6b095a04-cb9f-484d-a4b6-e9e963337d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"
Nov 03 02:44:50.176908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.176816    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5faf4d8d-lfh9w\" (UID: \"6b095a04-cb9f-484d-a4b6-e9e963337d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"
Nov 03 02:44:50.176908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.176870    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7zx\" (UniqueName: \"kubernetes.io/projected/6b095a04-cb9f-484d-a4b6-e9e963337d69-kube-api-access-ph7zx\") pod \"maas-api-revoke-sub-5faf4d8d-lfh9w\" (UID: \"6b095a04-cb9f-484d-a4b6-e9e963337d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"
Nov 03 02:44:50.179453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.179431    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:44:50.182508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.182484    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"]
Nov 03 02:44:50.277722 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.277678    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5faf4d8d-lfh9w\" (UID: \"6b095a04-cb9f-484d-a4b6-e9e963337d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"
Nov 03 02:44:50.277916 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.277807    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5faf4d8d-lfh9w\" (UID: \"6b095a04-cb9f-484d-a4b6-e9e963337d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"
Nov 03 02:44:50.277916 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.277864    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7zx\" (UniqueName: \"kubernetes.io/projected/6b095a04-cb9f-484d-a4b6-e9e963337d69-kube-api-access-ph7zx\") pod \"maas-api-revoke-sub-5faf4d8d-lfh9w\" (UID: \"6b095a04-cb9f-484d-a4b6-e9e963337d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"
Nov 03 02:44:50.278359 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.278322    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5faf4d8d-lfh9w\" (UID: \"6b095a04-cb9f-484d-a4b6-e9e963337d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"
Nov 03 02:44:50.280097 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.280070    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5faf4d8d-lfh9w\" (UID: \"6b095a04-cb9f-484d-a4b6-e9e963337d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"
Nov 03 02:44:50.335416 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.335305    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7zx\" (UniqueName: \"kubernetes.io/projected/6b095a04-cb9f-484d-a4b6-e9e963337d69-kube-api-access-ph7zx\") pod \"maas-api-revoke-sub-5faf4d8d-lfh9w\" (UID: \"6b095a04-cb9f-484d-a4b6-e9e963337d69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"
Nov 03 02:44:50.397705 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.397676    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"
Nov 03 02:44:50.768144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:50.768110    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"]
Nov 03 02:44:50.776978 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:44:50.776945    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b095a04_cb9f_484d_a4b6_e9e963337d69.slice/crio-82fd6b0ac68d519833b822b86ff2a7edb974c9b73257de6352499574c6b450e1 WatchSource:0}: Error finding container 82fd6b0ac68d519833b822b86ff2a7edb974c9b73257de6352499574c6b450e1: Status 404 returned error can't find the container with id 82fd6b0ac68d519833b822b86ff2a7edb974c9b73257de6352499574c6b450e1
Nov 03 02:44:51.030999 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:51.030913    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w" event={"ID":"6b095a04-cb9f-484d-a4b6-e9e963337d69","Type":"ContainerStarted","Data":"82fd6b0ac68d519833b822b86ff2a7edb974c9b73257de6352499574c6b450e1"}
Nov 03 02:44:52.040537 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:52.040500    5508 generic.go:356] "Generic (PLEG): container finished" podID="6b095a04-cb9f-484d-a4b6-e9e963337d69" containerID="d0c50fe9a7dcaad48ae16952586b746ec9cf70e1bd28e86e4d65f61cb529deff" exitCode=0
Nov 03 02:44:52.040960 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:52.040586    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w" event={"ID":"6b095a04-cb9f-484d-a4b6-e9e963337d69","Type":"ContainerDied","Data":"d0c50fe9a7dcaad48ae16952586b746ec9cf70e1bd28e86e4d65f61cb529deff"}
Nov 03 02:44:53.176759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:53.176733    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"
Nov 03 02:44:53.310470 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:53.310392    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b095a04-cb9f-484d-a4b6-e9e963337d69-kube-api-access-ph7zx\" (UniqueName: \"kubernetes.io/projected/6b095a04-cb9f-484d-a4b6-e9e963337d69-kube-api-access-ph7zx\") pod \"6b095a04-cb9f-484d-a4b6-e9e963337d69\" (UID: \"6b095a04-cb9f-484d-a4b6-e9e963337d69\") "
Nov 03 02:44:53.310638 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:53.310503    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-xks-ca\") pod \"6b095a04-cb9f-484d-a4b6-e9e963337d69\" (UID: \"6b095a04-cb9f-484d-a4b6-e9e963337d69\") "
Nov 03 02:44:53.310638 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:53.310539    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-openshift-ca\") pod \"6b095a04-cb9f-484d-a4b6-e9e963337d69\" (UID: \"6b095a04-cb9f-484d-a4b6-e9e963337d69\") "
Nov 03 02:44:53.310956 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:53.310931    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-openshift-ca" pod "6b095a04-cb9f-484d-a4b6-e9e963337d69" (UID: "6b095a04-cb9f-484d-a4b6-e9e963337d69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:44:53.312611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:53.312581    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b095a04-cb9f-484d-a4b6-e9e963337d69-kube-api-access-ph7zx" pod "6b095a04-cb9f-484d-a4b6-e9e963337d69" (UID: "6b095a04-cb9f-484d-a4b6-e9e963337d69"). InnerVolumeSpecName "kube-api-access-ph7zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:44:53.312714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:53.312621    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-xks-ca" pod "6b095a04-cb9f-484d-a4b6-e9e963337d69" (UID: "6b095a04-cb9f-484d-a4b6-e9e963337d69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:44:53.411489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:53.411455    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph7zx\" (UniqueName: \"kubernetes.io/projected/6b095a04-cb9f-484d-a4b6-e9e963337d69-kube-api-access-ph7zx\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:53.411489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:53.411484    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:53.411489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:53.411497    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b095a04-cb9f-484d-a4b6-e9e963337d69-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:44:54.052812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:54.052781    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"
Nov 03 02:44:54.052812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:54.052808    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w" event={"ID":"6b095a04-cb9f-484d-a4b6-e9e963337d69","Type":"ContainerDied","Data":"82fd6b0ac68d519833b822b86ff2a7edb974c9b73257de6352499574c6b450e1"}
Nov 03 02:44:54.053103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:54.052853    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82fd6b0ac68d519833b822b86ff2a7edb974c9b73257de6352499574c6b450e1"
Nov 03 02:44:55.102060 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:44:55.102021    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:45:00.155401 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.155365    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"]
Nov 03 02:45:00.155826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.155813    5508 cpu_manager.go:414] "containerMap: removing container" podUID="6b095a04-cb9f-484d-a4b6-e9e963337d69" containerName="revoke-keys"
Nov 03 02:45:00.155872 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.155828    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b095a04-cb9f-484d-a4b6-e9e963337d69" containerName="revoke-keys"
Nov 03 02:45:00.155920 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.155912    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b095a04-cb9f-484d-a4b6-e9e963337d69" containerName="revoke-keys"
Nov 03 02:45:00.159364 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.159331    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"
Nov 03 02:45:00.162806 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.162783    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"]
Nov 03 02:45:00.166290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.166269    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"
Nov 03 02:45:00.208547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.208512    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:45:00.262118 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.262078    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"]
Nov 03 02:45:00.266707 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.266674    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"]
Nov 03 02:45:00.285358 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.285310    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc9e76c-9d0d-4c86-b539-6d5604480ef4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv\" (UID: \"adc9e76c-9d0d-4c86-b539-6d5604480ef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"
Nov 03 02:45:00.285549 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.285490    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbtd\" (UniqueName: \"kubernetes.io/projected/adc9e76c-9d0d-4c86-b539-6d5604480ef4-kube-api-access-5qbtd\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv\" (UID: \"adc9e76c-9d0d-4c86-b539-6d5604480ef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"
Nov 03 02:45:00.285549 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.285526    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7dcafb-e4e5-4d8d-9044-532909690dff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf\" (UID: \"0b7dcafb-e4e5-4d8d-9044-532909690dff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"
Nov 03 02:45:00.285666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.285558    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk2db\" (UniqueName: \"kubernetes.io/projected/0b7dcafb-e4e5-4d8d-9044-532909690dff-kube-api-access-rk2db\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf\" (UID: \"0b7dcafb-e4e5-4d8d-9044-532909690dff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"
Nov 03 02:45:00.387837 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.387804    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbtd\" (UniqueName: \"kubernetes.io/projected/adc9e76c-9d0d-4c86-b539-6d5604480ef4-kube-api-access-5qbtd\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv\" (UID: \"adc9e76c-9d0d-4c86-b539-6d5604480ef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"
Nov 03 02:45:00.388021 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.387865    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7dcafb-e4e5-4d8d-9044-532909690dff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf\" (UID: \"0b7dcafb-e4e5-4d8d-9044-532909690dff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"
Nov 03 02:45:00.388021 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.387916    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk2db\" (UniqueName: \"kubernetes.io/projected/0b7dcafb-e4e5-4d8d-9044-532909690dff-kube-api-access-rk2db\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf\" (UID: \"0b7dcafb-e4e5-4d8d-9044-532909690dff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"
Nov 03 02:45:00.388021 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.387958    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc9e76c-9d0d-4c86-b539-6d5604480ef4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv\" (UID: \"adc9e76c-9d0d-4c86-b539-6d5604480ef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"
Nov 03 02:45:00.388657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.388636    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7dcafb-e4e5-4d8d-9044-532909690dff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf\" (UID: \"0b7dcafb-e4e5-4d8d-9044-532909690dff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"
Nov 03 02:45:00.388707 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.388637    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc9e76c-9d0d-4c86-b539-6d5604480ef4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv\" (UID: \"adc9e76c-9d0d-4c86-b539-6d5604480ef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"
Nov 03 02:45:00.445336 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.445249    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbtd\" (UniqueName: \"kubernetes.io/projected/adc9e76c-9d0d-4c86-b539-6d5604480ef4-kube-api-access-5qbtd\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv\" (UID: \"adc9e76c-9d0d-4c86-b539-6d5604480ef4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"
Nov 03 02:45:00.445336 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.445276    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk2db\" (UniqueName: \"kubernetes.io/projected/0b7dcafb-e4e5-4d8d-9044-532909690dff-kube-api-access-rk2db\") pod \"maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf\" (UID: \"0b7dcafb-e4e5-4d8d-9044-532909690dff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"
Nov 03 02:45:00.472127 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.472095    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"
Nov 03 02:45:00.480465 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.480444    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"
Nov 03 02:45:00.848939 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.848856    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"]
Nov 03 02:45:00.855871 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:45:00.855842    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7dcafb_e4e5_4d8d_9044_532909690dff.slice/crio-bbfe3feb29d2782c683a1d6b2fab24d6168ad7a5c0aebd33f7c362a496a3b484 WatchSource:0}: Error finding container bbfe3feb29d2782c683a1d6b2fab24d6168ad7a5c0aebd33f7c362a496a3b484: Status 404 returned error can't find the container with id bbfe3feb29d2782c683a1d6b2fab24d6168ad7a5c0aebd33f7c362a496a3b484
Nov 03 02:45:00.899011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:00.898981    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"]
Nov 03 02:45:00.905751 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:45:00.905728    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc9e76c_9d0d_4c86_b539_6d5604480ef4.slice/crio-ffda88bc9d62c8a05423601d68e40c4cec28902cbe8f19aa3a7cd648d6313691 WatchSource:0}: Error finding container ffda88bc9d62c8a05423601d68e40c4cec28902cbe8f19aa3a7cd648d6313691: Status 404 returned error can't find the container with id ffda88bc9d62c8a05423601d68e40c4cec28902cbe8f19aa3a7cd648d6313691
Nov 03 02:45:01.094735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:01.094697    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf" event={"ID":"0b7dcafb-e4e5-4d8d-9044-532909690dff","Type":"ContainerStarted","Data":"bbfe3feb29d2782c683a1d6b2fab24d6168ad7a5c0aebd33f7c362a496a3b484"}
Nov 03 02:45:01.096216 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:01.096192    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv" event={"ID":"adc9e76c-9d0d-4c86-b539-6d5604480ef4","Type":"ContainerStarted","Data":"ffda88bc9d62c8a05423601d68e40c4cec28902cbe8f19aa3a7cd648d6313691"}
Nov 03 02:45:02.103067 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:02.102977    5508 generic.go:356] "Generic (PLEG): container finished" podID="0b7dcafb-e4e5-4d8d-9044-532909690dff" containerID="bcbb354dca4b446092ca4d65463cac2739b6202ee586bb42d449fb17e8998a2c" exitCode=60
Nov 03 02:45:02.103067 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:02.103054    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf" event={"ID":"0b7dcafb-e4e5-4d8d-9044-532909690dff","Type":"ContainerDied","Data":"bcbb354dca4b446092ca4d65463cac2739b6202ee586bb42d449fb17e8998a2c"}
Nov 03 02:45:02.103573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:02.103311    5508 scope.go:122] "RemoveContainer" containerID="bcbb354dca4b446092ca4d65463cac2739b6202ee586bb42d449fb17e8998a2c"
Nov 03 02:45:02.105177 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:02.105154    5508 generic.go:356] "Generic (PLEG): container finished" podID="adc9e76c-9d0d-4c86-b539-6d5604480ef4" containerID="b0cdd02dc08ba5ba16614928f467b982ed46d78c6bb91e2a686dbde109ae81e5" exitCode=60
Nov 03 02:45:02.105292 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:02.105189    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv" event={"ID":"adc9e76c-9d0d-4c86-b539-6d5604480ef4","Type":"ContainerDied","Data":"b0cdd02dc08ba5ba16614928f467b982ed46d78c6bb91e2a686dbde109ae81e5"}
Nov 03 02:45:02.105464 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:02.105445    5508 scope.go:122] "RemoveContainer" containerID="b0cdd02dc08ba5ba16614928f467b982ed46d78c6bb91e2a686dbde109ae81e5"
Nov 03 02:45:03.112598 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:03.112510    5508 generic.go:356] "Generic (PLEG): container finished" podID="adc9e76c-9d0d-4c86-b539-6d5604480ef4" containerID="e39b3ba6f26a0e89301108469c17e44ae55c67a28e6f4d3fb539c6a2ba6b233c" exitCode=60
Nov 03 02:45:03.113071 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:03.112597    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv" event={"ID":"adc9e76c-9d0d-4c86-b539-6d5604480ef4","Type":"ContainerDied","Data":"e39b3ba6f26a0e89301108469c17e44ae55c67a28e6f4d3fb539c6a2ba6b233c"}
Nov 03 02:45:03.113071 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:03.112654    5508 scope.go:122] "RemoveContainer" containerID="b0cdd02dc08ba5ba16614928f467b982ed46d78c6bb91e2a686dbde109ae81e5"
Nov 03 02:45:03.113071 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:03.112917    5508 scope.go:122] "RemoveContainer" containerID="e39b3ba6f26a0e89301108469c17e44ae55c67a28e6f4d3fb539c6a2ba6b233c"
Nov 03 02:45:03.113216 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:45:03.113193    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv_odh-ai-gateway-infra(adc9e76c-9d0d-4c86-b539-6d5604480ef4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv" podUID="adc9e76c-9d0d-4c86-b539-6d5604480ef4"
Nov 03 02:45:03.114878 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:03.114857    5508 generic.go:356] "Generic (PLEG): container finished" podID="0b7dcafb-e4e5-4d8d-9044-532909690dff" containerID="2ac4b62f25577044a24bd954ab7af4a9d85e1c4b2b7aa3e95f3bb6b1b24f9e1e" exitCode=60
Nov 03 02:45:03.115007 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:03.114933    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf" event={"ID":"0b7dcafb-e4e5-4d8d-9044-532909690dff","Type":"ContainerDied","Data":"2ac4b62f25577044a24bd954ab7af4a9d85e1c4b2b7aa3e95f3bb6b1b24f9e1e"}
Nov 03 02:45:03.115196 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:03.115180    5508 scope.go:122] "RemoveContainer" containerID="2ac4b62f25577044a24bd954ab7af4a9d85e1c4b2b7aa3e95f3bb6b1b24f9e1e"
Nov 03 02:45:03.115483 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:45:03.115462    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf_odh-ai-gateway-infra(0b7dcafb-e4e5-4d8d-9044-532909690dff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf" podUID="0b7dcafb-e4e5-4d8d-9044-532909690dff"
Nov 03 02:45:03.124722 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:03.124699    5508 scope.go:122] "RemoveContainer" containerID="bcbb354dca4b446092ca4d65463cac2739b6202ee586bb42d449fb17e8998a2c"
Nov 03 02:45:04.122490 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:04.122454    5508 scope.go:122] "RemoveContainer" containerID="e39b3ba6f26a0e89301108469c17e44ae55c67a28e6f4d3fb539c6a2ba6b233c"
Nov 03 02:45:04.122972 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:45:04.122777    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv_odh-ai-gateway-infra(adc9e76c-9d0d-4c86-b539-6d5604480ef4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv" podUID="adc9e76c-9d0d-4c86-b539-6d5604480ef4"
Nov 03 02:45:04.124066 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:04.124044    5508 scope.go:122] "RemoveContainer" containerID="2ac4b62f25577044a24bd954ab7af4a9d85e1c4b2b7aa3e95f3bb6b1b24f9e1e"
Nov 03 02:45:04.124262 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:45:04.124244    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf_odh-ai-gateway-infra(0b7dcafb-e4e5-4d8d-9044-532909690dff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf" podUID="0b7dcafb-e4e5-4d8d-9044-532909690dff"
Nov 03 02:45:12.048344 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:12.048300    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"]
Nov 03 02:45:12.094389 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:12.094331    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee878647-7lxrr"]
Nov 03 02:45:12.691309 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:12.691274    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b862333-9903-4c4c-b84a-a8519c7c0b14" path="/var/lib/kubelet/pods/6b862333-9903-4c4c-b84a-a8519c7c0b14/volumes"
Nov 03 02:45:13.811567 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:45:13.811526    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:45:14.685628 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:14.685591    5508 scope.go:122] "RemoveContainer" containerID="e39b3ba6f26a0e89301108469c17e44ae55c67a28e6f4d3fb539c6a2ba6b233c"
Nov 03 02:45:15.190208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:15.190170    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv" event={"ID":"adc9e76c-9d0d-4c86-b539-6d5604480ef4","Type":"ContainerStarted","Data":"428074a296bdc94eb444d6fed1696586356f62f9ea0f414579f67a1018c4bfbf"}
Nov 03 02:45:15.301398 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:15.301279    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv" podStartSLOduration=15.301255279 podStartE2EDuration="15.301255279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:15.297632681 +0000 UTC m=+5194.723239157" watchObservedRunningTime="2026-11-03 02:45:15.301255279 +0000 UTC m=+5194.726861742"
Nov 03 02:45:16.197299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:16.197262    5508 generic.go:356] "Generic (PLEG): container finished" podID="adc9e76c-9d0d-4c86-b539-6d5604480ef4" containerID="428074a296bdc94eb444d6fed1696586356f62f9ea0f414579f67a1018c4bfbf" exitCode=60
Nov 03 02:45:16.197845 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:16.197330    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv" event={"ID":"adc9e76c-9d0d-4c86-b539-6d5604480ef4","Type":"ContainerDied","Data":"428074a296bdc94eb444d6fed1696586356f62f9ea0f414579f67a1018c4bfbf"}
Nov 03 02:45:16.197845 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:16.197415    5508 scope.go:122] "RemoveContainer" containerID="e39b3ba6f26a0e89301108469c17e44ae55c67a28e6f4d3fb539c6a2ba6b233c"
Nov 03 02:45:16.197845 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:16.197747    5508 scope.go:122] "RemoveContainer" containerID="428074a296bdc94eb444d6fed1696586356f62f9ea0f414579f67a1018c4bfbf"
Nov 03 02:45:16.198059 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:45:16.198041    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv_odh-ai-gateway-infra(adc9e76c-9d0d-4c86-b539-6d5604480ef4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv" podUID="adc9e76c-9d0d-4c86-b539-6d5604480ef4"
Nov 03 02:45:16.252557 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:16.252515    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:45:16.308794 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:16.308756    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"]
Nov 03 02:45:17.332014 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:17.331988    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"
Nov 03 02:45:17.452821 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:17.452788    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc9e76c-9d0d-4c86-b539-6d5604480ef4-kube-api-access-5qbtd\" (UniqueName: \"kubernetes.io/projected/adc9e76c-9d0d-4c86-b539-6d5604480ef4-kube-api-access-5qbtd\") pod \"adc9e76c-9d0d-4c86-b539-6d5604480ef4\" (UID: \"adc9e76c-9d0d-4c86-b539-6d5604480ef4\") "
Nov 03 02:45:17.453006 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:17.452915    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc9e76c-9d0d-4c86-b539-6d5604480ef4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc9e76c-9d0d-4c86-b539-6d5604480ef4-maas-api-service-ca\") pod \"adc9e76c-9d0d-4c86-b539-6d5604480ef4\" (UID: \"adc9e76c-9d0d-4c86-b539-6d5604480ef4\") "
Nov 03 02:45:17.453282 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:17.453249    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc9e76c-9d0d-4c86-b539-6d5604480ef4-maas-api-service-ca" pod "adc9e76c-9d0d-4c86-b539-6d5604480ef4" (UID: "adc9e76c-9d0d-4c86-b539-6d5604480ef4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:45:17.454982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:17.454926    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc9e76c-9d0d-4c86-b539-6d5604480ef4-kube-api-access-5qbtd" pod "adc9e76c-9d0d-4c86-b539-6d5604480ef4" (UID: "adc9e76c-9d0d-4c86-b539-6d5604480ef4"). InnerVolumeSpecName "kube-api-access-5qbtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:45:17.553915 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:17.553876    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qbtd\" (UniqueName: \"kubernetes.io/projected/adc9e76c-9d0d-4c86-b539-6d5604480ef4-kube-api-access-5qbtd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:17.553915 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:17.553910    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adc9e76c-9d0d-4c86-b539-6d5604480ef4-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:17.684847 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:17.684813    5508 scope.go:122] "RemoveContainer" containerID="2ac4b62f25577044a24bd954ab7af4a9d85e1c4b2b7aa3e95f3bb6b1b24f9e1e"
Nov 03 02:45:18.212566 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:18.212535    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"
Nov 03 02:45:18.212777 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:18.212567    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv" event={"ID":"adc9e76c-9d0d-4c86-b539-6d5604480ef4","Type":"ContainerDied","Data":"ffda88bc9d62c8a05423601d68e40c4cec28902cbe8f19aa3a7cd648d6313691"}
Nov 03 02:45:18.212777 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:18.212631    5508 scope.go:122] "RemoveContainer" containerID="428074a296bdc94eb444d6fed1696586356f62f9ea0f414579f67a1018c4bfbf"
Nov 03 02:45:18.214823 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:18.214795    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf" event={"ID":"0b7dcafb-e4e5-4d8d-9044-532909690dff","Type":"ContainerStarted","Data":"ba4056e532fa9253eb009bf9df1723b6fb7c048eb8ebe9f3e8591d1175a0ddb3"}
Nov 03 02:45:18.365432 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:18.365398    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"]
Nov 03 02:45:18.369474 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:18.369442    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f0391a-29894565-k6qxv"]
Nov 03 02:45:18.503059 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:18.502999    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf" podStartSLOduration=18.502984259 podStartE2EDuration="18.502984259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:45:18.50261164 +0000 UTC m=+5197.928218122" watchObservedRunningTime="2026-11-03 02:45:18.502984259 +0000 UTC m=+5197.928590725"
Nov 03 02:45:18.691006 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:18.690973    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc9e76c-9d0d-4c86-b539-6d5604480ef4" path="/var/lib/kubelet/pods/adc9e76c-9d0d-4c86-b539-6d5604480ef4/volumes"
Nov 03 02:45:19.225129 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:19.225090    5508 generic.go:356] "Generic (PLEG): container finished" podID="0b7dcafb-e4e5-4d8d-9044-532909690dff" containerID="ba4056e532fa9253eb009bf9df1723b6fb7c048eb8ebe9f3e8591d1175a0ddb3" exitCode=60
Nov 03 02:45:19.225376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:19.225162    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf" event={"ID":"0b7dcafb-e4e5-4d8d-9044-532909690dff","Type":"ContainerDied","Data":"ba4056e532fa9253eb009bf9df1723b6fb7c048eb8ebe9f3e8591d1175a0ddb3"}
Nov 03 02:45:19.225376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:19.225235    5508 scope.go:122] "RemoveContainer" containerID="2ac4b62f25577044a24bd954ab7af4a9d85e1c4b2b7aa3e95f3bb6b1b24f9e1e"
Nov 03 02:45:19.225646 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:19.225631    5508 scope.go:122] "RemoveContainer" containerID="ba4056e532fa9253eb009bf9df1723b6fb7c048eb8ebe9f3e8591d1175a0ddb3"
Nov 03 02:45:19.225972 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:45:19.225924    5508 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf_odh-ai-gateway-infra(0b7dcafb-e4e5-4d8d-9044-532909690dff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf" podUID="0b7dcafb-e4e5-4d8d-9044-532909690dff"
Nov 03 02:45:19.513225 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:19.513147    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"]
Nov 03 02:45:20.371231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:20.371207    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"
Nov 03 02:45:20.485473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:20.484887    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7dcafb-e4e5-4d8d-9044-532909690dff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7dcafb-e4e5-4d8d-9044-532909690dff-maas-api-service-ca\") pod \"0b7dcafb-e4e5-4d8d-9044-532909690dff\" (UID: \"0b7dcafb-e4e5-4d8d-9044-532909690dff\") "
Nov 03 02:45:20.485473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:20.484973    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7dcafb-e4e5-4d8d-9044-532909690dff-kube-api-access-rk2db\" (UniqueName: \"kubernetes.io/projected/0b7dcafb-e4e5-4d8d-9044-532909690dff-kube-api-access-rk2db\") pod \"0b7dcafb-e4e5-4d8d-9044-532909690dff\" (UID: \"0b7dcafb-e4e5-4d8d-9044-532909690dff\") "
Nov 03 02:45:20.485473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:20.485335    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7dcafb-e4e5-4d8d-9044-532909690dff-maas-api-service-ca" pod "0b7dcafb-e4e5-4d8d-9044-532909690dff" (UID: "0b7dcafb-e4e5-4d8d-9044-532909690dff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:45:20.487958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:20.487932    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7dcafb-e4e5-4d8d-9044-532909690dff-kube-api-access-rk2db" pod "0b7dcafb-e4e5-4d8d-9044-532909690dff" (UID: "0b7dcafb-e4e5-4d8d-9044-532909690dff"). InnerVolumeSpecName "kube-api-access-rk2db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:45:20.586871 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:20.586835    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7dcafb-e4e5-4d8d-9044-532909690dff-maas-api-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:20.586871 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:20.586866    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk2db\" (UniqueName: \"kubernetes.io/projected/0b7dcafb-e4e5-4d8d-9044-532909690dff-kube-api-access-rk2db\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:21.239145 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:21.239108    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"
Nov 03 02:45:21.239342 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:21.239145    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf" event={"ID":"0b7dcafb-e4e5-4d8d-9044-532909690dff","Type":"ContainerDied","Data":"bbfe3feb29d2782c683a1d6b2fab24d6168ad7a5c0aebd33f7c362a496a3b484"}
Nov 03 02:45:21.239342 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:21.239207    5508 scope.go:122] "RemoveContainer" containerID="ba4056e532fa9253eb009bf9df1723b6fb7c048eb8ebe9f3e8591d1175a0ddb3"
Nov 03 02:45:21.396406 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:21.396371    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"]
Nov 03 02:45:21.400152 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:21.400091    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b6fb9b-29894565-vcqbf"]
Nov 03 02:45:22.698948 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:22.698910    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7dcafb-e4e5-4d8d-9044-532909690dff" path="/var/lib/kubelet/pods/0b7dcafb-e4e5-4d8d-9044-532909690dff/volumes"
Nov 03 02:45:28.051714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:28.051678    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"]
Nov 03 02:45:28.054829 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:28.054797    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9962ecc5-fp448"]
Nov 03 02:45:28.690331 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:28.690289    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce6081b1-f32a-4206-9041-4adb4147dcdf" path="/var/lib/kubelet/pods/ce6081b1-f32a-4206-9041-4adb4147dcdf/volumes"
Nov 03 02:45:33.094772 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:33.094734    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:45:34.050341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.050305    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"]
Nov 03 02:45:34.050577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.050538    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679" podUID="d393ed80-639b-4ae5-8c63-207fda9ec10f" containerName="maas-api" containerID="cri-o://5fc04583748bd15a0d5d245c31ea32a27b9ced95b5b4aec73873c167dfaacfa5" gracePeriod=30
Nov 03 02:45:34.310140 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.310078    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:45:34.329948 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.329914    5508 generic.go:356] "Generic (PLEG): container finished" podID="d393ed80-639b-4ae5-8c63-207fda9ec10f" containerID="5fc04583748bd15a0d5d245c31ea32a27b9ced95b5b4aec73873c167dfaacfa5" exitCode=0
Nov 03 02:45:34.330125 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.329962    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"
Nov 03 02:45:34.330125 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.329992    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679" event={"ID":"d393ed80-639b-4ae5-8c63-207fda9ec10f","Type":"ContainerDied","Data":"5fc04583748bd15a0d5d245c31ea32a27b9ced95b5b4aec73873c167dfaacfa5"}
Nov 03 02:45:34.330125 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.330050    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679" event={"ID":"d393ed80-639b-4ae5-8c63-207fda9ec10f","Type":"ContainerDied","Data":"579ba95a58911f13ce80c82676416449119ba304dbdfa841719871ea7ffb9325"}
Nov 03 02:45:34.330125 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.330074    5508 scope.go:122] "RemoveContainer" containerID="5fc04583748bd15a0d5d245c31ea32a27b9ced95b5b4aec73873c167dfaacfa5"
Nov 03 02:45:34.339124 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.339103    5508 scope.go:122] "RemoveContainer" containerID="5fc04583748bd15a0d5d245c31ea32a27b9ced95b5b4aec73873c167dfaacfa5"
Nov 03 02:45:34.339436 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:45:34.339410    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc04583748bd15a0d5d245c31ea32a27b9ced95b5b4aec73873c167dfaacfa5\": container with ID starting with 5fc04583748bd15a0d5d245c31ea32a27b9ced95b5b4aec73873c167dfaacfa5 not found: ID does not exist" containerID="5fc04583748bd15a0d5d245c31ea32a27b9ced95b5b4aec73873c167dfaacfa5"
Nov 03 02:45:34.339517 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.339447    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc04583748bd15a0d5d245c31ea32a27b9ced95b5b4aec73873c167dfaacfa5"} err="failed to get container status \"5fc04583748bd15a0d5d245c31ea32a27b9ced95b5b4aec73873c167dfaacfa5\": rpc error: code = NotFound desc = could not find container \"5fc04583748bd15a0d5d245c31ea32a27b9ced95b5b4aec73873c167dfaacfa5\": container with ID starting with 5fc04583748bd15a0d5d245c31ea32a27b9ced95b5b4aec73873c167dfaacfa5 not found: ID does not exist"
Nov 03 02:45:34.440228 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.440198    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-metrics-certs\") pod \"d393ed80-639b-4ae5-8c63-207fda9ec10f\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") "
Nov 03 02:45:34.440440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.440237    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-maas-api-tls\") pod \"d393ed80-639b-4ae5-8c63-207fda9ec10f\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") "
Nov 03 02:45:34.440440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.440265    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d393ed80-639b-4ae5-8c63-207fda9ec10f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d393ed80-639b-4ae5-8c63-207fda9ec10f-openshift-service-ca\") pod \"d393ed80-639b-4ae5-8c63-207fda9ec10f\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") "
Nov 03 02:45:34.440440 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.440311    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d393ed80-639b-4ae5-8c63-207fda9ec10f-kube-api-access-hfdl4\" (UniqueName: \"kubernetes.io/projected/d393ed80-639b-4ae5-8c63-207fda9ec10f-kube-api-access-hfdl4\") pod \"d393ed80-639b-4ae5-8c63-207fda9ec10f\" (UID: \"d393ed80-639b-4ae5-8c63-207fda9ec10f\") "
Nov 03 02:45:34.440750 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.440719    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d393ed80-639b-4ae5-8c63-207fda9ec10f-openshift-service-ca" pod "d393ed80-639b-4ae5-8c63-207fda9ec10f" (UID: "d393ed80-639b-4ae5-8c63-207fda9ec10f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:45:34.442482 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.442446    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-metrics-certs" pod "d393ed80-639b-4ae5-8c63-207fda9ec10f" (UID: "d393ed80-639b-4ae5-8c63-207fda9ec10f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:45:34.442611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.442516    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d393ed80-639b-4ae5-8c63-207fda9ec10f-kube-api-access-hfdl4" pod "d393ed80-639b-4ae5-8c63-207fda9ec10f" (UID: "d393ed80-639b-4ae5-8c63-207fda9ec10f"). InnerVolumeSpecName "kube-api-access-hfdl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:45:34.442611 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.442532    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-maas-api-tls" pod "d393ed80-639b-4ae5-8c63-207fda9ec10f" (UID: "d393ed80-639b-4ae5-8c63-207fda9ec10f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:45:34.541171 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.541137    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:34.541171 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.541169    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d393ed80-639b-4ae5-8c63-207fda9ec10f-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:34.541438 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.541181    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d393ed80-639b-4ae5-8c63-207fda9ec10f-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:34.541438 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.541191    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfdl4\" (UniqueName: \"kubernetes.io/projected/d393ed80-639b-4ae5-8c63-207fda9ec10f-kube-api-access-hfdl4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:34.586035 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.585949    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"]
Nov 03 02:45:34.638559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.638524    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"]
Nov 03 02:45:34.931515 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.931479    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"]
Nov 03 02:45:34.956245 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:34.956203    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a3dce7-7f8bbcdf87-xr679"]
Nov 03 02:45:36.691131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:36.691096    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d393ed80-639b-4ae5-8c63-207fda9ec10f" path="/var/lib/kubelet/pods/d393ed80-639b-4ae5-8c63-207fda9ec10f/volumes"
Nov 03 02:45:39.586995 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:39.586935    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll" podUID="0b500fc0-6bf5-4eca-8cd4-677f8bc466aa" containerName="payload-processing" containerID="cri-o://25102b6b81c25472afbbf41adbcf6b395c1855a966fc1eb5ff7c93d90038e5b9" gracePeriod=25
Nov 03 02:45:39.639115 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:39.639049    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7" podUID="328a97c3-2db1-41c2-b4e2-5d86dadcb453" containerName="payload-pre-processing" containerID="cri-o://b16daf20fa77d8964a8c73ec02bff9ba729079798be50690aefb5ad1834f367c" gracePeriod=25
Nov 03 02:45:40.188457 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:40.188413    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7" podUID="328a97c3-2db1-41c2-b4e2-5d86dadcb453" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:45:41.196735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:41.196697    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll" podUID="0b500fc0-6bf5-4eca-8cd4-677f8bc466aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:45:42.848073 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:42.848036    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"]
Nov 03 02:45:42.848591 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:42.848311    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd" podUID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerName="istio-proxy" containerID="cri-o://411f20fa3c295e7c4719d94c954d8bcd4dfdc484ddd931c29bd4a3b2415066f0" gracePeriod=30
Nov 03 02:45:43.813697 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:45:43.813660    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:45:44.045115 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:44.045079    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"]
Nov 03 02:45:44.091238 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:44.091167    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c63fcec7-wwmrg"]
Nov 03 02:45:44.690879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:44.690846    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="312d031e-0dcd-4f2d-b648-7037d07e75a8" path="/var/lib/kubelet/pods/312d031e-0dcd-4f2d-b648-7037d07e75a8/volumes"
Nov 03 02:45:46.094866 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:46.094768    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:45:48.093197 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.093173    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:45:48.180577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.180542    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istiod-ca-cert\") pod \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") "
Nov 03 02:45:48.180835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.180657    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-token\" (UniqueName: \"kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-token\") pod \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") "
Nov 03 02:45:48.180835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.180681    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-envoy\") pod \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") "
Nov 03 02:45:48.180835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.180701    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-kube-api-access-6f9ms\" (UniqueName: \"kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-kube-api-access-6f9ms\") pod \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") "
Nov 03 02:45:48.180835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.180721    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-podinfo\") pod \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") "
Nov 03 02:45:48.180835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.180749    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-socket\") pod \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") "
Nov 03 02:45:48.180835 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.180786    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-data\") pod \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") "
Nov 03 02:45:48.181179 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.180846    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-certs\") pod \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") "
Nov 03 02:45:48.181179 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.180874    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-credential-socket\") pod \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\" (UID: \"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4\") "
Nov 03 02:45:48.181179 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.180959    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istiod-ca-cert" pod "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" (UID: "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:45:48.181395 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.181244    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-data" pod "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" (UID: "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:45:48.181395 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.181300    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:48.181395 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.181339    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-certs" pod "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" (UID: "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:45:48.181584 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.181457    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-socket" pod "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" (UID: "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:45:48.181584 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.181535    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-credential-socket" pod "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" (UID: "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:45:48.182974 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.182943    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-envoy" pod "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" (UID: "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:45:48.183107 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.183061    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-podinfo" pod "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" (UID: "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:45:48.183331 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.183309    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-token" pod "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" (UID: "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:45:48.183331 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.183318    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-kube-api-access-6f9ms" pod "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" (UID: "e0865f37-29a7-4ef7-a522-8b7ee7d8bea4"). InnerVolumeSpecName "kube-api-access-6f9ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:45:48.283015 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.282964    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f9ms\" (UniqueName: \"kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-kube-api-access-6f9ms\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:48.283015 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.283014    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:48.283256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.283029    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:48.283256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.283043    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:48.283256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.283059    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:48.283256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.283074    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:48.283256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.283087    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:48.283256 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.283102    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:45:48.412048 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.412016    5508 generic.go:356] "Generic (PLEG): container finished" podID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerID="411f20fa3c295e7c4719d94c954d8bcd4dfdc484ddd931c29bd4a3b2415066f0" exitCode=0
Nov 03 02:45:48.412234 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.412068    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"
Nov 03 02:45:48.412234 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.412097    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd" event={"ID":"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4","Type":"ContainerDied","Data":"411f20fa3c295e7c4719d94c954d8bcd4dfdc484ddd931c29bd4a3b2415066f0"}
Nov 03 02:45:48.412234 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.412153    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd" event={"ID":"e0865f37-29a7-4ef7-a522-8b7ee7d8bea4","Type":"ContainerDied","Data":"7a74d9e2a1ae9e54c54ec8446662964ed934435fe542ede7360ef5d8052cdd92"}
Nov 03 02:45:48.412234 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.412175    5508 scope.go:122] "RemoveContainer" containerID="411f20fa3c295e7c4719d94c954d8bcd4dfdc484ddd931c29bd4a3b2415066f0"
Nov 03 02:45:48.421905 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.421882    5508 scope.go:122] "RemoveContainer" containerID="a9ecaa08945b651757ca39644425b279884e04eb830e09194588b57eee8a18df"
Nov 03 02:45:48.430371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.430339    5508 scope.go:122] "RemoveContainer" containerID="411f20fa3c295e7c4719d94c954d8bcd4dfdc484ddd931c29bd4a3b2415066f0"
Nov 03 02:45:48.431170 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:45:48.431127    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411f20fa3c295e7c4719d94c954d8bcd4dfdc484ddd931c29bd4a3b2415066f0\": container with ID starting with 411f20fa3c295e7c4719d94c954d8bcd4dfdc484ddd931c29bd4a3b2415066f0 not found: ID does not exist" containerID="411f20fa3c295e7c4719d94c954d8bcd4dfdc484ddd931c29bd4a3b2415066f0"
Nov 03 02:45:48.431260 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.431157    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411f20fa3c295e7c4719d94c954d8bcd4dfdc484ddd931c29bd4a3b2415066f0"} err="failed to get container status \"411f20fa3c295e7c4719d94c954d8bcd4dfdc484ddd931c29bd4a3b2415066f0\": rpc error: code = NotFound desc = could not find container \"411f20fa3c295e7c4719d94c954d8bcd4dfdc484ddd931c29bd4a3b2415066f0\": container with ID starting with 411f20fa3c295e7c4719d94c954d8bcd4dfdc484ddd931c29bd4a3b2415066f0 not found: ID does not exist"
Nov 03 02:45:48.431260 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.431185    5508 scope.go:122] "RemoveContainer" containerID="a9ecaa08945b651757ca39644425b279884e04eb830e09194588b57eee8a18df"
Nov 03 02:45:48.431445 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:45:48.431427    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ecaa08945b651757ca39644425b279884e04eb830e09194588b57eee8a18df\": container with ID starting with a9ecaa08945b651757ca39644425b279884e04eb830e09194588b57eee8a18df not found: ID does not exist" containerID="a9ecaa08945b651757ca39644425b279884e04eb830e09194588b57eee8a18df"
Nov 03 02:45:48.431502 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.431453    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ecaa08945b651757ca39644425b279884e04eb830e09194588b57eee8a18df"} err="failed to get container status \"a9ecaa08945b651757ca39644425b279884e04eb830e09194588b57eee8a18df\": rpc error: code = NotFound desc = could not find container \"a9ecaa08945b651757ca39644425b279884e04eb830e09194588b57eee8a18df\": container with ID starting with a9ecaa08945b651757ca39644425b279884e04eb830e09194588b57eee8a18df not found: ID does not exist"
Nov 03 02:45:48.577607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.577536    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"]
Nov 03 02:45:48.586360 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.586310    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a3dce7-openshift-default-6b6d5cc6b5-chxpd"]
Nov 03 02:45:48.690936 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:48.690855    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" path="/var/lib/kubelet/pods/e0865f37-29a7-4ef7-a522-8b7ee7d8bea4/volumes"
Nov 03 02:45:50.094720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:50.094679    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"]
Nov 03 02:45:50.187879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:50.187832    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7" podUID="328a97c3-2db1-41c2-b4e2-5d86dadcb453" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:45:50.193946 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:50.193916    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"]
Nov 03 02:45:51.196416 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:51.196380    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll" podUID="0b500fc0-6bf5-4eca-8cd4-677f8bc466aa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:45:55.095915 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:55.095843    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms" podUID="4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b" containerName="payload-processing" containerID="cri-o://3be1340a0966e6fbdd60467384e961f66bd1a18ed9ecad2a1808c3d33255f81a" gracePeriod=25
Nov 03 02:45:55.195177 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:55.195126    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8" podUID="e6502ec5-52cb-4824-894e-d77e374f67a4" containerName="payload-pre-processing" containerID="cri-o://8b04ad9506c7612475fe7fd8d48770d5ae05c142e3a9237a3cf1f9b4f893438d" gracePeriod=25
Nov 03 02:45:56.044917 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:56.044878    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"]
Nov 03 02:45:56.090824 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:56.090789    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab02f30b-jh2ps"]
Nov 03 02:45:56.690966 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:56.690934    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78804323-a28a-441e-a1c7-506a502e0469" path="/var/lib/kubelet/pods/78804323-a28a-441e-a1c7-506a502e0469/volumes"
Nov 03 02:45:57.666698 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:57.666657    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"]
Nov 03 02:45:57.666984 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:57.666959    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy" containerID="cri-o://108b4b81af888a5fc075b8c18b3222921eca8a386a149c1506438127bb9ebc65" gracePeriod=30
Nov 03 02:45:58.782579 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.782541    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"]
Nov 03 02:45:58.783010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.782940    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerName="istio-proxy"
Nov 03 02:45:58.783010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.782951    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerName="istio-proxy"
Nov 03 02:45:58.783010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.782964    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0b7dcafb-e4e5-4d8d-9044-532909690dff" containerName="cleanup"
Nov 03 02:45:58.783010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.782969    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7dcafb-e4e5-4d8d-9044-532909690dff" containerName="cleanup"
Nov 03 02:45:58.783010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.782979    5508 cpu_manager.go:414] "containerMap: removing container" podUID="adc9e76c-9d0d-4c86-b539-6d5604480ef4" containerName="cleanup"
Nov 03 02:45:58.783010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.782985    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc9e76c-9d0d-4c86-b539-6d5604480ef4" containerName="cleanup"
Nov 03 02:45:58.783010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.782993    5508 cpu_manager.go:414] "containerMap: removing container" podUID="adc9e76c-9d0d-4c86-b539-6d5604480ef4" containerName="cleanup"
Nov 03 02:45:58.783010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.782998    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc9e76c-9d0d-4c86-b539-6d5604480ef4" containerName="cleanup"
Nov 03 02:45:58.783010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.783006    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0b7dcafb-e4e5-4d8d-9044-532909690dff" containerName="cleanup"
Nov 03 02:45:58.783010 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.783012    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7dcafb-e4e5-4d8d-9044-532909690dff" containerName="cleanup"
Nov 03 02:45:58.783340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.783038    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d393ed80-639b-4ae5-8c63-207fda9ec10f" containerName="maas-api"
Nov 03 02:45:58.783340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.783043    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d393ed80-639b-4ae5-8c63-207fda9ec10f" containerName="maas-api"
Nov 03 02:45:58.783340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.783105    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc9e76c-9d0d-4c86-b539-6d5604480ef4" containerName="cleanup"
Nov 03 02:45:58.783340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.783112    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc9e76c-9d0d-4c86-b539-6d5604480ef4" containerName="cleanup"
Nov 03 02:45:58.783340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.783118    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7dcafb-e4e5-4d8d-9044-532909690dff" containerName="cleanup"
Nov 03 02:45:58.783340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.783128    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d393ed80-639b-4ae5-8c63-207fda9ec10f" containerName="maas-api"
Nov 03 02:45:58.783340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.783135    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerName="istio-proxy"
Nov 03 02:45:58.783340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.783141    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerName="istio-proxy"
Nov 03 02:45:58.783340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.783147    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7dcafb-e4e5-4d8d-9044-532909690dff" containerName="cleanup"
Nov 03 02:45:58.786603 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.786579    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"
Nov 03 02:45:58.881735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.881697    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:45:58.881990 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.881767    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:45:58.881990 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.881780    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:45:58.882140 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.881835    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:45:58.884214 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.884191    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"]
Nov 03 02:45:58.893478 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.893451    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a4c203d-5b6kf\" (UID: \"99f31f72-2947-435f-ae60-d429a765edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"
Nov 03 02:45:58.893616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.893498    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85cpz\" (UniqueName: \"kubernetes.io/projected/99f31f72-2947-435f-ae60-d429a765edd3-kube-api-access-85cpz\") pod \"maas-api-revoke-sub-4a4c203d-5b6kf\" (UID: \"99f31f72-2947-435f-ae60-d429a765edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"
Nov 03 02:45:58.893616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.893597    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a4c203d-5b6kf\" (UID: \"99f31f72-2947-435f-ae60-d429a765edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"
Nov 03 02:45:58.995225 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.995185    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a4c203d-5b6kf\" (UID: \"99f31f72-2947-435f-ae60-d429a765edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"
Nov 03 02:45:58.995427 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.995239    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85cpz\" (UniqueName: \"kubernetes.io/projected/99f31f72-2947-435f-ae60-d429a765edd3-kube-api-access-85cpz\") pod \"maas-api-revoke-sub-4a4c203d-5b6kf\" (UID: \"99f31f72-2947-435f-ae60-d429a765edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"
Nov 03 02:45:58.995427 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.995288    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a4c203d-5b6kf\" (UID: \"99f31f72-2947-435f-ae60-d429a765edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"
Nov 03 02:45:58.996021 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.996000    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a4c203d-5b6kf\" (UID: \"99f31f72-2947-435f-ae60-d429a765edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"
Nov 03 02:45:58.997710 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:58.997689    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a4c203d-5b6kf\" (UID: \"99f31f72-2947-435f-ae60-d429a765edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"
Nov 03 02:45:59.053052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:59.052977    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85cpz\" (UniqueName: \"kubernetes.io/projected/99f31f72-2947-435f-ae60-d429a765edd3-kube-api-access-85cpz\") pod \"maas-api-revoke-sub-4a4c203d-5b6kf\" (UID: \"99f31f72-2947-435f-ae60-d429a765edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"
Nov 03 02:45:59.077742 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:59.077697    5508 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:45:59.077908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:59.077769    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:45:59.099091 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:59.099054    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"
Nov 03 02:45:59.470383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:59.470338    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"]
Nov 03 02:45:59.477425 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:45:59.477395    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f31f72_2947_435f_ae60_d429a765edd3.slice/crio-0a1359b46ea7a2beeedda844a9b3978d3c332818bd3f22996f3de2918b8a84d2 WatchSource:0}: Error finding container 0a1359b46ea7a2beeedda844a9b3978d3c332818bd3f22996f3de2918b8a84d2: Status 404 returned error can't find the container with id 0a1359b46ea7a2beeedda844a9b3978d3c332818bd3f22996f3de2918b8a84d2
Nov 03 02:45:59.890672 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:59.890645    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"
Nov 03 02:45:59.928676 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:45:59.928647    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"
Nov 03 02:46:00.005421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.005232    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-kube-api-access-24g9p\" (UniqueName: \"kubernetes.io/projected/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-kube-api-access-24g9p\") pod \"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa\" (UID: \"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa\") "
Nov 03 02:46:00.005421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.005313    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/328a97c3-2db1-41c2-b4e2-5d86dadcb453-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/328a97c3-2db1-41c2-b4e2-5d86dadcb453-plugins-config-volume\") pod \"328a97c3-2db1-41c2-b4e2-5d86dadcb453\" (UID: \"328a97c3-2db1-41c2-b4e2-5d86dadcb453\") "
Nov 03 02:46:00.005421 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.005390    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-plugins-config-volume\") pod \"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa\" (UID: \"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa\") "
Nov 03 02:46:00.005698 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.005524    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/328a97c3-2db1-41c2-b4e2-5d86dadcb453-kube-api-access-wqtqh\" (UniqueName: \"kubernetes.io/projected/328a97c3-2db1-41c2-b4e2-5d86dadcb453-kube-api-access-wqtqh\") pod \"328a97c3-2db1-41c2-b4e2-5d86dadcb453\" (UID: \"328a97c3-2db1-41c2-b4e2-5d86dadcb453\") "
Nov 03 02:46:00.006517 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.006491    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328a97c3-2db1-41c2-b4e2-5d86dadcb453-plugins-config-volume" pod "328a97c3-2db1-41c2-b4e2-5d86dadcb453" (UID: "328a97c3-2db1-41c2-b4e2-5d86dadcb453"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:46:00.006875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.006848    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-plugins-config-volume" pod "0b500fc0-6bf5-4eca-8cd4-677f8bc466aa" (UID: "0b500fc0-6bf5-4eca-8cd4-677f8bc466aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:46:00.009778 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.009092    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-kube-api-access-24g9p" pod "0b500fc0-6bf5-4eca-8cd4-677f8bc466aa" (UID: "0b500fc0-6bf5-4eca-8cd4-677f8bc466aa"). InnerVolumeSpecName "kube-api-access-24g9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:46:00.009778 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.009215    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328a97c3-2db1-41c2-b4e2-5d86dadcb453-kube-api-access-wqtqh" pod "328a97c3-2db1-41c2-b4e2-5d86dadcb453" (UID: "328a97c3-2db1-41c2-b4e2-5d86dadcb453"). InnerVolumeSpecName "kube-api-access-wqtqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:46:00.107137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.107098    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqtqh\" (UniqueName: \"kubernetes.io/projected/328a97c3-2db1-41c2-b4e2-5d86dadcb453-kube-api-access-wqtqh\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:00.107137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.107132    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24g9p\" (UniqueName: \"kubernetes.io/projected/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-kube-api-access-24g9p\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:00.107137 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.107143    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/328a97c3-2db1-41c2-b4e2-5d86dadcb453-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:00.107449 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.107153    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:00.483730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.483697    5508 generic.go:356] "Generic (PLEG): container finished" podID="0b500fc0-6bf5-4eca-8cd4-677f8bc466aa" containerID="25102b6b81c25472afbbf41adbcf6b395c1855a966fc1eb5ff7c93d90038e5b9" exitCode=0
Nov 03 02:46:00.483954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.483747    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"
Nov 03 02:46:00.483954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.483789    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll" event={"ID":"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa","Type":"ContainerDied","Data":"25102b6b81c25472afbbf41adbcf6b395c1855a966fc1eb5ff7c93d90038e5b9"}
Nov 03 02:46:00.483954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.483853    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll" event={"ID":"0b500fc0-6bf5-4eca-8cd4-677f8bc466aa","Type":"ContainerDied","Data":"d0429605acf7521d24f389ec5c9ce24fcf12f88d532852074cdf7f597b5704e2"}
Nov 03 02:46:00.483954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.483884    5508 scope.go:122] "RemoveContainer" containerID="25102b6b81c25472afbbf41adbcf6b395c1855a966fc1eb5ff7c93d90038e5b9"
Nov 03 02:46:00.485479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.485455    5508 generic.go:356] "Generic (PLEG): container finished" podID="99f31f72-2947-435f-ae60-d429a765edd3" containerID="307b58b822c9feb13ecf028844f60459466c387ae8f9004d275c9d295092baad" exitCode=0
Nov 03 02:46:00.485608 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.485582    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf" event={"ID":"99f31f72-2947-435f-ae60-d429a765edd3","Type":"ContainerDied","Data":"307b58b822c9feb13ecf028844f60459466c387ae8f9004d275c9d295092baad"}
Nov 03 02:46:00.485716 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.485628    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf" event={"ID":"99f31f72-2947-435f-ae60-d429a765edd3","Type":"ContainerStarted","Data":"0a1359b46ea7a2beeedda844a9b3978d3c332818bd3f22996f3de2918b8a84d2"}
Nov 03 02:46:00.487053 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.487028    5508 generic.go:356] "Generic (PLEG): container finished" podID="328a97c3-2db1-41c2-b4e2-5d86dadcb453" containerID="b16daf20fa77d8964a8c73ec02bff9ba729079798be50690aefb5ad1834f367c" exitCode=0
Nov 03 02:46:00.487173 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.487068    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"
Nov 03 02:46:00.487173 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.487065    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7" event={"ID":"328a97c3-2db1-41c2-b4e2-5d86dadcb453","Type":"ContainerDied","Data":"b16daf20fa77d8964a8c73ec02bff9ba729079798be50690aefb5ad1834f367c"}
Nov 03 02:46:00.487285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.487199    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7" event={"ID":"328a97c3-2db1-41c2-b4e2-5d86dadcb453","Type":"ContainerDied","Data":"e1a7c73e4731cb2c021cddc71499aa6c0c03a3dfb2521a22ae24ac27346f68d1"}
Nov 03 02:46:00.493815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.493791    5508 scope.go:122] "RemoveContainer" containerID="25102b6b81c25472afbbf41adbcf6b395c1855a966fc1eb5ff7c93d90038e5b9"
Nov 03 02:46:00.494099 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:46:00.494079    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25102b6b81c25472afbbf41adbcf6b395c1855a966fc1eb5ff7c93d90038e5b9\": container with ID starting with 25102b6b81c25472afbbf41adbcf6b395c1855a966fc1eb5ff7c93d90038e5b9 not found: ID does not exist" containerID="25102b6b81c25472afbbf41adbcf6b395c1855a966fc1eb5ff7c93d90038e5b9"
Nov 03 02:46:00.494153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.494112    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25102b6b81c25472afbbf41adbcf6b395c1855a966fc1eb5ff7c93d90038e5b9"} err="failed to get container status \"25102b6b81c25472afbbf41adbcf6b395c1855a966fc1eb5ff7c93d90038e5b9\": rpc error: code = NotFound desc = could not find container \"25102b6b81c25472afbbf41adbcf6b395c1855a966fc1eb5ff7c93d90038e5b9\": container with ID starting with 25102b6b81c25472afbbf41adbcf6b395c1855a966fc1eb5ff7c93d90038e5b9 not found: ID does not exist"
Nov 03 02:46:00.494153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.494140    5508 scope.go:122] "RemoveContainer" containerID="b16daf20fa77d8964a8c73ec02bff9ba729079798be50690aefb5ad1834f367c"
Nov 03 02:46:00.502570 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.502554    5508 scope.go:122] "RemoveContainer" containerID="b16daf20fa77d8964a8c73ec02bff9ba729079798be50690aefb5ad1834f367c"
Nov 03 02:46:00.502829 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:46:00.502810    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b16daf20fa77d8964a8c73ec02bff9ba729079798be50690aefb5ad1834f367c\": container with ID starting with b16daf20fa77d8964a8c73ec02bff9ba729079798be50690aefb5ad1834f367c not found: ID does not exist" containerID="b16daf20fa77d8964a8c73ec02bff9ba729079798be50690aefb5ad1834f367c"
Nov 03 02:46:00.502873 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.502835    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b16daf20fa77d8964a8c73ec02bff9ba729079798be50690aefb5ad1834f367c"} err="failed to get container status \"b16daf20fa77d8964a8c73ec02bff9ba729079798be50690aefb5ad1834f367c\": rpc error: code = NotFound desc = could not find container \"b16daf20fa77d8964a8c73ec02bff9ba729079798be50690aefb5ad1834f367c\": container with ID starting with b16daf20fa77d8964a8c73ec02bff9ba729079798be50690aefb5ad1834f367c not found: ID does not exist"
Nov 03 02:46:00.749976 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.749934    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"]
Nov 03 02:46:00.754208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.754174    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a3dce7-77cb47d99-tjvm7"]
Nov 03 02:46:00.855321 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.855283    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"]
Nov 03 02:46:00.859720 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:00.859691    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a3dce7-54c776d7dc-v4bll"]
Nov 03 02:46:01.569486 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:01.569444    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8" podUID="e6502ec5-52cb-4824-894e-d77e374f67a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:46:01.569952 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:01.569444    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms" podUID="4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:46:01.629966 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:01.629939    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"
Nov 03 02:46:01.725589 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:01.725554    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-openshift-ca\") pod \"99f31f72-2947-435f-ae60-d429a765edd3\" (UID: \"99f31f72-2947-435f-ae60-d429a765edd3\") "
Nov 03 02:46:01.725768 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:01.725613    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-xks-ca\") pod \"99f31f72-2947-435f-ae60-d429a765edd3\" (UID: \"99f31f72-2947-435f-ae60-d429a765edd3\") "
Nov 03 02:46:01.725768 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:01.725647    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f31f72-2947-435f-ae60-d429a765edd3-kube-api-access-85cpz\" (UniqueName: \"kubernetes.io/projected/99f31f72-2947-435f-ae60-d429a765edd3-kube-api-access-85cpz\") pod \"99f31f72-2947-435f-ae60-d429a765edd3\" (UID: \"99f31f72-2947-435f-ae60-d429a765edd3\") "
Nov 03 02:46:01.725939 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:01.725912    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-openshift-ca" pod "99f31f72-2947-435f-ae60-d429a765edd3" (UID: "99f31f72-2947-435f-ae60-d429a765edd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:46:01.727779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:01.727748    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-xks-ca" pod "99f31f72-2947-435f-ae60-d429a765edd3" (UID: "99f31f72-2947-435f-ae60-d429a765edd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:46:01.727908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:01.727783    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f31f72-2947-435f-ae60-d429a765edd3-kube-api-access-85cpz" pod "99f31f72-2947-435f-ae60-d429a765edd3" (UID: "99f31f72-2947-435f-ae60-d429a765edd3"). InnerVolumeSpecName "kube-api-access-85cpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:46:01.827218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:01.827134    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:01.827218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:01.827170    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f31f72-2947-435f-ae60-d429a765edd3-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:01.827218 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:01.827185    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85cpz\" (UniqueName: \"kubernetes.io/projected/99f31f72-2947-435f-ae60-d429a765edd3-kube-api-access-85cpz\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:02.172062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:02.172025    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:46:02.502062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:02.501982    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"
Nov 03 02:46:02.502062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:02.501987    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf" event={"ID":"99f31f72-2947-435f-ae60-d429a765edd3","Type":"ContainerDied","Data":"0a1359b46ea7a2beeedda844a9b3978d3c332818bd3f22996f3de2918b8a84d2"}
Nov 03 02:46:02.502062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:02.502037    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a1359b46ea7a2beeedda844a9b3978d3c332818bd3f22996f3de2918b8a84d2"
Nov 03 02:46:02.690659 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:02.690631    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b500fc0-6bf5-4eca-8cd4-677f8bc466aa" path="/var/lib/kubelet/pods/0b500fc0-6bf5-4eca-8cd4-677f8bc466aa/volumes"
Nov 03 02:46:02.691011 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:02.691000    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="328a97c3-2db1-41c2-b4e2-5d86dadcb453" path="/var/lib/kubelet/pods/328a97c3-2db1-41c2-b4e2-5d86dadcb453/volumes"
Nov 03 02:46:02.927989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:02.927956    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:46:03.037263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037175    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-podinfo\") pod \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") "
Nov 03 02:46:03.037263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037217    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-data\") pod \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") "
Nov 03 02:46:03.037263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037258    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-credential-socket\") pod \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") "
Nov 03 02:46:03.037612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037277    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-certs\") pod \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") "
Nov 03 02:46:03.037612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037297    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-token\" (UniqueName: \"kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-token\") pod \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") "
Nov 03 02:46:03.037612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037333    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-kube-api-access-8227j\" (UniqueName: \"kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-kube-api-access-8227j\") pod \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") "
Nov 03 02:46:03.037612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037394    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istiod-ca-cert\") pod \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") "
Nov 03 02:46:03.037612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037415    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-socket\") pod \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") "
Nov 03 02:46:03.037612 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037436    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-envoy\") pod \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\" (UID: \"04df94de-2714-4273-b4c2-3f38f5b3fbd4\") "
Nov 03 02:46:03.037934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037673    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-credential-socket" pod "04df94de-2714-4273-b4c2-3f38f5b3fbd4" (UID: "04df94de-2714-4273-b4c2-3f38f5b3fbd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:03.037934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037761    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-socket" pod "04df94de-2714-4273-b4c2-3f38f5b3fbd4" (UID: "04df94de-2714-4273-b4c2-3f38f5b3fbd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:03.037934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037777    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-data" pod "04df94de-2714-4273-b4c2-3f38f5b3fbd4" (UID: "04df94de-2714-4273-b4c2-3f38f5b3fbd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:03.037934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037888    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:03.037934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.037904    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-certs" pod "04df94de-2714-4273-b4c2-3f38f5b3fbd4" (UID: "04df94de-2714-4273-b4c2-3f38f5b3fbd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:03.038211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.038057    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istiod-ca-cert" pod "04df94de-2714-4273-b4c2-3f38f5b3fbd4" (UID: "04df94de-2714-4273-b4c2-3f38f5b3fbd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:46:03.039542 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.039520    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-podinfo" pod "04df94de-2714-4273-b4c2-3f38f5b3fbd4" (UID: "04df94de-2714-4273-b4c2-3f38f5b3fbd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:46:03.039655 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.039586    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-token" pod "04df94de-2714-4273-b4c2-3f38f5b3fbd4" (UID: "04df94de-2714-4273-b4c2-3f38f5b3fbd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:46:03.039995 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.039980    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-kube-api-access-8227j" pod "04df94de-2714-4273-b4c2-3f38f5b3fbd4" (UID: "04df94de-2714-4273-b4c2-3f38f5b3fbd4"). InnerVolumeSpecName "kube-api-access-8227j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:46:03.040057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.039986    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-envoy" pod "04df94de-2714-4273-b4c2-3f38f5b3fbd4" (UID: "04df94de-2714-4273-b4c2-3f38f5b3fbd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:03.139425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.139386    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:03.139425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.139418    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:03.139425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.139427    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:03.139691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.139436    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:03.139691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.139448    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8227j\" (UniqueName: \"kubernetes.io/projected/04df94de-2714-4273-b4c2-3f38f5b3fbd4-kube-api-access-8227j\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:03.139691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.139458    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:03.139691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.139468    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:03.139691 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.139477    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04df94de-2714-4273-b4c2-3f38f5b3fbd4-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:03.510286 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.510246    5508 generic.go:356] "Generic (PLEG): container finished" podID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerID="108b4b81af888a5fc075b8c18b3222921eca8a386a149c1506438127bb9ebc65" exitCode=0
Nov 03 02:46:03.510492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.510307    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"
Nov 03 02:46:03.510492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.510342    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" event={"ID":"04df94de-2714-4273-b4c2-3f38f5b3fbd4","Type":"ContainerDied","Data":"108b4b81af888a5fc075b8c18b3222921eca8a386a149c1506438127bb9ebc65"}
Nov 03 02:46:03.510492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.510411    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt" event={"ID":"04df94de-2714-4273-b4c2-3f38f5b3fbd4","Type":"ContainerDied","Data":"b262dc570e4d3b4054ddbe4312252d00b453040a4d74165f80a85bc81847fd41"}
Nov 03 02:46:03.510492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.510433    5508 scope.go:122] "RemoveContainer" containerID="108b4b81af888a5fc075b8c18b3222921eca8a386a149c1506438127bb9ebc65"
Nov 03 02:46:03.520950 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.520929    5508 scope.go:122] "RemoveContainer" containerID="177fda0f752940c72f56c326490e8dcc649ee163eb0ab90584c313d4cf267784"
Nov 03 02:46:03.529608 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.529585    5508 scope.go:122] "RemoveContainer" containerID="108b4b81af888a5fc075b8c18b3222921eca8a386a149c1506438127bb9ebc65"
Nov 03 02:46:03.529894 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:46:03.529874    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"108b4b81af888a5fc075b8c18b3222921eca8a386a149c1506438127bb9ebc65\": container with ID starting with 108b4b81af888a5fc075b8c18b3222921eca8a386a149c1506438127bb9ebc65 not found: ID does not exist" containerID="108b4b81af888a5fc075b8c18b3222921eca8a386a149c1506438127bb9ebc65"
Nov 03 02:46:03.529972 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.529907    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"108b4b81af888a5fc075b8c18b3222921eca8a386a149c1506438127bb9ebc65"} err="failed to get container status \"108b4b81af888a5fc075b8c18b3222921eca8a386a149c1506438127bb9ebc65\": rpc error: code = NotFound desc = could not find container \"108b4b81af888a5fc075b8c18b3222921eca8a386a149c1506438127bb9ebc65\": container with ID starting with 108b4b81af888a5fc075b8c18b3222921eca8a386a149c1506438127bb9ebc65 not found: ID does not exist"
Nov 03 02:46:03.529972 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.529932    5508 scope.go:122] "RemoveContainer" containerID="177fda0f752940c72f56c326490e8dcc649ee163eb0ab90584c313d4cf267784"
Nov 03 02:46:03.530200 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:46:03.530183    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177fda0f752940c72f56c326490e8dcc649ee163eb0ab90584c313d4cf267784\": container with ID starting with 177fda0f752940c72f56c326490e8dcc649ee163eb0ab90584c313d4cf267784 not found: ID does not exist" containerID="177fda0f752940c72f56c326490e8dcc649ee163eb0ab90584c313d4cf267784"
Nov 03 02:46:03.530239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.530207    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177fda0f752940c72f56c326490e8dcc649ee163eb0ab90584c313d4cf267784"} err="failed to get container status \"177fda0f752940c72f56c326490e8dcc649ee163eb0ab90584c313d4cf267784\": rpc error: code = NotFound desc = could not find container \"177fda0f752940c72f56c326490e8dcc649ee163eb0ab90584c313d4cf267784\": container with ID starting with 177fda0f752940c72f56c326490e8dcc649ee163eb0ab90584c313d4cf267784 not found: ID does not exist"
Nov 03 02:46:03.712052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.712017    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"]
Nov 03 02:46:03.758674 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:03.758636    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f0391a-openshift-default-7fb886fbcc-bpvxt"]
Nov 03 02:46:04.691456 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:04.691342    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" path="/var/lib/kubelet/pods/04df94de-2714-4273-b4c2-3f38f5b3fbd4/volumes"
Nov 03 02:46:11.569469 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.569427    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms" podUID="4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:46:11.569971 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.569437    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8" podUID="e6502ec5-52cb-4824-894e-d77e374f67a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:46:11.582946 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.582901    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"]
Nov 03 02:46:11.583571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.583546    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0b500fc0-6bf5-4eca-8cd4-677f8bc466aa" containerName="payload-processing"
Nov 03 02:46:11.583786 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.583771    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b500fc0-6bf5-4eca-8cd4-677f8bc466aa" containerName="payload-processing"
Nov 03 02:46:11.583958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.583935    5508 cpu_manager.go:414] "containerMap: removing container" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy"
Nov 03 02:46:11.583958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.583948    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy"
Nov 03 02:46:11.584120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.583964    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerName="istio-proxy"
Nov 03 02:46:11.584120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.583972    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0865f37-29a7-4ef7-a522-8b7ee7d8bea4" containerName="istio-proxy"
Nov 03 02:46:11.584120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.583990    5508 cpu_manager.go:414] "containerMap: removing container" podUID="adc9e76c-9d0d-4c86-b539-6d5604480ef4" containerName="cleanup"
Nov 03 02:46:11.584120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584000    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc9e76c-9d0d-4c86-b539-6d5604480ef4" containerName="cleanup"
Nov 03 02:46:11.584120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584013    5508 cpu_manager.go:414] "containerMap: removing container" podUID="99f31f72-2947-435f-ae60-d429a765edd3" containerName="revoke-keys"
Nov 03 02:46:11.584120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584021    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f31f72-2947-435f-ae60-d429a765edd3" containerName="revoke-keys"
Nov 03 02:46:11.584120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584031    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0b7dcafb-e4e5-4d8d-9044-532909690dff" containerName="cleanup"
Nov 03 02:46:11.584120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584039    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7dcafb-e4e5-4d8d-9044-532909690dff" containerName="cleanup"
Nov 03 02:46:11.584120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584071    5508 cpu_manager.go:414] "containerMap: removing container" podUID="328a97c3-2db1-41c2-b4e2-5d86dadcb453" containerName="payload-pre-processing"
Nov 03 02:46:11.584120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584078    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="328a97c3-2db1-41c2-b4e2-5d86dadcb453" containerName="payload-pre-processing"
Nov 03 02:46:11.584120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584093    5508 cpu_manager.go:414] "containerMap: removing container" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy"
Nov 03 02:46:11.584120 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584100    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy"
Nov 03 02:46:11.584635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584228    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="328a97c3-2db1-41c2-b4e2-5d86dadcb453" containerName="payload-pre-processing"
Nov 03 02:46:11.584635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584240    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy"
Nov 03 02:46:11.584635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584251    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b500fc0-6bf5-4eca-8cd4-677f8bc466aa" containerName="payload-processing"
Nov 03 02:46:11.584635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584260    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="04df94de-2714-4273-b4c2-3f38f5b3fbd4" containerName="istio-proxy"
Nov 03 02:46:11.584635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584270    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc9e76c-9d0d-4c86-b539-6d5604480ef4" containerName="cleanup"
Nov 03 02:46:11.584635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584278    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f31f72-2947-435f-ae60-d429a765edd3" containerName="revoke-keys"
Nov 03 02:46:11.584635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.584286    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7dcafb-e4e5-4d8d-9044-532909690dff" containerName="cleanup"
Nov 03 02:46:11.589528 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.589505    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"
Nov 03 02:46:11.643199 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.643157    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"]
Nov 03 02:46:11.669371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.669299    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:46:11.669371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.669339    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:46:11.669744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.669719    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:46:11.686944 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.686911    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fgwsl\""
Nov 03 02:46:11.725575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.725540    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c142b96-gpkml\" (UID: \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"
Nov 03 02:46:11.725778 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.725616    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c142b96-gpkml\" (UID: \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"
Nov 03 02:46:11.725778 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.725666    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72npw\" (UniqueName: \"kubernetes.io/projected/d95f2e42-accd-4be5-bcdb-aec368ad11a0-kube-api-access-72npw\") pod \"maas-api-revoke-sub-2c142b96-gpkml\" (UID: \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"
Nov 03 02:46:11.827084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.826988    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c142b96-gpkml\" (UID: \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"
Nov 03 02:46:11.827084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.827054    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c142b96-gpkml\" (UID: \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"
Nov 03 02:46:11.827295 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.827096    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72npw\" (UniqueName: \"kubernetes.io/projected/d95f2e42-accd-4be5-bcdb-aec368ad11a0-kube-api-access-72npw\") pod \"maas-api-revoke-sub-2c142b96-gpkml\" (UID: \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"
Nov 03 02:46:11.827695 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.827676    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c142b96-gpkml\" (UID: \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"
Nov 03 02:46:11.829519 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.829491    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c142b96-gpkml\" (UID: \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"
Nov 03 02:46:11.886493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.885998    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72npw\" (UniqueName: \"kubernetes.io/projected/d95f2e42-accd-4be5-bcdb-aec368ad11a0-kube-api-access-72npw\") pod \"maas-api-revoke-sub-2c142b96-gpkml\" (UID: \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"
Nov 03 02:46:11.906000 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:11.905779    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"
Nov 03 02:46:12.300596 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:12.300551    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"]
Nov 03 02:46:12.308789 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:46:12.308763    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95f2e42_accd_4be5_bcdb_aec368ad11a0.slice/crio-10c99e1b2648ee42cc4aabb9f9588c5aa1a91d69531ed9b7bdee474791095745 WatchSource:0}: Error finding container 10c99e1b2648ee42cc4aabb9f9588c5aa1a91d69531ed9b7bdee474791095745: Status 404 returned error can't find the container with id 10c99e1b2648ee42cc4aabb9f9588c5aa1a91d69531ed9b7bdee474791095745
Nov 03 02:46:12.571161 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:12.571126    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml" event={"ID":"d95f2e42-accd-4be5-bcdb-aec368ad11a0","Type":"ContainerStarted","Data":"6dd7231c896dfad31837574fd4b95348795a85c133888ed1d5179b5dfa71c4a2"}
Nov 03 02:46:12.571568 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:12.571191    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml" event={"ID":"d95f2e42-accd-4be5-bcdb-aec368ad11a0","Type":"ContainerStarted","Data":"10c99e1b2648ee42cc4aabb9f9588c5aa1a91d69531ed9b7bdee474791095745"}
Nov 03 02:46:12.682139 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:12.682070    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml" podStartSLOduration=1.6820546859999999 podStartE2EDuration="1.682054686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:46:12.679862125 +0000 UTC m=+5252.105468591" watchObservedRunningTime="2026-11-03 02:46:12.682054686 +0000 UTC m=+5252.107661150"
Nov 03 02:46:13.579496 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:13.579457    5508 generic.go:356] "Generic (PLEG): container finished" podID="d95f2e42-accd-4be5-bcdb-aec368ad11a0" containerID="6dd7231c896dfad31837574fd4b95348795a85c133888ed1d5179b5dfa71c4a2" exitCode=0
Nov 03 02:46:13.579920 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:13.579534    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml" event={"ID":"d95f2e42-accd-4be5-bcdb-aec368ad11a0","Type":"ContainerDied","Data":"6dd7231c896dfad31837574fd4b95348795a85c133888ed1d5179b5dfa71c4a2"}
Nov 03 02:46:13.811791 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:46:13.811755    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:46:14.716773 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:14.716745    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"
Nov 03 02:46:14.856505 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:14.856418    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-openshift-ca\") pod \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\" (UID: \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\") "
Nov 03 02:46:14.856665 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:14.856546    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-xks-ca\") pod \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\" (UID: \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\") "
Nov 03 02:46:14.856665 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:14.856637    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95f2e42-accd-4be5-bcdb-aec368ad11a0-kube-api-access-72npw\" (UniqueName: \"kubernetes.io/projected/d95f2e42-accd-4be5-bcdb-aec368ad11a0-kube-api-access-72npw\") pod \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\" (UID: \"d95f2e42-accd-4be5-bcdb-aec368ad11a0\") "
Nov 03 02:46:14.856861 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:14.856831    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-openshift-ca" pod "d95f2e42-accd-4be5-bcdb-aec368ad11a0" (UID: "d95f2e42-accd-4be5-bcdb-aec368ad11a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:46:14.856953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:14.856939    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:14.858647 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:14.858619    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-xks-ca" pod "d95f2e42-accd-4be5-bcdb-aec368ad11a0" (UID: "d95f2e42-accd-4be5-bcdb-aec368ad11a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:46:14.858802 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:14.858775    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95f2e42-accd-4be5-bcdb-aec368ad11a0-kube-api-access-72npw" pod "d95f2e42-accd-4be5-bcdb-aec368ad11a0" (UID: "d95f2e42-accd-4be5-bcdb-aec368ad11a0"). InnerVolumeSpecName "kube-api-access-72npw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:46:14.957575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:14.957535    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95f2e42-accd-4be5-bcdb-aec368ad11a0-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:14.957575 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:14.957567    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72npw\" (UniqueName: \"kubernetes.io/projected/d95f2e42-accd-4be5-bcdb-aec368ad11a0-kube-api-access-72npw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:15.340044 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.340012    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"
Nov 03 02:46:15.444148 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.444121    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"
Nov 03 02:46:15.462625 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.462595    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-kube-api-access-x7s26\" (UniqueName: \"kubernetes.io/projected/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-kube-api-access-x7s26\") pod \"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b\" (UID: \"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b\") "
Nov 03 02:46:15.462820 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.462660    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-plugins-config-volume\") pod \"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b\" (UID: \"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b\") "
Nov 03 02:46:15.463108 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.463085    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-plugins-config-volume" pod "4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b" (UID: "4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:46:15.464848 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.464820    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-kube-api-access-x7s26" pod "4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b" (UID: "4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b"). InnerVolumeSpecName "kube-api-access-x7s26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:46:15.570853 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.570808    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6502ec5-52cb-4824-894e-d77e374f67a4-plugins-config-volume" pod "e6502ec5-52cb-4824-894e-d77e374f67a4" (UID: "e6502ec5-52cb-4824-894e-d77e374f67a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:46:15.571503 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.571476    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6502ec5-52cb-4824-894e-d77e374f67a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6502ec5-52cb-4824-894e-d77e374f67a4-plugins-config-volume\") pod \"e6502ec5-52cb-4824-894e-d77e374f67a4\" (UID: \"e6502ec5-52cb-4824-894e-d77e374f67a4\") "
Nov 03 02:46:15.571740 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.571587    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6502ec5-52cb-4824-894e-d77e374f67a4-kube-api-access-vqzp9\" (UniqueName: \"kubernetes.io/projected/e6502ec5-52cb-4824-894e-d77e374f67a4-kube-api-access-vqzp9\") pod \"e6502ec5-52cb-4824-894e-d77e374f67a4\" (UID: \"e6502ec5-52cb-4824-894e-d77e374f67a4\") "
Nov 03 02:46:15.572166 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.572121    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7s26\" (UniqueName: \"kubernetes.io/projected/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-kube-api-access-x7s26\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:15.572166 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.572153    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:15.572426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.572172    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6502ec5-52cb-4824-894e-d77e374f67a4-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:15.604380 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.603690    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6502ec5-52cb-4824-894e-d77e374f67a4-kube-api-access-vqzp9" pod "e6502ec5-52cb-4824-894e-d77e374f67a4" (UID: "e6502ec5-52cb-4824-894e-d77e374f67a4"). InnerVolumeSpecName "kube-api-access-vqzp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:46:15.609299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.609264    5508 generic.go:356] "Generic (PLEG): container finished" podID="4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b" containerID="3be1340a0966e6fbdd60467384e961f66bd1a18ed9ecad2a1808c3d33255f81a" exitCode=0
Nov 03 02:46:15.609522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.609342    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"
Nov 03 02:46:15.609522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.609369    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms" event={"ID":"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b","Type":"ContainerDied","Data":"3be1340a0966e6fbdd60467384e961f66bd1a18ed9ecad2a1808c3d33255f81a"}
Nov 03 02:46:15.609522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.609465    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms" event={"ID":"4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b","Type":"ContainerDied","Data":"08ec46881f0d826724b79e53a550779e60adf98905fead1ccf44690f7ff6b62e"}
Nov 03 02:46:15.609522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.609495    5508 scope.go:122] "RemoveContainer" containerID="3be1340a0966e6fbdd60467384e961f66bd1a18ed9ecad2a1808c3d33255f81a"
Nov 03 02:46:15.611109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.610923    5508 generic.go:356] "Generic (PLEG): container finished" podID="e6502ec5-52cb-4824-894e-d77e374f67a4" containerID="8b04ad9506c7612475fe7fd8d48770d5ae05c142e3a9237a3cf1f9b4f893438d" exitCode=0
Nov 03 02:46:15.611109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.611001    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8" event={"ID":"e6502ec5-52cb-4824-894e-d77e374f67a4","Type":"ContainerDied","Data":"8b04ad9506c7612475fe7fd8d48770d5ae05c142e3a9237a3cf1f9b4f893438d"}
Nov 03 02:46:15.611109 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.611044    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8" event={"ID":"e6502ec5-52cb-4824-894e-d77e374f67a4","Type":"ContainerDied","Data":"edebe6af840824ac70a39fbfd6d64154be1275c47cc635980a1da79ab3cb00e3"}
Nov 03 02:46:15.612030 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.611228    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"
Nov 03 02:46:15.613323 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.613293    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml" event={"ID":"d95f2e42-accd-4be5-bcdb-aec368ad11a0","Type":"ContainerDied","Data":"10c99e1b2648ee42cc4aabb9f9588c5aa1a91d69531ed9b7bdee474791095745"}
Nov 03 02:46:15.613473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.613336    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10c99e1b2648ee42cc4aabb9f9588c5aa1a91d69531ed9b7bdee474791095745"
Nov 03 02:46:15.613473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.613413    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"
Nov 03 02:46:15.629797 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.629777    5508 scope.go:122] "RemoveContainer" containerID="3be1340a0966e6fbdd60467384e961f66bd1a18ed9ecad2a1808c3d33255f81a"
Nov 03 02:46:15.630104 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:46:15.630082    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be1340a0966e6fbdd60467384e961f66bd1a18ed9ecad2a1808c3d33255f81a\": container with ID starting with 3be1340a0966e6fbdd60467384e961f66bd1a18ed9ecad2a1808c3d33255f81a not found: ID does not exist" containerID="3be1340a0966e6fbdd60467384e961f66bd1a18ed9ecad2a1808c3d33255f81a"
Nov 03 02:46:15.630164 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.630113    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be1340a0966e6fbdd60467384e961f66bd1a18ed9ecad2a1808c3d33255f81a"} err="failed to get container status \"3be1340a0966e6fbdd60467384e961f66bd1a18ed9ecad2a1808c3d33255f81a\": rpc error: code = NotFound desc = could not find container \"3be1340a0966e6fbdd60467384e961f66bd1a18ed9ecad2a1808c3d33255f81a\": container with ID starting with 3be1340a0966e6fbdd60467384e961f66bd1a18ed9ecad2a1808c3d33255f81a not found: ID does not exist"
Nov 03 02:46:15.630164 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.630133    5508 scope.go:122] "RemoveContainer" containerID="8b04ad9506c7612475fe7fd8d48770d5ae05c142e3a9237a3cf1f9b4f893438d"
Nov 03 02:46:15.638938 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.638916    5508 scope.go:122] "RemoveContainer" containerID="8b04ad9506c7612475fe7fd8d48770d5ae05c142e3a9237a3cf1f9b4f893438d"
Nov 03 02:46:15.639242 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:46:15.639226    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b04ad9506c7612475fe7fd8d48770d5ae05c142e3a9237a3cf1f9b4f893438d\": container with ID starting with 8b04ad9506c7612475fe7fd8d48770d5ae05c142e3a9237a3cf1f9b4f893438d not found: ID does not exist" containerID="8b04ad9506c7612475fe7fd8d48770d5ae05c142e3a9237a3cf1f9b4f893438d"
Nov 03 02:46:15.639290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.639252    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b04ad9506c7612475fe7fd8d48770d5ae05c142e3a9237a3cf1f9b4f893438d"} err="failed to get container status \"8b04ad9506c7612475fe7fd8d48770d5ae05c142e3a9237a3cf1f9b4f893438d\": rpc error: code = NotFound desc = could not find container \"8b04ad9506c7612475fe7fd8d48770d5ae05c142e3a9237a3cf1f9b4f893438d\": container with ID starting with 8b04ad9506c7612475fe7fd8d48770d5ae05c142e3a9237a3cf1f9b4f893438d not found: ID does not exist"
Nov 03 02:46:15.672807 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.672764    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqzp9\" (UniqueName: \"kubernetes.io/projected/e6502ec5-52cb-4824-894e-d77e374f67a4-kube-api-access-vqzp9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:15.763758 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.763722    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"]
Nov 03 02:46:15.768124 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.768095    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f0391a-54bfdbf45f-qcpms"]
Nov 03 02:46:15.871383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.871280    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"]
Nov 03 02:46:15.874185 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:15.874156    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f0391a-674f854fbc-8qzf8"]
Nov 03 02:46:16.622239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:16.622204    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:46:16.690822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:16.690790    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b" path="/var/lib/kubelet/pods/4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b/volumes"
Nov 03 02:46:16.691164 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:16.691152    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6502ec5-52cb-4824-894e-d77e374f67a4" path="/var/lib/kubelet/pods/e6502ec5-52cb-4824-894e-d77e374f67a4/volumes"
Nov 03 02:46:30.257486 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:30.257446    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"]
Nov 03 02:46:30.258094 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:30.257697    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2" podUID="d9b767dd-0d0d-4736-931c-29debe051aaf" containerName="istio-proxy" containerID="cri-o://a66cecf7c1fcdd2541a17f766f60f08e86e29b9bf66135485d57f91989ebe2f4" gracePeriod=30
Nov 03 02:46:31.399455 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:31.399413    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:46:35.509088 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.509060    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:46:35.574886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.574795    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-kube-api-access-2kmvk\" (UniqueName: \"kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-kube-api-access-2kmvk\") pod \"d9b767dd-0d0d-4736-931c-29debe051aaf\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") "
Nov 03 02:46:35.574886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.574839    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b767dd-0d0d-4736-931c-29debe051aaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b767dd-0d0d-4736-931c-29debe051aaf-istiod-ca-cert\") pod \"d9b767dd-0d0d-4736-931c-29debe051aaf\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") "
Nov 03 02:46:35.574886 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.574871    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-data\") pod \"d9b767dd-0d0d-4736-931c-29debe051aaf\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") "
Nov 03 02:46:35.575193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.574989    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-envoy\") pod \"d9b767dd-0d0d-4736-931c-29debe051aaf\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") "
Nov 03 02:46:35.575193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.575044    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-token\" (UniqueName: \"kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-token\") pod \"d9b767dd-0d0d-4736-931c-29debe051aaf\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") "
Nov 03 02:46:35.575193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.575120    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-socket\") pod \"d9b767dd-0d0d-4736-931c-29debe051aaf\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") "
Nov 03 02:46:35.575193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.575138    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-data" pod "d9b767dd-0d0d-4736-931c-29debe051aaf" (UID: "d9b767dd-0d0d-4736-931c-29debe051aaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:35.575193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.575154    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-podinfo\") pod \"d9b767dd-0d0d-4736-931c-29debe051aaf\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") "
Nov 03 02:46:35.575502 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.575210    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-credential-socket\") pod \"d9b767dd-0d0d-4736-931c-29debe051aaf\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") "
Nov 03 02:46:35.575502 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.575242    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-certs\") pod \"d9b767dd-0d0d-4736-931c-29debe051aaf\" (UID: \"d9b767dd-0d0d-4736-931c-29debe051aaf\") "
Nov 03 02:46:35.575502 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.575374    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-socket" pod "d9b767dd-0d0d-4736-931c-29debe051aaf" (UID: "d9b767dd-0d0d-4736-931c-29debe051aaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:35.575685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.575623    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-certs" pod "d9b767dd-0d0d-4736-931c-29debe051aaf" (UID: "d9b767dd-0d0d-4736-931c-29debe051aaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:35.575752 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.575714    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:35.575752 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.575731    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:35.575752 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.575748    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:35.575925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.575754    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-credential-socket" pod "d9b767dd-0d0d-4736-931c-29debe051aaf" (UID: "d9b767dd-0d0d-4736-931c-29debe051aaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:35.575986 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.575971    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b767dd-0d0d-4736-931c-29debe051aaf-istiod-ca-cert" pod "d9b767dd-0d0d-4736-931c-29debe051aaf" (UID: "d9b767dd-0d0d-4736-931c-29debe051aaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:46:35.577618 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.577588    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-token" pod "d9b767dd-0d0d-4736-931c-29debe051aaf" (UID: "d9b767dd-0d0d-4736-931c-29debe051aaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:46:35.577746 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.577616    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-podinfo" pod "d9b767dd-0d0d-4736-931c-29debe051aaf" (UID: "d9b767dd-0d0d-4736-931c-29debe051aaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:46:35.577864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.577843    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-envoy" pod "d9b767dd-0d0d-4736-931c-29debe051aaf" (UID: "d9b767dd-0d0d-4736-931c-29debe051aaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:35.577928 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.577880    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-kube-api-access-2kmvk" pod "d9b767dd-0d0d-4736-931c-29debe051aaf" (UID: "d9b767dd-0d0d-4736-931c-29debe051aaf"). InnerVolumeSpecName "kube-api-access-2kmvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:46:35.677121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.677018    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:35.677121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.677053    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:35.677121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.677065    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:35.677121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.677079    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kmvk\" (UniqueName: \"kubernetes.io/projected/d9b767dd-0d0d-4736-931c-29debe051aaf-kube-api-access-2kmvk\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:35.677121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.677091    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9b767dd-0d0d-4736-931c-29debe051aaf-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:35.677121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.677100    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9b767dd-0d0d-4736-931c-29debe051aaf-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:35.736992 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.736956    5508 generic.go:356] "Generic (PLEG): container finished" podID="d9b767dd-0d0d-4736-931c-29debe051aaf" containerID="a66cecf7c1fcdd2541a17f766f60f08e86e29b9bf66135485d57f91989ebe2f4" exitCode=0
Nov 03 02:46:35.737182 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.737002    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"
Nov 03 02:46:35.737182 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.737034    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2" event={"ID":"d9b767dd-0d0d-4736-931c-29debe051aaf","Type":"ContainerDied","Data":"a66cecf7c1fcdd2541a17f766f60f08e86e29b9bf66135485d57f91989ebe2f4"}
Nov 03 02:46:35.737182 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.737088    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2" event={"ID":"d9b767dd-0d0d-4736-931c-29debe051aaf","Type":"ContainerDied","Data":"310fb2e07a551c327659edbce07cbc1f0b3f4a6c182df41c12869ffa9f4b7bbe"}
Nov 03 02:46:35.737182 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.737109    5508 scope.go:122] "RemoveContainer" containerID="a66cecf7c1fcdd2541a17f766f60f08e86e29b9bf66135485d57f91989ebe2f4"
Nov 03 02:46:35.747159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.747142    5508 scope.go:122] "RemoveContainer" containerID="a66cecf7c1fcdd2541a17f766f60f08e86e29b9bf66135485d57f91989ebe2f4"
Nov 03 02:46:35.747477 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:46:35.747458    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66cecf7c1fcdd2541a17f766f60f08e86e29b9bf66135485d57f91989ebe2f4\": container with ID starting with a66cecf7c1fcdd2541a17f766f60f08e86e29b9bf66135485d57f91989ebe2f4 not found: ID does not exist" containerID="a66cecf7c1fcdd2541a17f766f60f08e86e29b9bf66135485d57f91989ebe2f4"
Nov 03 02:46:35.747566 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.747484    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66cecf7c1fcdd2541a17f766f60f08e86e29b9bf66135485d57f91989ebe2f4"} err="failed to get container status \"a66cecf7c1fcdd2541a17f766f60f08e86e29b9bf66135485d57f91989ebe2f4\": rpc error: code = NotFound desc = could not find container \"a66cecf7c1fcdd2541a17f766f60f08e86e29b9bf66135485d57f91989ebe2f4\": container with ID starting with a66cecf7c1fcdd2541a17f766f60f08e86e29b9bf66135485d57f91989ebe2f4 not found: ID does not exist"
Nov 03 02:46:35.893052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.892959    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"]
Nov 03 02:46:35.897656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:35.897622    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-5t2x2"]
Nov 03 02:46:36.691339 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:36.691303    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b767dd-0d0d-4736-931c-29debe051aaf" path="/var/lib/kubelet/pods/d9b767dd-0d0d-4736-931c-29debe051aaf/volumes"
Nov 03 02:46:43.811470 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:46:43.811435    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:46:43.970177 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:43.970144    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"]
Nov 03 02:46:43.970413 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:43.970391    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl" podUID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerName="istio-proxy" containerID="cri-o://123c591325d9bbc960f98671ab7afc24dd51af539b3e835c7109d0efaabdea4f" gracePeriod=30
Nov 03 02:46:45.214976 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:45.214941    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b6b9b56d5-vpvgj_43b5424f-45e9-49e4-975c-bc9c0dcda261/authorino/0.log"
Nov 03 02:46:46.833925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:46.833891    5508 scope.go:122] "RemoveContainer" containerID="8b51aa8791277bdaa0c811abcfbb2f545ce099bebb85833f33612de6826fef20"
Nov 03 02:46:46.842861 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:46.842842    5508 scope.go:122] "RemoveContainer" containerID="addbfdbbe253cece2819ec244287542f844396881ce7e169d021dedd04e8f3eb"
Nov 03 02:46:46.851170 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:46.851148    5508 scope.go:122] "RemoveContainer" containerID="5032afd5cc12934bf8e54c5c73cd25b0470de117fa2ff433e23f010c706627a4"
Nov 03 02:46:47.648456 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:47.648414    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-jr6bp_6a99ffc0-6cc1-4188-8878-82a0dce662cc/manager/0.log"
Nov 03 02:46:47.891607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:47.891577    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4pqmh_6904580a-c37d-4058-9f86-dc69f52efdf6/limitador/0.log"
Nov 03 02:46:49.225537 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.225510    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 02:46:49.415180 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415143    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-socket\") pod \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") "
Nov 03 02:46:49.415403 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415231    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-credential-socket\") pod \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") "
Nov 03 02:46:49.415403 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415263    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istiod-ca-cert\") pod \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") "
Nov 03 02:46:49.415403 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415297    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-token\" (UniqueName: \"kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-token\") pod \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") "
Nov 03 02:46:49.415403 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415330    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-data\") pod \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") "
Nov 03 02:46:49.415665 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415434    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-envoy\") pod \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") "
Nov 03 02:46:49.415665 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415453    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-socket" pod "8e99b241-cf23-4d04-9087-87e5b89aaf9b" (UID: "8e99b241-cf23-4d04-9087-87e5b89aaf9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:49.415665 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415504    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-certs\") pod \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") "
Nov 03 02:46:49.415665 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415534    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-kube-api-access-tfkzm\" (UniqueName: \"kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-kube-api-access-tfkzm\") pod \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") "
Nov 03 02:46:49.415665 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415575    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-podinfo\") pod \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\" (UID: \"8e99b241-cf23-4d04-9087-87e5b89aaf9b\") "
Nov 03 02:46:49.415665 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415576    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-credential-socket" pod "8e99b241-cf23-4d04-9087-87e5b89aaf9b" (UID: "8e99b241-cf23-4d04-9087-87e5b89aaf9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:49.416029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415800    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istiod-ca-cert" pod "8e99b241-cf23-4d04-9087-87e5b89aaf9b" (UID: "8e99b241-cf23-4d04-9087-87e5b89aaf9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:46:49.416029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415832    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-certs" pod "8e99b241-cf23-4d04-9087-87e5b89aaf9b" (UID: "8e99b241-cf23-4d04-9087-87e5b89aaf9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:49.416029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415899    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:49.416029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415918    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:49.416029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415932    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:49.416029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.415946    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:49.416367 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.416033    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-data" pod "8e99b241-cf23-4d04-9087-87e5b89aaf9b" (UID: "8e99b241-cf23-4d04-9087-87e5b89aaf9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:49.418085 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.417884    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-envoy" pod "8e99b241-cf23-4d04-9087-87e5b89aaf9b" (UID: "8e99b241-cf23-4d04-9087-87e5b89aaf9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:46:49.418233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.417956    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-podinfo" pod "8e99b241-cf23-4d04-9087-87e5b89aaf9b" (UID: "8e99b241-cf23-4d04-9087-87e5b89aaf9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:46:49.418233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.418050    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-token" pod "8e99b241-cf23-4d04-9087-87e5b89aaf9b" (UID: "8e99b241-cf23-4d04-9087-87e5b89aaf9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:46:49.418499 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.418478    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-kube-api-access-tfkzm" pod "8e99b241-cf23-4d04-9087-87e5b89aaf9b" (UID: "8e99b241-cf23-4d04-9087-87e5b89aaf9b"). InnerVolumeSpecName "kube-api-access-tfkzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:46:49.516417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.516384    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfkzm\" (UniqueName: \"kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-kube-api-access-tfkzm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:49.516417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.516410    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:49.516417 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.516422    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:49.516685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.516430    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:49.516685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.516439    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e99b241-cf23-4d04-9087-87e5b89aaf9b-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:46:49.817913 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.817809    5508 generic.go:356] "Generic (PLEG): container finished" podID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerID="123c591325d9bbc960f98671ab7afc24dd51af539b3e835c7109d0efaabdea4f" exitCode=0
Nov 03 02:46:49.817913 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.817857    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"
Nov 03 02:46:49.817913 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.817895    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl" event={"ID":"8e99b241-cf23-4d04-9087-87e5b89aaf9b","Type":"ContainerDied","Data":"123c591325d9bbc960f98671ab7afc24dd51af539b3e835c7109d0efaabdea4f"}
Nov 03 02:46:49.818183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.817955    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl" event={"ID":"8e99b241-cf23-4d04-9087-87e5b89aaf9b","Type":"ContainerDied","Data":"bf6c1c895ef16429a876d7c4efcca3275784aa1555cfb579fffb9ef500441473"}
Nov 03 02:46:49.818183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.817987    5508 scope.go:122] "RemoveContainer" containerID="123c591325d9bbc960f98671ab7afc24dd51af539b3e835c7109d0efaabdea4f"
Nov 03 02:46:49.829566 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.829544    5508 scope.go:122] "RemoveContainer" containerID="8a5d0b4639a3103d1d71a767196b77d62f4c9de25693cafbf543dd0a1a8414cd"
Nov 03 02:46:49.838957 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.838933    5508 scope.go:122] "RemoveContainer" containerID="123c591325d9bbc960f98671ab7afc24dd51af539b3e835c7109d0efaabdea4f"
Nov 03 02:46:49.839296 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:46:49.839275    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"123c591325d9bbc960f98671ab7afc24dd51af539b3e835c7109d0efaabdea4f\": container with ID starting with 123c591325d9bbc960f98671ab7afc24dd51af539b3e835c7109d0efaabdea4f not found: ID does not exist" containerID="123c591325d9bbc960f98671ab7afc24dd51af539b3e835c7109d0efaabdea4f"
Nov 03 02:46:49.839400 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.839308    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"123c591325d9bbc960f98671ab7afc24dd51af539b3e835c7109d0efaabdea4f"} err="failed to get container status \"123c591325d9bbc960f98671ab7afc24dd51af539b3e835c7109d0efaabdea4f\": rpc error: code = NotFound desc = could not find container \"123c591325d9bbc960f98671ab7afc24dd51af539b3e835c7109d0efaabdea4f\": container with ID starting with 123c591325d9bbc960f98671ab7afc24dd51af539b3e835c7109d0efaabdea4f not found: ID does not exist"
Nov 03 02:46:49.839400 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.839331    5508 scope.go:122] "RemoveContainer" containerID="8a5d0b4639a3103d1d71a767196b77d62f4c9de25693cafbf543dd0a1a8414cd"
Nov 03 02:46:49.839623 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:46:49.839598    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5d0b4639a3103d1d71a767196b77d62f4c9de25693cafbf543dd0a1a8414cd\": container with ID starting with 8a5d0b4639a3103d1d71a767196b77d62f4c9de25693cafbf543dd0a1a8414cd not found: ID does not exist" containerID="8a5d0b4639a3103d1d71a767196b77d62f4c9de25693cafbf543dd0a1a8414cd"
Nov 03 02:46:49.839666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:49.839634    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5d0b4639a3103d1d71a767196b77d62f4c9de25693cafbf543dd0a1a8414cd"} err="failed to get container status \"8a5d0b4639a3103d1d71a767196b77d62f4c9de25693cafbf543dd0a1a8414cd\": rpc error: code = NotFound desc = could not find container \"8a5d0b4639a3103d1d71a767196b77d62f4c9de25693cafbf543dd0a1a8414cd\": container with ID starting with 8a5d0b4639a3103d1d71a767196b77d62f4c9de25693cafbf543dd0a1a8414cd not found: ID does not exist"
Nov 03 02:46:50.019747 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:50.019712    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"]
Nov 03 02:46:50.066184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:50.066155    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b6fb9b-openshift-default-cfbb9984c-wwndl"]
Nov 03 02:46:50.689887 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:50.689851    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" path="/var/lib/kubelet/pods/8e99b241-cf23-4d04-9087-87e5b89aaf9b/volumes"
Nov 03 02:46:52.064889 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:52.064852    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vzvsm_2bfd7d91-c916-4aa1-882f-a068f42ccfc8/proxy/0.log"
Nov 03 02:46:58.170416 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170378    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"]
Nov 03 02:46:58.170833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170728    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d95f2e42-accd-4be5-bcdb-aec368ad11a0" containerName="revoke-keys"
Nov 03 02:46:58.170833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170739    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95f2e42-accd-4be5-bcdb-aec368ad11a0" containerName="revoke-keys"
Nov 03 02:46:58.170833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170758    5508 cpu_manager.go:414] "containerMap: removing container" podUID="d9b767dd-0d0d-4736-931c-29debe051aaf" containerName="istio-proxy"
Nov 03 02:46:58.170833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170764    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b767dd-0d0d-4736-931c-29debe051aaf" containerName="istio-proxy"
Nov 03 02:46:58.170833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170771    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerName="istio-proxy"
Nov 03 02:46:58.170833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170776    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerName="istio-proxy"
Nov 03 02:46:58.170833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170784    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b" containerName="payload-processing"
Nov 03 02:46:58.170833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170789    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b" containerName="payload-processing"
Nov 03 02:46:58.170833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170797    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerName="istio-proxy"
Nov 03 02:46:58.170833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170802    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerName="istio-proxy"
Nov 03 02:46:58.170833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170818    5508 cpu_manager.go:414] "containerMap: removing container" podUID="e6502ec5-52cb-4824-894e-d77e374f67a4" containerName="payload-pre-processing"
Nov 03 02:46:58.170833 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170822    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6502ec5-52cb-4824-894e-d77e374f67a4" containerName="payload-pre-processing"
Nov 03 02:46:58.171211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170871    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerName="istio-proxy"
Nov 03 02:46:58.171211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170880    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6502ec5-52cb-4824-894e-d77e374f67a4" containerName="payload-pre-processing"
Nov 03 02:46:58.171211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170889    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95f2e42-accd-4be5-bcdb-aec368ad11a0" containerName="revoke-keys"
Nov 03 02:46:58.171211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170897    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b767dd-0d0d-4736-931c-29debe051aaf" containerName="istio-proxy"
Nov 03 02:46:58.171211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170903    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e99b241-cf23-4d04-9087-87e5b89aaf9b" containerName="istio-proxy"
Nov 03 02:46:58.171211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.170909    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8701d1-ea9a-4aa3-87c3-095d8eb6ff2b" containerName="payload-processing"
Nov 03 02:46:58.174074 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.174054    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.193402 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.193339    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-certs\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.193557 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.193428    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-data\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.193557 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.193464    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxlb4\" (UniqueName: \"kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-kube-api-access-mxlb4\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.193632 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.193569    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istiod-ca-cert\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.193632 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.193616    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-token\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.193704 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.193658    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-socket\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.193741 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.193722    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-credential-socket\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.193776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.193747    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-envoy\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.193808 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.193776    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-podinfo\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.223133 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.223097    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-82d1bf6f-openshift-default-dockercfg-s7b2g\""
Nov 03 02:46:58.274868 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.274834    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"]
Nov 03 02:46:58.294678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.294643    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-token\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.294678 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.294680    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-socket\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.294909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.294703    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-credential-socket\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.294909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.294820    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-envoy\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.294909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.294902    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-podinfo\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.295060 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.294977    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-certs\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.295060 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.295028    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-data\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.295181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.295074    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlb4\" (UniqueName: \"kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-kube-api-access-mxlb4\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.295181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.295081    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-credential-socket\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.295181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.295135    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-socket\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.295181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.295169    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istiod-ca-cert\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.295425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.295283    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-certs\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.295497 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.295447    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-data\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.295724 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.295701    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istiod-ca-cert\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.297008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.296987    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-envoy\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.297744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.297723    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-podinfo\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.352007 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.351969    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-token\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.352175 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.352153    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxlb4\" (UniqueName: \"kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-kube-api-access-mxlb4\") pod \"e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") " pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.486548 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.486449    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:46:58.921204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.921166    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"]
Nov 03 02:46:58.928741 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:46:58.928711    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca36783e_c015_4c0c_9a67_c39c9e1f1207.slice/crio-2f5fa3073e30be4e09687b093f8c56ee3ae5e03bd875f3070568007d5cc9836c WatchSource:0}: Error finding container 2f5fa3073e30be4e09687b093f8c56ee3ae5e03bd875f3070568007d5cc9836c: Status 404 returned error can't find the container with id 2f5fa3073e30be4e09687b093f8c56ee3ae5e03bd875f3070568007d5cc9836c
Nov 03 02:46:58.931081 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.931044    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:46:58.931165 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.931117    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:46:58.931165 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:58.931150    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:46:59.872686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:59.872641    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" event={"ID":"ca36783e-c015-4c0c-9a67-c39c9e1f1207","Type":"ContainerStarted","Data":"fb71b4c57d46d6c4a8fd3c1e69a6816943f1b51c6aff437eba2039515abb95a0"}
Nov 03 02:46:59.873095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:46:59.872716    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" event={"ID":"ca36783e-c015-4c0c-9a67-c39c9e1f1207","Type":"ContainerStarted","Data":"2f5fa3073e30be4e09687b093f8c56ee3ae5e03bd875f3070568007d5cc9836c"}
Nov 03 02:47:00.033289 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:00.033222    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" podStartSLOduration=2.033208045 podStartE2EDuration="2.033208045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:47:00.031056523 +0000 UTC m=+5299.456663005" watchObservedRunningTime="2026-11-03 02:47:00.033208045 +0000 UTC m=+5299.458814575"
Nov 03 02:47:00.486883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:00.486847    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:47:00.492009 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:00.491977    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:47:00.878479 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:00.878403    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:47:00.879542 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:00.879518    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:47:05.600478 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.600444    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"]
Nov 03 02:47:05.604114 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.604093    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"
Nov 03 02:47:05.664442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.664404    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkp7\" (UniqueName: \"kubernetes.io/projected/8699ae06-aa54-471c-86a7-a5542d3ead70-kube-api-access-xzkp7\") pod \"payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5\" (UID: \"8699ae06-aa54-471c-86a7-a5542d3ead70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"
Nov 03 02:47:05.664656 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.664455    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8699ae06-aa54-471c-86a7-a5542d3ead70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5\" (UID: \"8699ae06-aa54-471c-86a7-a5542d3ead70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"
Nov 03 02:47:05.699093 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.699062    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-82d1bf6f-dockercfg-9nt7x\""
Nov 03 02:47:05.699093 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.699061    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-82d1bf6f\""
Nov 03 02:47:05.701429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.701406    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"]
Nov 03 02:47:05.762897 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.762860    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"]
Nov 03 02:47:05.765943 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.765909    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkp7\" (UniqueName: \"kubernetes.io/projected/8699ae06-aa54-471c-86a7-a5542d3ead70-kube-api-access-xzkp7\") pod \"payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5\" (UID: \"8699ae06-aa54-471c-86a7-a5542d3ead70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"
Nov 03 02:47:05.766119 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.765953    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8699ae06-aa54-471c-86a7-a5542d3ead70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5\" (UID: \"8699ae06-aa54-471c-86a7-a5542d3ead70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"
Nov 03 02:47:05.766559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.766541    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"
Nov 03 02:47:05.766801 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.766781    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8699ae06-aa54-471c-86a7-a5542d3ead70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5\" (UID: \"8699ae06-aa54-471c-86a7-a5542d3ead70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"
Nov 03 02:47:05.823555 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.823524    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkp7\" (UniqueName: \"kubernetes.io/projected/8699ae06-aa54-471c-86a7-a5542d3ead70-kube-api-access-xzkp7\") pod \"payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5\" (UID: \"8699ae06-aa54-471c-86a7-a5542d3ead70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"
Nov 03 02:47:05.824713 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.824688    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"]
Nov 03 02:47:05.867459 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.867334    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-plugins-config-volume\") pod \"payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj\" (UID: \"0ac0e3fc-4b11-4113-b66f-02320ac59c2c\") " pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"
Nov 03 02:47:05.867640 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.867504    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8jdn\" (UniqueName: \"kubernetes.io/projected/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-kube-api-access-d8jdn\") pod \"payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj\" (UID: \"0ac0e3fc-4b11-4113-b66f-02320ac59c2c\") " pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"
Nov 03 02:47:05.917834 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.917800    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"
Nov 03 02:47:05.968635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.968608    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-plugins-config-volume\") pod \"payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj\" (UID: \"0ac0e3fc-4b11-4113-b66f-02320ac59c2c\") " pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"
Nov 03 02:47:05.968780 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.968674    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8jdn\" (UniqueName: \"kubernetes.io/projected/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-kube-api-access-d8jdn\") pod \"payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj\" (UID: \"0ac0e3fc-4b11-4113-b66f-02320ac59c2c\") " pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"
Nov 03 02:47:05.969728 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:05.969667    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-plugins-config-volume\") pod \"payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj\" (UID: \"0ac0e3fc-4b11-4113-b66f-02320ac59c2c\") " pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"
Nov 03 02:47:06.026503 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:06.026469    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8jdn\" (UniqueName: \"kubernetes.io/projected/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-kube-api-access-d8jdn\") pod \"payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj\" (UID: \"0ac0e3fc-4b11-4113-b66f-02320ac59c2c\") " pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"
Nov 03 02:47:06.079047 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:06.079014    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"
Nov 03 02:47:06.290124 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:06.290087    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"]
Nov 03 02:47:06.297376 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:47:06.297332    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8699ae06_aa54_471c_86a7_a5542d3ead70.slice/crio-bf0317afbbaef4aaef964cb919bb7116407d90c2ade46bd53a758051d7d13797 WatchSource:0}: Error finding container bf0317afbbaef4aaef964cb919bb7116407d90c2ade46bd53a758051d7d13797: Status 404 returned error can't find the container with id bf0317afbbaef4aaef964cb919bb7116407d90c2ade46bd53a758051d7d13797
Nov 03 02:47:06.450124 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:06.450088    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"]
Nov 03 02:47:06.458534 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:47:06.458509    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac0e3fc_4b11_4113_b66f_02320ac59c2c.slice/crio-e80d1a356e1ba987960ccc050419d4498ac8b93333fec371dcd27df97dc908c5 WatchSource:0}: Error finding container e80d1a356e1ba987960ccc050419d4498ac8b93333fec371dcd27df97dc908c5: Status 404 returned error can't find the container with id e80d1a356e1ba987960ccc050419d4498ac8b93333fec371dcd27df97dc908c5
Nov 03 02:47:06.912824 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:06.912784    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5" event={"ID":"8699ae06-aa54-471c-86a7-a5542d3ead70","Type":"ContainerStarted","Data":"7c8da7e3429e168f0f87664918308b97f5d80a88772a15f7f8791cd49b6b0db2"}
Nov 03 02:47:06.913276 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:06.912856    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"
Nov 03 02:47:06.913276 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:06.912874    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5" event={"ID":"8699ae06-aa54-471c-86a7-a5542d3ead70","Type":"ContainerStarted","Data":"bf0317afbbaef4aaef964cb919bb7116407d90c2ade46bd53a758051d7d13797"}
Nov 03 02:47:06.914474 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:06.914440    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj" event={"ID":"0ac0e3fc-4b11-4113-b66f-02320ac59c2c","Type":"ContainerStarted","Data":"e9a3e1002ed6ae1e39015fbe25f3b64899ec5936999f0c60e99dd3ca9d0be796"}
Nov 03 02:47:06.914606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:06.914494    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj" event={"ID":"0ac0e3fc-4b11-4113-b66f-02320ac59c2c","Type":"ContainerStarted","Data":"e80d1a356e1ba987960ccc050419d4498ac8b93333fec371dcd27df97dc908c5"}
Nov 03 02:47:06.914606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:06.914528    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"
Nov 03 02:47:07.023287 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:07.023234    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5" podStartSLOduration=2.023221943 podStartE2EDuration="2.023221943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:47:07.021626561 +0000 UTC m=+5306.447233024" watchObservedRunningTime="2026-11-03 02:47:07.023221943 +0000 UTC m=+5306.448828407"
Nov 03 02:47:07.130744 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:07.130645    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj" podStartSLOduration=2.130626675 podStartE2EDuration="2.130626675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:47:07.130080426 +0000 UTC m=+5306.555686889" watchObservedRunningTime="2026-11-03 02:47:07.130626675 +0000 UTC m=+5306.556233140"
Nov 03 02:47:08.177247 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.177208    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"]
Nov 03 02:47:08.180974 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.180949    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:08.276267 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.276204    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-82d1bf6f-serving-cert\""
Nov 03 02:47:08.276267 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.276236    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sxqdt\""
Nov 03 02:47:08.276267 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.276236    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 02:47:08.276267 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.276241    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:47:08.276687 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.276383    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:47:08.279043 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.279016    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"]
Nov 03 02:47:08.290831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.290800    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-maas-api-tls\") pod \"maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:08.290958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.290847    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-metrics-certs\") pod \"maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:08.290958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.290870    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c150264b-bd92-475f-8ac8-5591b848bc49-openshift-service-ca\") pod \"maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:08.291043 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.290991    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn6v9\" (UniqueName: \"kubernetes.io/projected/c150264b-bd92-475f-8ac8-5591b848bc49-kube-api-access-vn6v9\") pod \"maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:08.391576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.391538    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-maas-api-tls\") pod \"maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:08.391768 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.391590    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-metrics-certs\") pod \"maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:08.391768 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:47:08.391681    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-serving-cert: secret "maas-api-e2e-ait-82d1bf6f-serving-cert" not found
Nov 03 02:47:08.391768 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.391693    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c150264b-bd92-475f-8ac8-5591b848bc49-openshift-service-ca\") pod \"maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:08.391768 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:47:08.391736    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-maas-api-tls podName:c150264b-bd92-475f-8ac8-5591b848bc49 nodeName:}" failed. No retries permitted until 2026-11-03 02:47:08.891720569 +0000 UTC m=+5308.317327047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-maas-api-tls") pod "maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt" (UID: "c150264b-bd92-475f-8ac8-5591b848bc49") : secret "maas-api-e2e-ait-82d1bf6f-serving-cert" not found
Nov 03 02:47:08.391943 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.391784    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn6v9\" (UniqueName: \"kubernetes.io/projected/c150264b-bd92-475f-8ac8-5591b848bc49-kube-api-access-vn6v9\") pod \"maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:08.392379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.392342    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c150264b-bd92-475f-8ac8-5591b848bc49-openshift-service-ca\") pod \"maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:08.394055 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.394029    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-metrics-certs\") pod \"maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:08.449429 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.449328    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn6v9\" (UniqueName: \"kubernetes.io/projected/c150264b-bd92-475f-8ac8-5591b848bc49-kube-api-access-vn6v9\") pod \"maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:08.896827 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.896800    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-maas-api-tls\") pod \"maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:08.899285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:08.899252    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-maas-api-tls\") pod \"maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:09.094759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:09.094721    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:09.467037 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:09.466952    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"]
Nov 03 02:47:09.473383 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:47:09.473325    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc150264b_bd92_475f_8ac8_5591b848bc49.slice/crio-4c20b4273dfd2f85dc6108aefce3b0d2e27d0487b14a125af5292aab7a9725f4 WatchSource:0}: Error finding container 4c20b4273dfd2f85dc6108aefce3b0d2e27d0487b14a125af5292aab7a9725f4: Status 404 returned error can't find the container with id 4c20b4273dfd2f85dc6108aefce3b0d2e27d0487b14a125af5292aab7a9725f4
Nov 03 02:47:09.474873 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:09.474855    5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 02:47:09.931787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:09.931751    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt" event={"ID":"c150264b-bd92-475f-8ac8-5591b848bc49","Type":"ContainerStarted","Data":"4c20b4273dfd2f85dc6108aefce3b0d2e27d0487b14a125af5292aab7a9725f4"}
Nov 03 02:47:10.939787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:10.939745    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt" event={"ID":"c150264b-bd92-475f-8ac8-5591b848bc49","Type":"ContainerStarted","Data":"a1f3fe4f30f9ad9de7c216bf37390718eb2462c17e784ea2aa752fe1d3f3cac1"}
Nov 03 02:47:10.940296 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:10.939811    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:11.052237 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:11.052183    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt" podStartSLOduration=1.946246023 podStartE2EDuration="3.052167893s" totalImagesPullingTime="1.10592187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:47:11.050252234 +0000 UTC m=+5310.475858698" watchObservedRunningTime="2026-11-03 02:47:11.052167893 +0000 UTC m=+5310.477774356"
Nov 03 02:47:13.811306 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:47:13.811270    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:47:16.184231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:16.184131    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:47:16.980140 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:16.980087    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:17.923671 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:17.923643    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"
Nov 03 02:47:17.924152 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:17.923717    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"
Nov 03 02:47:25.031911 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:25.031879    5508 generic.go:356] "Generic (PLEG): container finished" podID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerID="fb71b4c57d46d6c4a8fd3c1e69a6816943f1b51c6aff437eba2039515abb95a0" exitCode=0
Nov 03 02:47:25.032301 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:25.031939    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" event={"ID":"ca36783e-c015-4c0c-9a67-c39c9e1f1207","Type":"ContainerDied","Data":"fb71b4c57d46d6c4a8fd3c1e69a6816943f1b51c6aff437eba2039515abb95a0"}
Nov 03 02:47:25.032371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:25.032302    5508 scope.go:122] "RemoveContainer" containerID="fb71b4c57d46d6c4a8fd3c1e69a6816943f1b51c6aff437eba2039515abb95a0"
Nov 03 02:47:25.033464 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:25.033436    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:47:25.033539 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:25.033527    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:47:25.033577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:25.033564    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:47:26.039632 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:26.039594    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" event={"ID":"ca36783e-c015-4c0c-9a67-c39c9e1f1207","Type":"ContainerStarted","Data":"07d256c9c7e24350f88f8495ffb32dcbef46bb66cb0421ebfea2acc1a185ab6e"}
Nov 03 02:47:26.486828 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:26.486785    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:47:26.487998 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:26.487976    5508 patch_prober.go:29] interesting pod/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused" start-of-body=
Nov 03 02:47:26.488073 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:26.488024    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused"
Nov 03 02:47:27.045107 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:27.045068    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"]
Nov 03 02:47:27.091207 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:27.091177    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04223fd8-8t2p2"]
Nov 03 02:47:27.486958 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:27.486921    5508 patch_prober.go:29] interesting pod/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused" start-of-body=
Nov 03 02:47:27.487157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:27.486974    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused"
Nov 03 02:47:28.487881 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:28.487840    5508 patch_prober.go:29] interesting pod/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused" start-of-body=
Nov 03 02:47:28.488317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:28.487914    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused"
Nov 03 02:47:28.689843 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:28.689805    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f671fa22-98b0-4dc3-bb0b-fa278afd2991" path="/var/lib/kubelet/pods/f671fa22-98b0-4dc3-bb0b-fa278afd2991/volumes"
Nov 03 02:47:29.487153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:29.487115    5508 patch_prober.go:29] interesting pod/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused" start-of-body=
Nov 03 02:47:29.487392 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:29.487180    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused"
Nov 03 02:47:30.487111 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:30.487072    5508 patch_prober.go:29] interesting pod/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused" start-of-body=
Nov 03 02:47:30.487600 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:30.487128    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.118:15021/healthz/ready\": dial tcp 10.133.1.118:15021: connect: connection refused"
Nov 03 02:47:30.878713 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:30.878663    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:47:31.491250 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:31.491222    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:47:31.492399 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:31.492378    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:47:38.046090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:38.046052    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"]
Nov 03 02:47:38.092363 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:38.092327    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4747842-59tfl"]
Nov 03 02:47:38.691844 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:38.691813    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce2b589-5743-4821-9b2c-0e46087fba80" path="/var/lib/kubelet/pods/9ce2b589-5743-4821-9b2c-0e46087fba80/volumes"
Nov 03 02:47:43.812231 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:47:43.812178    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:47:46.910875 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:46.910838    5508 scope.go:122] "RemoveContainer" containerID="d692eafaa37cd0d37620109274d05f440f86ddb2b5974b9c3318997efe2840d0"
Nov 03 02:47:50.060138 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:50.060097    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"]
Nov 03 02:47:50.064494 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:50.064466    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f02f7cbd-ktplc"]
Nov 03 02:47:50.106205 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:50.106174    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"]
Nov 03 02:47:50.189801 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:50.189763    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"]
Nov 03 02:47:50.691898 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:50.691861    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57696021-91bd-41a6-b396-7cbc018dcf9a" path="/var/lib/kubelet/pods/57696021-91bd-41a6-b396-7cbc018dcf9a/volumes"
Nov 03 02:47:55.107155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.107068    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj" podUID="0ac0e3fc-4b11-4113-b66f-02320ac59c2c" containerName="payload-processing" containerID="cri-o://e9a3e1002ed6ae1e39015fbe25f3b64899ec5936999f0c60e99dd3ca9d0be796" gracePeriod=25
Nov 03 02:47:55.190594 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.190543    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5" podUID="8699ae06-aa54-471c-86a7-a5542d3ead70" containerName="payload-pre-processing" containerID="cri-o://7c8da7e3429e168f0f87664918308b97f5d80a88772a15f7f8791cd49b6b0db2" gracePeriod=25
Nov 03 02:47:55.223276 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.223244    5508 generic.go:356] "Generic (PLEG): container finished" podID="0ac0e3fc-4b11-4113-b66f-02320ac59c2c" containerID="e9a3e1002ed6ae1e39015fbe25f3b64899ec5936999f0c60e99dd3ca9d0be796" exitCode=0
Nov 03 02:47:55.223467 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.223316    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj" event={"ID":"0ac0e3fc-4b11-4113-b66f-02320ac59c2c","Type":"ContainerDied","Data":"e9a3e1002ed6ae1e39015fbe25f3b64899ec5936999f0c60e99dd3ca9d0be796"}
Nov 03 02:47:55.431919 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.431890    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"
Nov 03 02:47:55.459657 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.459629    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-plugins-config-volume\") pod \"0ac0e3fc-4b11-4113-b66f-02320ac59c2c\" (UID: \"0ac0e3fc-4b11-4113-b66f-02320ac59c2c\") "
Nov 03 02:47:55.459815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.459668    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-kube-api-access-d8jdn\" (UniqueName: \"kubernetes.io/projected/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-kube-api-access-d8jdn\") pod \"0ac0e3fc-4b11-4113-b66f-02320ac59c2c\" (UID: \"0ac0e3fc-4b11-4113-b66f-02320ac59c2c\") "
Nov 03 02:47:55.459962 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.459941    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-plugins-config-volume" pod "0ac0e3fc-4b11-4113-b66f-02320ac59c2c" (UID: "0ac0e3fc-4b11-4113-b66f-02320ac59c2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:47:55.460087 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.460070    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:47:55.461736 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.461711    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-kube-api-access-d8jdn" pod "0ac0e3fc-4b11-4113-b66f-02320ac59c2c" (UID: "0ac0e3fc-4b11-4113-b66f-02320ac59c2c"). InnerVolumeSpecName "kube-api-access-d8jdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:47:55.465037 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.465018    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"
Nov 03 02:47:55.560853 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.560819    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8699ae06-aa54-471c-86a7-a5542d3ead70-kube-api-access-xzkp7\" (UniqueName: \"kubernetes.io/projected/8699ae06-aa54-471c-86a7-a5542d3ead70-kube-api-access-xzkp7\") pod \"8699ae06-aa54-471c-86a7-a5542d3ead70\" (UID: \"8699ae06-aa54-471c-86a7-a5542d3ead70\") "
Nov 03 02:47:55.561036 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.560945    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8699ae06-aa54-471c-86a7-a5542d3ead70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8699ae06-aa54-471c-86a7-a5542d3ead70-plugins-config-volume\") pod \"8699ae06-aa54-471c-86a7-a5542d3ead70\" (UID: \"8699ae06-aa54-471c-86a7-a5542d3ead70\") "
Nov 03 02:47:55.561333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.561311    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8jdn\" (UniqueName: \"kubernetes.io/projected/0ac0e3fc-4b11-4113-b66f-02320ac59c2c-kube-api-access-d8jdn\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:47:55.561451 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.561324    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8699ae06-aa54-471c-86a7-a5542d3ead70-plugins-config-volume" pod "8699ae06-aa54-471c-86a7-a5542d3ead70" (UID: "8699ae06-aa54-471c-86a7-a5542d3ead70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:47:55.562936 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.562911    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8699ae06-aa54-471c-86a7-a5542d3ead70-kube-api-access-xzkp7" pod "8699ae06-aa54-471c-86a7-a5542d3ead70" (UID: "8699ae06-aa54-471c-86a7-a5542d3ead70"). InnerVolumeSpecName "kube-api-access-xzkp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:47:55.661844 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.661764    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzkp7\" (UniqueName: \"kubernetes.io/projected/8699ae06-aa54-471c-86a7-a5542d3ead70-kube-api-access-xzkp7\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:47:55.661844 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:55.661791    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8699ae06-aa54-471c-86a7-a5542d3ead70-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:47:56.229339 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.229305    5508 generic.go:356] "Generic (PLEG): container finished" podID="8699ae06-aa54-471c-86a7-a5542d3ead70" containerID="7c8da7e3429e168f0f87664918308b97f5d80a88772a15f7f8791cd49b6b0db2" exitCode=0
Nov 03 02:47:56.229826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.229369    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"
Nov 03 02:47:56.229826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.229396    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5" event={"ID":"8699ae06-aa54-471c-86a7-a5542d3ead70","Type":"ContainerDied","Data":"7c8da7e3429e168f0f87664918308b97f5d80a88772a15f7f8791cd49b6b0db2"}
Nov 03 02:47:56.229826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.229460    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5" event={"ID":"8699ae06-aa54-471c-86a7-a5542d3ead70","Type":"ContainerDied","Data":"bf0317afbbaef4aaef964cb919bb7116407d90c2ade46bd53a758051d7d13797"}
Nov 03 02:47:56.229826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.229495    5508 scope.go:122] "RemoveContainer" containerID="7c8da7e3429e168f0f87664918308b97f5d80a88772a15f7f8791cd49b6b0db2"
Nov 03 02:47:56.231077 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.231058    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj" event={"ID":"0ac0e3fc-4b11-4113-b66f-02320ac59c2c","Type":"ContainerDied","Data":"e80d1a356e1ba987960ccc050419d4498ac8b93333fec371dcd27df97dc908c5"}
Nov 03 02:47:56.231160 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.231066    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"
Nov 03 02:47:56.238695 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.238677    5508 scope.go:122] "RemoveContainer" containerID="7c8da7e3429e168f0f87664918308b97f5d80a88772a15f7f8791cd49b6b0db2"
Nov 03 02:47:56.239131 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:47:56.238985    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8da7e3429e168f0f87664918308b97f5d80a88772a15f7f8791cd49b6b0db2\": container with ID starting with 7c8da7e3429e168f0f87664918308b97f5d80a88772a15f7f8791cd49b6b0db2 not found: ID does not exist" containerID="7c8da7e3429e168f0f87664918308b97f5d80a88772a15f7f8791cd49b6b0db2"
Nov 03 02:47:56.239131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.239026    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8da7e3429e168f0f87664918308b97f5d80a88772a15f7f8791cd49b6b0db2"} err="failed to get container status \"7c8da7e3429e168f0f87664918308b97f5d80a88772a15f7f8791cd49b6b0db2\": rpc error: code = NotFound desc = could not find container \"7c8da7e3429e168f0f87664918308b97f5d80a88772a15f7f8791cd49b6b0db2\": container with ID starting with 7c8da7e3429e168f0f87664918308b97f5d80a88772a15f7f8791cd49b6b0db2 not found: ID does not exist"
Nov 03 02:47:56.239131 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.239049    5508 scope.go:122] "RemoveContainer" containerID="e9a3e1002ed6ae1e39015fbe25f3b64899ec5936999f0c60e99dd3ca9d0be796"
Nov 03 02:47:56.384016 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.383935    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"]
Nov 03 02:47:56.388176 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.388147    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-82d1bf6f-7f6f55fc7-bfxnj"]
Nov 03 02:47:56.494522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.494483    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"]
Nov 03 02:47:56.498448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.498417    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-82d1bf6f-68d98bcd87-xh6q5"]
Nov 03 02:47:56.511308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.511274    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"]
Nov 03 02:47:56.511530 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.511506    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt" podUID="c150264b-bd92-475f-8ac8-5591b848bc49" containerName="maas-api" containerID="cri-o://a1f3fe4f30f9ad9de7c216bf37390718eb2462c17e784ea2aa752fe1d3f3cac1" gracePeriod=30
Nov 03 02:47:56.701819 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.701786    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac0e3fc-4b11-4113-b66f-02320ac59c2c" path="/var/lib/kubelet/pods/0ac0e3fc-4b11-4113-b66f-02320ac59c2c/volumes"
Nov 03 02:47:56.702283 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.702266    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8699ae06-aa54-471c-86a7-a5542d3ead70" path="/var/lib/kubelet/pods/8699ae06-aa54-471c-86a7-a5542d3ead70/volumes"
Nov 03 02:47:56.762998 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.762971    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:56.770944 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.770920    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-maas-api-tls\") pod \"c150264b-bd92-475f-8ac8-5591b848bc49\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") "
Nov 03 02:47:56.771072 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.770961    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c150264b-bd92-475f-8ac8-5591b848bc49-kube-api-access-vn6v9\" (UniqueName: \"kubernetes.io/projected/c150264b-bd92-475f-8ac8-5591b848bc49-kube-api-access-vn6v9\") pod \"c150264b-bd92-475f-8ac8-5591b848bc49\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") "
Nov 03 02:47:56.771147 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.771081    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-metrics-certs\") pod \"c150264b-bd92-475f-8ac8-5591b848bc49\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") "
Nov 03 02:47:56.771207 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.771156    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c150264b-bd92-475f-8ac8-5591b848bc49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c150264b-bd92-475f-8ac8-5591b848bc49-openshift-service-ca\") pod \"c150264b-bd92-475f-8ac8-5591b848bc49\" (UID: \"c150264b-bd92-475f-8ac8-5591b848bc49\") "
Nov 03 02:47:56.771603 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.771576    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c150264b-bd92-475f-8ac8-5591b848bc49-openshift-service-ca" pod "c150264b-bd92-475f-8ac8-5591b848bc49" (UID: "c150264b-bd92-475f-8ac8-5591b848bc49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:47:56.773226 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.773201    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-maas-api-tls" pod "c150264b-bd92-475f-8ac8-5591b848bc49" (UID: "c150264b-bd92-475f-8ac8-5591b848bc49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:47:56.773329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.773243    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-metrics-certs" pod "c150264b-bd92-475f-8ac8-5591b848bc49" (UID: "c150264b-bd92-475f-8ac8-5591b848bc49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:47:56.773329 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.773286    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c150264b-bd92-475f-8ac8-5591b848bc49-kube-api-access-vn6v9" pod "c150264b-bd92-475f-8ac8-5591b848bc49" (UID: "c150264b-bd92-475f-8ac8-5591b848bc49"). InnerVolumeSpecName "kube-api-access-vn6v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:47:56.872910 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.872864    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c150264b-bd92-475f-8ac8-5591b848bc49-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:47:56.872910 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.872903    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:47:56.873184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.872920    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn6v9\" (UniqueName: \"kubernetes.io/projected/c150264b-bd92-475f-8ac8-5591b848bc49-kube-api-access-vn6v9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:47:56.873184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:56.872953    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c150264b-bd92-475f-8ac8-5591b848bc49-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:47:57.240287 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:57.240247    5508 generic.go:356] "Generic (PLEG): container finished" podID="c150264b-bd92-475f-8ac8-5591b848bc49" containerID="a1f3fe4f30f9ad9de7c216bf37390718eb2462c17e784ea2aa752fe1d3f3cac1" exitCode=0
Nov 03 02:47:57.240815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:57.240293    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt" event={"ID":"c150264b-bd92-475f-8ac8-5591b848bc49","Type":"ContainerDied","Data":"a1f3fe4f30f9ad9de7c216bf37390718eb2462c17e784ea2aa752fe1d3f3cac1"}
Nov 03 02:47:57.240815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:57.240298    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"
Nov 03 02:47:57.240815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:57.240360    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt" event={"ID":"c150264b-bd92-475f-8ac8-5591b848bc49","Type":"ContainerDied","Data":"4c20b4273dfd2f85dc6108aefce3b0d2e27d0487b14a125af5292aab7a9725f4"}
Nov 03 02:47:57.240815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:57.240389    5508 scope.go:122] "RemoveContainer" containerID="a1f3fe4f30f9ad9de7c216bf37390718eb2462c17e784ea2aa752fe1d3f3cac1"
Nov 03 02:47:57.250212 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:57.250187    5508 scope.go:122] "RemoveContainer" containerID="a1f3fe4f30f9ad9de7c216bf37390718eb2462c17e784ea2aa752fe1d3f3cac1"
Nov 03 02:47:57.250544 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:47:57.250509    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f3fe4f30f9ad9de7c216bf37390718eb2462c17e784ea2aa752fe1d3f3cac1\": container with ID starting with a1f3fe4f30f9ad9de7c216bf37390718eb2462c17e784ea2aa752fe1d3f3cac1 not found: ID does not exist" containerID="a1f3fe4f30f9ad9de7c216bf37390718eb2462c17e784ea2aa752fe1d3f3cac1"
Nov 03 02:47:57.250599 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:57.250559    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f3fe4f30f9ad9de7c216bf37390718eb2462c17e784ea2aa752fe1d3f3cac1"} err="failed to get container status \"a1f3fe4f30f9ad9de7c216bf37390718eb2462c17e784ea2aa752fe1d3f3cac1\": rpc error: code = NotFound desc = could not find container \"a1f3fe4f30f9ad9de7c216bf37390718eb2462c17e784ea2aa752fe1d3f3cac1\": container with ID starting with a1f3fe4f30f9ad9de7c216bf37390718eb2462c17e784ea2aa752fe1d3f3cac1 not found: ID does not exist"
Nov 03 02:47:57.394970 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:57.394933    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"]
Nov 03 02:47:57.399333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:57.399299    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-82d1bf6f-657d5d9986-zqhnt"]
Nov 03 02:47:58.689750 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:58.689717    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c150264b-bd92-475f-8ac8-5591b848bc49" path="/var/lib/kubelet/pods/c150264b-bd92-475f-8ac8-5591b848bc49/volumes"
Nov 03 02:47:59.956132 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:59.956088    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"]
Nov 03 02:47:59.956590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:47:59.956310    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerName="istio-proxy" containerID="cri-o://07d256c9c7e24350f88f8495ffb32dcbef46bb66cb0421ebfea2acc1a185ab6e" gracePeriod=30
Nov 03 02:48:00.046564 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:00.046528    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"]
Nov 03 02:48:00.049652 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:00.049626    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da6d111d-sdc5s"]
Nov 03 02:48:00.689741 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:00.689706    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d143031-d33e-4e37-815c-f0b6f3a12835" path="/var/lib/kubelet/pods/2d143031-d33e-4e37-815c-f0b6f3a12835/volumes"
Nov 03 02:48:01.494820 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:01.494778    5508 patch_prober.go:29] interesting pod/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 02:48:01.495275 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:01.494839    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 02:48:02.797646 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:02.797610    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:48:05.197694 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.197664    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:48:05.251211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.251114    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-token\" (UniqueName: \"kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-token\") pod \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") "
Nov 03 02:48:05.251211 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.251180    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-kube-api-access-mxlb4\" (UniqueName: \"kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-kube-api-access-mxlb4\") pod \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") "
Nov 03 02:48:05.251487 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.251223    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-socket\") pod \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") "
Nov 03 02:48:05.251487 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.251249    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-envoy\") pod \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") "
Nov 03 02:48:05.251487 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.251394    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-podinfo\") pod \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") "
Nov 03 02:48:05.251487 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.251482    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-certs\") pod \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") "
Nov 03 02:48:05.251727 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.251522    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-credential-socket\") pod \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") "
Nov 03 02:48:05.251727 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.251557    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istiod-ca-cert\") pod \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") "
Nov 03 02:48:05.251727 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.251612    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-data\") pod \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\" (UID: \"ca36783e-c015-4c0c-9a67-c39c9e1f1207\") "
Nov 03 02:48:05.251727 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.251619    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-socket" pod "ca36783e-c015-4c0c-9a67-c39c9e1f1207" (UID: "ca36783e-c015-4c0c-9a67-c39c9e1f1207"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:48:05.251947 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.251868    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-credential-socket" pod "ca36783e-c015-4c0c-9a67-c39c9e1f1207" (UID: "ca36783e-c015-4c0c-9a67-c39c9e1f1207"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:48:05.252008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.251947    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:48:05.252008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.251967    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:48:05.252168 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.252148    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-certs" pod "ca36783e-c015-4c0c-9a67-c39c9e1f1207" (UID: "ca36783e-c015-4c0c-9a67-c39c9e1f1207"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:48:05.252397 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.252369    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istiod-ca-cert" pod "ca36783e-c015-4c0c-9a67-c39c9e1f1207" (UID: "ca36783e-c015-4c0c-9a67-c39c9e1f1207"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:48:05.252493 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.252442    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-data" pod "ca36783e-c015-4c0c-9a67-c39c9e1f1207" (UID: "ca36783e-c015-4c0c-9a67-c39c9e1f1207"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:48:05.253603 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.253577    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-token" pod "ca36783e-c015-4c0c-9a67-c39c9e1f1207" (UID: "ca36783e-c015-4c0c-9a67-c39c9e1f1207"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:48:05.253741 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.253613    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-envoy" pod "ca36783e-c015-4c0c-9a67-c39c9e1f1207" (UID: "ca36783e-c015-4c0c-9a67-c39c9e1f1207"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 02:48:05.253741 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.253714    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-kube-api-access-mxlb4" pod "ca36783e-c015-4c0c-9a67-c39c9e1f1207" (UID: "ca36783e-c015-4c0c-9a67-c39c9e1f1207"). InnerVolumeSpecName "kube-api-access-mxlb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:48:05.253856 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.253771    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-podinfo" pod "ca36783e-c015-4c0c-9a67-c39c9e1f1207" (UID: "ca36783e-c015-4c0c-9a67-c39c9e1f1207"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 02:48:05.290804 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.290764    5508 generic.go:356] "Generic (PLEG): container finished" podID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerID="07d256c9c7e24350f88f8495ffb32dcbef46bb66cb0421ebfea2acc1a185ab6e" exitCode=0
Nov 03 02:48:05.290989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.290819    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"
Nov 03 02:48:05.290989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.290843    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" event={"ID":"ca36783e-c015-4c0c-9a67-c39c9e1f1207","Type":"ContainerDied","Data":"07d256c9c7e24350f88f8495ffb32dcbef46bb66cb0421ebfea2acc1a185ab6e"}
Nov 03 02:48:05.290989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.290900    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm" event={"ID":"ca36783e-c015-4c0c-9a67-c39c9e1f1207","Type":"ContainerDied","Data":"2f5fa3073e30be4e09687b093f8c56ee3ae5e03bd875f3070568007d5cc9836c"}
Nov 03 02:48:05.290989 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.290924    5508 scope.go:122] "RemoveContainer" containerID="07d256c9c7e24350f88f8495ffb32dcbef46bb66cb0421ebfea2acc1a185ab6e"
Nov 03 02:48:05.300745 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.300726    5508 scope.go:122] "RemoveContainer" containerID="fb71b4c57d46d6c4a8fd3c1e69a6816943f1b51c6aff437eba2039515abb95a0"
Nov 03 02:48:05.309701 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.309681    5508 scope.go:122] "RemoveContainer" containerID="07d256c9c7e24350f88f8495ffb32dcbef46bb66cb0421ebfea2acc1a185ab6e"
Nov 03 02:48:05.310006 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:48:05.309984    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d256c9c7e24350f88f8495ffb32dcbef46bb66cb0421ebfea2acc1a185ab6e\": container with ID starting with 07d256c9c7e24350f88f8495ffb32dcbef46bb66cb0421ebfea2acc1a185ab6e not found: ID does not exist" containerID="07d256c9c7e24350f88f8495ffb32dcbef46bb66cb0421ebfea2acc1a185ab6e"
Nov 03 02:48:05.310080 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.310016    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d256c9c7e24350f88f8495ffb32dcbef46bb66cb0421ebfea2acc1a185ab6e"} err="failed to get container status \"07d256c9c7e24350f88f8495ffb32dcbef46bb66cb0421ebfea2acc1a185ab6e\": rpc error: code = NotFound desc = could not find container \"07d256c9c7e24350f88f8495ffb32dcbef46bb66cb0421ebfea2acc1a185ab6e\": container with ID starting with 07d256c9c7e24350f88f8495ffb32dcbef46bb66cb0421ebfea2acc1a185ab6e not found: ID does not exist"
Nov 03 02:48:05.310080 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.310036    5508 scope.go:122] "RemoveContainer" containerID="fb71b4c57d46d6c4a8fd3c1e69a6816943f1b51c6aff437eba2039515abb95a0"
Nov 03 02:48:05.310274 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:48:05.310257    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb71b4c57d46d6c4a8fd3c1e69a6816943f1b51c6aff437eba2039515abb95a0\": container with ID starting with fb71b4c57d46d6c4a8fd3c1e69a6816943f1b51c6aff437eba2039515abb95a0 not found: ID does not exist" containerID="fb71b4c57d46d6c4a8fd3c1e69a6816943f1b51c6aff437eba2039515abb95a0"
Nov 03 02:48:05.310331 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.310284    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb71b4c57d46d6c4a8fd3c1e69a6816943f1b51c6aff437eba2039515abb95a0"} err="failed to get container status \"fb71b4c57d46d6c4a8fd3c1e69a6816943f1b51c6aff437eba2039515abb95a0\": rpc error: code = NotFound desc = could not find container \"fb71b4c57d46d6c4a8fd3c1e69a6816943f1b51c6aff437eba2039515abb95a0\": container with ID starting with fb71b4c57d46d6c4a8fd3c1e69a6816943f1b51c6aff437eba2039515abb95a0 not found: ID does not exist"
Nov 03 02:48:05.353291 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.353247    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:48:05.353291 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.353285    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:48:05.353291 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.353301    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxlb4\" (UniqueName: \"kubernetes.io/projected/ca36783e-c015-4c0c-9a67-c39c9e1f1207-kube-api-access-mxlb4\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:48:05.353576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.353311    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:48:05.353576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.353320    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:48:05.353576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.353329    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca36783e-c015-4c0c-9a67-c39c9e1f1207-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:48:05.353576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.353339    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca36783e-c015-4c0c-9a67-c39c9e1f1207-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:48:05.464762 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.464726    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"]
Nov 03 02:48:05.469045 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:05.469016    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-82d1bf6f-openshift-default-6c65b6dbff-4pgcm"]
Nov 03 02:48:06.690490 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:06.690458    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" path="/var/lib/kubelet/pods/ca36783e-c015-4c0c-9a67-c39c9e1f1207/volumes"
Nov 03 02:48:13.810889 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:48:13.810846    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:48:28.352742 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.352698    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"]
Nov 03 02:48:28.353167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.353037    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerName="istio-proxy"
Nov 03 02:48:28.353167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.353048    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerName="istio-proxy"
Nov 03 02:48:28.353167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.353072    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8699ae06-aa54-471c-86a7-a5542d3ead70" containerName="payload-pre-processing"
Nov 03 02:48:28.353167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.353077    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8699ae06-aa54-471c-86a7-a5542d3ead70" containerName="payload-pre-processing"
Nov 03 02:48:28.353167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.353096    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c150264b-bd92-475f-8ac8-5591b848bc49" containerName="maas-api"
Nov 03 02:48:28.353167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.353101    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c150264b-bd92-475f-8ac8-5591b848bc49" containerName="maas-api"
Nov 03 02:48:28.353167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.353108    5508 cpu_manager.go:414] "containerMap: removing container" podUID="0ac0e3fc-4b11-4113-b66f-02320ac59c2c" containerName="payload-processing"
Nov 03 02:48:28.353167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.353113    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac0e3fc-4b11-4113-b66f-02320ac59c2c" containerName="payload-processing"
Nov 03 02:48:28.353167 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.353167    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8699ae06-aa54-471c-86a7-a5542d3ead70" containerName="payload-pre-processing"
Nov 03 02:48:28.353508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.353178    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c150264b-bd92-475f-8ac8-5591b848bc49" containerName="maas-api"
Nov 03 02:48:28.353508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.353185    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerName="istio-proxy"
Nov 03 02:48:28.353508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.353190    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac0e3fc-4b11-4113-b66f-02320ac59c2c" containerName="payload-processing"
Nov 03 02:48:28.353508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.353198    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerName="istio-proxy"
Nov 03 02:48:28.356532 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.356509    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"
Nov 03 02:48:28.405420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.405389    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:48:28.405680 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.405462    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:48:28.451996 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.451962    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:48:28.452182 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.452004    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 02:48:28.455170 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.455143    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"]
Nov 03 02:48:28.468896 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.468864    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwdh\" (UniqueName: \"kubernetes.io/projected/494fd1c5-6afe-4632-8064-6c000ec92389-kube-api-access-qqwdh\") pod \"maas-api-revoke-sub-85237e45-g87pl\" (UID: \"494fd1c5-6afe-4632-8064-6c000ec92389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"
Nov 03 02:48:28.469119 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.468933    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-g87pl\" (UID: \"494fd1c5-6afe-4632-8064-6c000ec92389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"
Nov 03 02:48:28.469119 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.469038    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-g87pl\" (UID: \"494fd1c5-6afe-4632-8064-6c000ec92389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"
Nov 03 02:48:28.570616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.570579    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-g87pl\" (UID: \"494fd1c5-6afe-4632-8064-6c000ec92389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"
Nov 03 02:48:28.570845 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.570632    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-g87pl\" (UID: \"494fd1c5-6afe-4632-8064-6c000ec92389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"
Nov 03 02:48:28.570845 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.570672    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwdh\" (UniqueName: \"kubernetes.io/projected/494fd1c5-6afe-4632-8064-6c000ec92389-kube-api-access-qqwdh\") pod \"maas-api-revoke-sub-85237e45-g87pl\" (UID: \"494fd1c5-6afe-4632-8064-6c000ec92389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"
Nov 03 02:48:28.571264 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.571240    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-g87pl\" (UID: \"494fd1c5-6afe-4632-8064-6c000ec92389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"
Nov 03 02:48:28.573079 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.573053    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-g87pl\" (UID: \"494fd1c5-6afe-4632-8064-6c000ec92389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"
Nov 03 02:48:28.628483 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.628410    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwdh\" (UniqueName: \"kubernetes.io/projected/494fd1c5-6afe-4632-8064-6c000ec92389-kube-api-access-qqwdh\") pod \"maas-api-revoke-sub-85237e45-g87pl\" (UID: \"494fd1c5-6afe-4632-8064-6c000ec92389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"
Nov 03 02:48:28.669383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:28.669328    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"
Nov 03 02:48:29.046042 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:29.046003    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"]
Nov 03 02:48:29.053898 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:48:29.053858    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494fd1c5_6afe_4632_8064_6c000ec92389.slice/crio-1317ed3c051671a9feb50bd4a197d8beb5d9afe9c17bc82c1e8653c6f84686b4 WatchSource:0}: Error finding container 1317ed3c051671a9feb50bd4a197d8beb5d9afe9c17bc82c1e8653c6f84686b4: Status 404 returned error can't find the container with id 1317ed3c051671a9feb50bd4a197d8beb5d9afe9c17bc82c1e8653c6f84686b4
Nov 03 02:48:29.415914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:29.415877    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl" event={"ID":"494fd1c5-6afe-4632-8064-6c000ec92389","Type":"ContainerStarted","Data":"79a7e562f992275cce4c649e009aecff63ec08174e6814feff373001a015c5f5"}
Nov 03 02:48:29.416316 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:29.415954    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl" event={"ID":"494fd1c5-6afe-4632-8064-6c000ec92389","Type":"ContainerStarted","Data":"1317ed3c051671a9feb50bd4a197d8beb5d9afe9c17bc82c1e8653c6f84686b4"}
Nov 03 02:48:29.529029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:29.528972    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl" podStartSLOduration=1.528956861 podStartE2EDuration="1.528956861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:48:29.527429072 +0000 UTC m=+5388.953035539" watchObservedRunningTime="2026-11-03 02:48:29.528956861 +0000 UTC m=+5388.954563324"
Nov 03 02:48:30.423033 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:30.422997    5508 generic.go:356] "Generic (PLEG): container finished" podID="494fd1c5-6afe-4632-8064-6c000ec92389" containerID="79a7e562f992275cce4c649e009aecff63ec08174e6814feff373001a015c5f5" exitCode=0
Nov 03 02:48:30.423477 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:30.423083    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl" event={"ID":"494fd1c5-6afe-4632-8064-6c000ec92389","Type":"ContainerDied","Data":"79a7e562f992275cce4c649e009aecff63ec08174e6814feff373001a015c5f5"}
Nov 03 02:48:31.209693 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.209659    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"]
Nov 03 02:48:31.210032 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.210019    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerName="istio-proxy"
Nov 03 02:48:31.210078 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.210033    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca36783e-c015-4c0c-9a67-c39c9e1f1207" containerName="istio-proxy"
Nov 03 02:48:31.213214 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.213197    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.237776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.237742    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"]
Nov 03 02:48:31.241168 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.241149    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.266282 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.266254    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-w2lkg\""
Nov 03 02:48:31.296071 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.296039    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.296071 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.296074    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d32d522-440d-42da-84ed-bb3b611a1a2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.296338 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.296093    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d32d522-440d-42da-84ed-bb3b611a1a2f-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.296338 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.296117    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.296338 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.296249    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.296338 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.296275    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mrbj\" (UniqueName: \"kubernetes.io/projected/9d32d522-440d-42da-84ed-bb3b611a1a2f-kube-api-access-9mrbj\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.296520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.296364    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d32d522-440d-42da-84ed-bb3b611a1a2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.296520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.296388    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.296520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.296417    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.316104 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.316068    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"]
Nov 03 02:48:31.366976 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.366943    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"]
Nov 03 02:48:31.397052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397004    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.397052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397060    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mrbj\" (UniqueName: \"kubernetes.io/projected/9d32d522-440d-42da-84ed-bb3b611a1a2f-kube-api-access-9mrbj\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.397388 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397089    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b416926-1f08-4197-80e5-5c3c82166abd-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.397388 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397127    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.397388 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397163    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d32d522-440d-42da-84ed-bb3b611a1a2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.397388 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397325    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b416926-1f08-4197-80e5-5c3c82166abd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.397388 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397367    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.397666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397394    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.397666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397524    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.397666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397557    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.397666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397585    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wksbq\" (UniqueName: \"kubernetes.io/projected/9b416926-1f08-4197-80e5-5c3c82166abd-kube-api-access-wksbq\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.397666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397614    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b416926-1f08-4197-80e5-5c3c82166abd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.397666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397637    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.397918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397690    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.397918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397712    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.397918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397833    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.397918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397884    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.397918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397888    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d32d522-440d-42da-84ed-bb3b611a1a2f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.397918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397912    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d32d522-440d-42da-84ed-bb3b611a1a2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.398150 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397939    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d32d522-440d-42da-84ed-bb3b611a1a2f-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.398150 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.397970    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.398231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.398150    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.398306 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.398281    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.399741 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.399721    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d32d522-440d-42da-84ed-bb3b611a1a2f-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.400187 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.400167    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d32d522-440d-42da-84ed-bb3b611a1a2f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.459379 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.458612    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d32d522-440d-42da-84ed-bb3b611a1a2f-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.460207 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.460156    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mrbj\" (UniqueName: \"kubernetes.io/projected/9d32d522-440d-42da-84ed-bb3b611a1a2f-kube-api-access-9mrbj\") pod \"maas-default-gateway-openshift-default-59f957fdd7-4jmtt\" (UID: \"9d32d522-440d-42da-84ed-bb3b611a1a2f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.498966 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.498926    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b416926-1f08-4197-80e5-5c3c82166abd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.498966 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.498964    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.499268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.498999    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wksbq\" (UniqueName: \"kubernetes.io/projected/9b416926-1f08-4197-80e5-5c3c82166abd-kube-api-access-wksbq\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.499268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.499022    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b416926-1f08-4197-80e5-5c3c82166abd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.499268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.499050    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.499268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.499080    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.499268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.499112    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.499268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.499180    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b416926-1f08-4197-80e5-5c3c82166abd-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.499268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.499233    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.499814 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.499517    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.499945 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.499871    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.500061 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.500035    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b416926-1f08-4197-80e5-5c3c82166abd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.500227 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.500196    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.500405 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.500383    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.502233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.502207    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b416926-1f08-4197-80e5-5c3c82166abd-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.503242 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.503219    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b416926-1f08-4197-80e5-5c3c82166abd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.525940 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.525906    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:31.545817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.545787    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"
Nov 03 02:48:31.557521 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.557491    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b416926-1f08-4197-80e5-5c3c82166abd-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.557639 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.557616    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wksbq\" (UniqueName: \"kubernetes.io/projected/9b416926-1f08-4197-80e5-5c3c82166abd-kube-api-access-wksbq\") pod \"maas-default-gateway-openshift-default-59f957fdd7-bh6ts\" (UID: \"9b416926-1f08-4197-80e5-5c3c82166abd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.700457 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.700415    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494fd1c5-6afe-4632-8064-6c000ec92389-kube-api-access-qqwdh\" (UniqueName: \"kubernetes.io/projected/494fd1c5-6afe-4632-8064-6c000ec92389-kube-api-access-qqwdh\") pod \"494fd1c5-6afe-4632-8064-6c000ec92389\" (UID: \"494fd1c5-6afe-4632-8064-6c000ec92389\") "
Nov 03 02:48:31.700679 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.700494    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-openshift-ca\") pod \"494fd1c5-6afe-4632-8064-6c000ec92389\" (UID: \"494fd1c5-6afe-4632-8064-6c000ec92389\") "
Nov 03 02:48:31.700679 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.700556    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-xks-ca\") pod \"494fd1c5-6afe-4632-8064-6c000ec92389\" (UID: \"494fd1c5-6afe-4632-8064-6c000ec92389\") "
Nov 03 02:48:31.700864 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.700838    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-openshift-ca" pod "494fd1c5-6afe-4632-8064-6c000ec92389" (UID: "494fd1c5-6afe-4632-8064-6c000ec92389"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:48:31.702743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.702715    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-xks-ca" pod "494fd1c5-6afe-4632-8064-6c000ec92389" (UID: "494fd1c5-6afe-4632-8064-6c000ec92389"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:48:31.702851 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.702720    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494fd1c5-6afe-4632-8064-6c000ec92389-kube-api-access-qqwdh" pod "494fd1c5-6afe-4632-8064-6c000ec92389" (UID: "494fd1c5-6afe-4632-8064-6c000ec92389"). InnerVolumeSpecName "kube-api-access-qqwdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:48:31.801593 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.801494    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:48:31.801593 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.801533    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqwdh\" (UniqueName: \"kubernetes.io/projected/494fd1c5-6afe-4632-8064-6c000ec92389-kube-api-access-qqwdh\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:48:31.801593 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.801549    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/494fd1c5-6afe-4632-8064-6c000ec92389-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:48:31.853556 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.853519    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:31.954441 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.954401    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"]
Nov 03 02:48:31.961527 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:48:31.961491    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d32d522_440d_42da_84ed_bb3b611a1a2f.slice/crio-53cf46d4ba060fb76be2e2c24f4a1240d44790d246993d6c8a28a93c642425fe WatchSource:0}: Error finding container 53cf46d4ba060fb76be2e2c24f4a1240d44790d246993d6c8a28a93c642425fe: Status 404 returned error can't find the container with id 53cf46d4ba060fb76be2e2c24f4a1240d44790d246993d6c8a28a93c642425fe
Nov 03 02:48:31.964087 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.964048    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:48:31.964192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.964145    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:48:31.964192 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:31.964183    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:48:32.246054 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.246010    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"]
Nov 03 02:48:32.252586 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:48:32.252554    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b416926_1f08_4197_80e5_5c3c82166abd.slice/crio-41b50160aa17a9c7a6c22280e8dd4d829526d5f2e140b1d0314157d9772314b5 WatchSource:0}: Error finding container 41b50160aa17a9c7a6c22280e8dd4d829526d5f2e140b1d0314157d9772314b5: Status 404 returned error can't find the container with id 41b50160aa17a9c7a6c22280e8dd4d829526d5f2e140b1d0314157d9772314b5
Nov 03 02:48:32.255049 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.255016    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:48:32.255153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.255085    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:48:32.255153 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.255114    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:48:32.446544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.444682    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl" event={"ID":"494fd1c5-6afe-4632-8064-6c000ec92389","Type":"ContainerDied","Data":"1317ed3c051671a9feb50bd4a197d8beb5d9afe9c17bc82c1e8653c6f84686b4"}
Nov 03 02:48:32.446544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.444745    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1317ed3c051671a9feb50bd4a197d8beb5d9afe9c17bc82c1e8653c6f84686b4"
Nov 03 02:48:32.446544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.444823    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"
Nov 03 02:48:32.454095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.453114    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" event={"ID":"9b416926-1f08-4197-80e5-5c3c82166abd","Type":"ContainerStarted","Data":"344aaa20273b7f5b9d3e15c931d751b730e926e10add649559bcde51d97bca31"}
Nov 03 02:48:32.454095 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.453188    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" event={"ID":"9b416926-1f08-4197-80e5-5c3c82166abd","Type":"ContainerStarted","Data":"41b50160aa17a9c7a6c22280e8dd4d829526d5f2e140b1d0314157d9772314b5"}
Nov 03 02:48:32.457436 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.456744    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" event={"ID":"9d32d522-440d-42da-84ed-bb3b611a1a2f","Type":"ContainerStarted","Data":"75452e10c97317ccdcef8c480c7733756f9a32231ff8b82b8d2d5c7fd0606d0e"}
Nov 03 02:48:32.457436 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.456803    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" event={"ID":"9d32d522-440d-42da-84ed-bb3b611a1a2f","Type":"ContainerStarted","Data":"53cf46d4ba060fb76be2e2c24f4a1240d44790d246993d6c8a28a93c642425fe"}
Nov 03 02:48:32.526239 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.526155    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:32.584160 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.583809    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" podStartSLOduration=1.583790635 podStartE2EDuration="1.583790635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:48:32.578494597 +0000 UTC m=+5392.004101062" watchObservedRunningTime="2026-11-03 02:48:32.583790635 +0000 UTC m=+5392.009397098"
Nov 03 02:48:32.742083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.742015    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" podStartSLOduration=1.741995143 podStartE2EDuration="1.741995143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:48:32.740825972 +0000 UTC m=+5392.166432435" watchObservedRunningTime="2026-11-03 02:48:32.741995143 +0000 UTC m=+5392.167601608"
Nov 03 02:48:32.856437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:32.855472    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:33.527749 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:33.527715    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4jmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused" start-of-body=
Nov 03 02:48:33.528225 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:33.528201    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" podUID="9d32d522-440d-42da-84ed-bb3b611a1a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused"
Nov 03 02:48:33.858571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:33.858469    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bh6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" start-of-body=
Nov 03 02:48:33.859317 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:33.859282    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" podUID="9b416926-1f08-4197-80e5-5c3c82166abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused"
Nov 03 02:48:34.526954 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:34.526912    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4jmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused" start-of-body=
Nov 03 02:48:34.527157 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:34.526985    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" podUID="9d32d522-440d-42da-84ed-bb3b611a1a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused"
Nov 03 02:48:34.865435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:34.864464    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bh6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" start-of-body=
Nov 03 02:48:34.865435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:34.864532    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" podUID="9b416926-1f08-4197-80e5-5c3c82166abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused"
Nov 03 02:48:35.526550 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:35.526510    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4jmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused" start-of-body=
Nov 03 02:48:35.526743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:35.526579    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" podUID="9d32d522-440d-42da-84ed-bb3b611a1a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused"
Nov 03 02:48:35.859434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:35.856794    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bh6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" start-of-body=
Nov 03 02:48:35.859434 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:35.856859    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" podUID="9b416926-1f08-4197-80e5-5c3c82166abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused"
Nov 03 02:48:36.526757 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:36.526721    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4jmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused" start-of-body=
Nov 03 02:48:36.527180 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:36.526776    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" podUID="9d32d522-440d-42da-84ed-bb3b611a1a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused"
Nov 03 02:48:36.866433 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:36.860716    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bh6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" start-of-body=
Nov 03 02:48:36.866433 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:36.860778    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" podUID="9b416926-1f08-4197-80e5-5c3c82166abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused"
Nov 03 02:48:37.526879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:37.526838    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4jmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused" start-of-body=
Nov 03 02:48:37.527338 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:37.526902    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" podUID="9d32d522-440d-42da-84ed-bb3b611a1a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused"
Nov 03 02:48:37.858313 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:37.857790    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bh6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" start-of-body=
Nov 03 02:48:37.858313 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:37.857857    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" podUID="9b416926-1f08-4197-80e5-5c3c82166abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused"
Nov 03 02:48:38.527262 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:38.527217    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4jmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused" start-of-body=
Nov 03 02:48:38.527817 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:38.527268    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" podUID="9d32d522-440d-42da-84ed-bb3b611a1a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": dial tcp 10.133.1.123:15021: connect: connection refused"
Nov 03 02:48:38.854858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:38.854753    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bh6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused" start-of-body=
Nov 03 02:48:38.854858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:38.854816    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" podUID="9b416926-1f08-4197-80e5-5c3c82166abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": dial tcp 10.133.1.124:15021: connect: connection refused"
Nov 03 02:48:40.530874 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:40.530823    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4jmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:48:40.531309 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:40.530957    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" podUID="9d32d522-440d-42da-84ed-bb3b611a1a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:48:40.855539 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:40.855422    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bh6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:48:40.855539 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:40.855495    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" podUID="9b416926-1f08-4197-80e5-5c3c82166abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:48:41.572586 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:41.572544    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4jmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:48:41.573130 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:41.572634    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" podUID="9d32d522-440d-42da-84ed-bb3b611a1a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:48:41.715577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:41.715546    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:48:41.834582 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:41.834505    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:48:41.897608 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:41.897557    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bh6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:48:41.897839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:41.897637    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" podUID="9b416926-1f08-4197-80e5-5c3c82166abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:48:42.613606 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:42.613542    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4jmtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:48:42.614181 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:42.613625    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" podUID="9d32d522-440d-42da-84ed-bb3b611a1a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:48:42.731206 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:42.731173    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:42.731598 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:42.731577    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:42.739148 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:42.739122    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt"
Nov 03 02:48:42.898490 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:42.898457    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bh6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:48:42.898690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:42.898513    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" podUID="9b416926-1f08-4197-80e5-5c3c82166abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:48:42.929297 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:42.929268    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:42.929604 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:42.929575    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:42.930263 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:42.930247    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts"
Nov 03 02:48:43.811506 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:48:43.811470    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:48:46.936636 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:46.936554    5508 scope.go:122] "RemoveContainer" containerID="ecc36e5ce76cb5f003d6685d7b7a923033a8b4bfabb0d8313e366a007da6573f"
Nov 03 02:48:46.945550 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:46.945523    5508 scope.go:122] "RemoveContainer" containerID="9ac121b40ff174f911ba753a59d23df7756c050de014ac20aecf5852ec29fb7d"
Nov 03 02:48:46.953361 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:48:46.953326    5508 scope.go:122] "RemoveContainer" containerID="256b385ea4124d3bb6f079575c1314fc888eddc8a597441acddfa68c5d7369ee"
Nov 03 02:49:10.202839 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:10.202796    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:49:13.810480 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:49:13.810441    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:49:15.951396 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:15.951360    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"]
Nov 03 02:49:15.951799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:15.951733    5508 cpu_manager.go:414] "containerMap: removing container" podUID="494fd1c5-6afe-4632-8064-6c000ec92389" containerName="revoke-keys"
Nov 03 02:49:15.951799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:15.951745    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494fd1c5-6afe-4632-8064-6c000ec92389" containerName="revoke-keys"
Nov 03 02:49:15.951880 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:15.951824    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="494fd1c5-6afe-4632-8064-6c000ec92389" containerName="revoke-keys"
Nov 03 02:49:15.955275 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:15.955252    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"
Nov 03 02:49:15.986424 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:15.986393    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k546v\" (UniqueName: \"kubernetes.io/projected/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-kube-api-access-k546v\") pod \"maas-api-revoke-sub-f904bff9-8tkgf\" (UID: \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"
Nov 03 02:49:15.986587 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:15.986457    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-8tkgf\" (UID: \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"
Nov 03 02:49:15.986587 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:15.986540    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-8tkgf\" (UID: \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"
Nov 03 02:49:16.004546 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.004512    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:49:16.050774 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.050736    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:49:16.050774 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.050761    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 02:49:16.050774 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.050761    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:49:16.052963 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.052944    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"]
Nov 03 02:49:16.087748 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.087716    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k546v\" (UniqueName: \"kubernetes.io/projected/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-kube-api-access-k546v\") pod \"maas-api-revoke-sub-f904bff9-8tkgf\" (UID: \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"
Nov 03 02:49:16.087904 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.087792    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-8tkgf\" (UID: \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"
Nov 03 02:49:16.087904 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.087822    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-8tkgf\" (UID: \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"
Nov 03 02:49:16.088568 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.088543    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-8tkgf\" (UID: \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"
Nov 03 02:49:16.090068 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.090046    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-8tkgf\" (UID: \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"
Nov 03 02:49:16.146540 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.146503    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k546v\" (UniqueName: \"kubernetes.io/projected/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-kube-api-access-k546v\") pod \"maas-api-revoke-sub-f904bff9-8tkgf\" (UID: \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"
Nov 03 02:49:16.268533 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.268446    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"
Nov 03 02:49:16.639268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.639190    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"]
Nov 03 02:49:16.647243 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:49:16.647209    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f64c47c_eb7b_4fbb_b6af_cd8b8a2aa5aa.slice/crio-ff570341467d56e797ee0220bd39d499600e136fc6fcfbdb57c7b84bc8497019 WatchSource:0}: Error finding container ff570341467d56e797ee0220bd39d499600e136fc6fcfbdb57c7b84bc8497019: Status 404 returned error can't find the container with id ff570341467d56e797ee0220bd39d499600e136fc6fcfbdb57c7b84bc8497019
Nov 03 02:49:16.744895 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:16.744860    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf" event={"ID":"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa","Type":"ContainerStarted","Data":"ff570341467d56e797ee0220bd39d499600e136fc6fcfbdb57c7b84bc8497019"}
Nov 03 02:49:17.751212 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:17.751125    5508 generic.go:356] "Generic (PLEG): container finished" podID="5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa" containerID="30c7d0901c18f56206d150259310864b7d36e7d486fc79b2939effebd55dc58b" exitCode=0
Nov 03 02:49:17.751704 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:17.751207    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf" event={"ID":"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa","Type":"ContainerDied","Data":"30c7d0901c18f56206d150259310864b7d36e7d486fc79b2939effebd55dc58b"}
Nov 03 02:49:18.887798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:18.887771    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"
Nov 03 02:49:18.914333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:18.914306    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-kube-api-access-k546v\" (UniqueName: \"kubernetes.io/projected/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-kube-api-access-k546v\") pod \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\" (UID: \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\") "
Nov 03 02:49:18.914554 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:18.914343    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-xks-ca\") pod \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\" (UID: \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\") "
Nov 03 02:49:18.914554 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:18.914412    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-openshift-ca\") pod \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\" (UID: \"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa\") "
Nov 03 02:49:18.914836 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:18.914814    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-openshift-ca" pod "5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa" (UID: "5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:49:18.916384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:18.916362    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-kube-api-access-k546v" pod "5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa" (UID: "5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa"). InnerVolumeSpecName "kube-api-access-k546v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:49:18.916513 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:18.916492    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-xks-ca" pod "5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa" (UID: "5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:49:19.016057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:19.015986    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k546v\" (UniqueName: \"kubernetes.io/projected/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-kube-api-access-k546v\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:49:19.016057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:19.016013    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:49:19.016057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:19.016024    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:49:19.762520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:19.762488    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"
Nov 03 02:49:19.762716 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:19.762492    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf" event={"ID":"5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa","Type":"ContainerDied","Data":"ff570341467d56e797ee0220bd39d499600e136fc6fcfbdb57c7b84bc8497019"}
Nov 03 02:49:19.762716 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:19.762620    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff570341467d56e797ee0220bd39d499600e136fc6fcfbdb57c7b84bc8497019"
Nov 03 02:49:20.046680 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:20.046587    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"]
Nov 03 02:49:20.068489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:20.068456    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83852c7-782hp"]
Nov 03 02:49:20.689807 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:20.689777    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5429f1b5-6d1b-4891-9c5d-80876ebd15b2" path="/var/lib/kubelet/pods/5429f1b5-6d1b-4891-9c5d-80876ebd15b2/volumes"
Nov 03 02:49:28.784614 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:28.784573    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:49:31.686390 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.686357    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"]
Nov 03 02:49:31.686792 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.686725    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa" containerName="revoke-keys"
Nov 03 02:49:31.686792 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.686735    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa" containerName="revoke-keys"
Nov 03 02:49:31.686862 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.686804    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa" containerName="revoke-keys"
Nov 03 02:49:31.689911 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.689893    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.733359 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.733317    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93283bb-8765-407f-86eb-af53fa0e399e-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.733524 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.733368    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93283bb-8765-407f-86eb-af53fa0e399e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.733524 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.733461    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rzf\" (UniqueName: \"kubernetes.io/projected/f93283bb-8765-407f-86eb-af53fa0e399e-kube-api-access-n5rzf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.733524 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.733505    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.733659 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.733552    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93283bb-8765-407f-86eb-af53fa0e399e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.733659 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.733582    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.733659 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.733633    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.733779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.733668    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.733779 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.733750    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.834713 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.834676    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.834918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.834728    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.834918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.834767    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93283bb-8765-407f-86eb-af53fa0e399e-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.834918 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.834787    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93283bb-8765-407f-86eb-af53fa0e399e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.835103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.834969    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rzf\" (UniqueName: \"kubernetes.io/projected/f93283bb-8765-407f-86eb-af53fa0e399e-kube-api-access-n5rzf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.835103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.835043    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.835103 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.835091    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93283bb-8765-407f-86eb-af53fa0e399e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.835285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.835108    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.835285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.835114    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.835285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.835150    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.835510 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.835408    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.835572 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.835505    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.835572 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.835558    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f93283bb-8765-407f-86eb-af53fa0e399e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.835659 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.835577    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.837208 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.837188    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f93283bb-8765-407f-86eb-af53fa0e399e-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.837648 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.837630    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f93283bb-8765-407f-86eb-af53fa0e399e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.893586 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.893540    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f93283bb-8765-407f-86eb-af53fa0e399e-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.894664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.894637    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rzf\" (UniqueName: \"kubernetes.io/projected/f93283bb-8765-407f-86eb-af53fa0e399e-kube-api-access-n5rzf\") pod \"maas-default-gateway-openshift-default-59f957fdd7-cnxmt\" (UID: \"f93283bb-8765-407f-86eb-af53fa0e399e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:31.895694 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:31.895636    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"]
Nov 03 02:49:32.002576 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:32.002493    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:32.046507 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:32.046427    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"]
Nov 03 02:49:32.049504 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:32.049469    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5efaa27-wtp5k"]
Nov 03 02:49:32.430388 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:32.430336    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"]
Nov 03 02:49:32.437418 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:49:32.437392    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93283bb_8765_407f_86eb_af53fa0e399e.slice/crio-f585458157a3f97d769f669dc3fa89707b2723ca61d539cfdd5d9361075df95b WatchSource:0}: Error finding container f585458157a3f97d769f669dc3fa89707b2723ca61d539cfdd5d9361075df95b: Status 404 returned error can't find the container with id f585458157a3f97d769f669dc3fa89707b2723ca61d539cfdd5d9361075df95b
Nov 03 02:49:32.439625 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:32.439595    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:49:32.439699 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:32.439659    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:49:32.439699 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:32.439688    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:49:32.692135 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:32.692043    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb156e1e-6a24-42d9-a471-64f097e7f548" path="/var/lib/kubelet/pods/eb156e1e-6a24-42d9-a471-64f097e7f548/volumes"
Nov 03 02:49:32.831025 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:32.830991    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" event={"ID":"f93283bb-8765-407f-86eb-af53fa0e399e","Type":"ContainerStarted","Data":"c6e08a92181a06a0ab7d78207c89919c83b4e39e6217e0b0cdeff12596fdaba6"}
Nov 03 02:49:32.831207 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:32.831039    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" event={"ID":"f93283bb-8765-407f-86eb-af53fa0e399e","Type":"ContainerStarted","Data":"f585458157a3f97d769f669dc3fa89707b2723ca61d539cfdd5d9361075df95b"}
Nov 03 02:49:32.946736 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:32.946635    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podStartSLOduration=1.946619349 podStartE2EDuration="1.946619349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:49:32.944715929 +0000 UTC m=+5452.370322411" watchObservedRunningTime="2026-11-03 02:49:32.946619349 +0000 UTC m=+5452.372225813"
Nov 03 02:49:33.003097 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:33.003066    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:33.004666 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:33.004629    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cnxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body=
Nov 03 02:49:33.004812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:33.004700    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podUID="f93283bb-8765-407f-86eb-af53fa0e399e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused"
Nov 03 02:49:34.002944 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:34.002904    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cnxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body=
Nov 03 02:49:34.003415 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:34.002985    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podUID="f93283bb-8765-407f-86eb-af53fa0e399e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused"
Nov 03 02:49:35.004372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:35.003733    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cnxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body=
Nov 03 02:49:35.004372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:35.003809    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podUID="f93283bb-8765-407f-86eb-af53fa0e399e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused"
Nov 03 02:49:36.004372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:36.003928    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cnxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body=
Nov 03 02:49:36.004372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:36.004009    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podUID="f93283bb-8765-407f-86eb-af53fa0e399e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused"
Nov 03 02:49:37.003412 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:37.003374    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cnxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body=
Nov 03 02:49:37.003622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:37.003436    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podUID="f93283bb-8765-407f-86eb-af53fa0e399e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused"
Nov 03 02:49:38.003781 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:38.003748    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cnxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body=
Nov 03 02:49:38.004201 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:38.003801    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podUID="f93283bb-8765-407f-86eb-af53fa0e399e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused"
Nov 03 02:49:38.678321 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:38.678280    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:49:39.003980 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:39.003892    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cnxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused" start-of-body=
Nov 03 02:49:39.003980 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:39.003962    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podUID="f93283bb-8765-407f-86eb-af53fa0e399e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": dial tcp 10.133.1.126:15021: connect: connection refused"
Nov 03 02:49:41.003774 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:41.003731    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cnxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:49:41.004238 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:41.003814    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podUID="f93283bb-8765-407f-86eb-af53fa0e399e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:49:42.045501 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:42.045462    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cnxmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:49:42.045934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:42.045533    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podUID="f93283bb-8765-407f-86eb-af53fa0e399e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:49:42.781985 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:42.781947    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:42.782398 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:42.782361    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:42.783158 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:42.783137    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt"
Nov 03 02:49:43.049843 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:43.049761    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"]
Nov 03 02:49:43.054541 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:43.054514    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47aa225f-zz72q"]
Nov 03 02:49:43.818650 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:49:43.818605    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:49:44.718374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:44.718314    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b5d5f3-b3e3-46a6-938b-fe7dc75add83" path="/var/lib/kubelet/pods/65b5d5f3-b3e3-46a6-938b-fe7dc75add83/volumes"
Nov 03 02:49:46.850733 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:46.850694    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"]
Nov 03 02:49:46.855568 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:46.855544    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:46.934652 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:46.934618    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"]
Nov 03 02:49:46.941411 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:46.941379    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:46.941559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:46.941435    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:46.941559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:46.941456    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:46.941559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:46.941482    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1185a711-3994-4129-a641-b93e62e8effe-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:46.941559 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:46.941527    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1185a711-3994-4129-a641-b93e62e8effe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:46.941719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:46.941562    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ltxg\" (UniqueName: \"kubernetes.io/projected/1185a711-3994-4129-a641-b93e62e8effe-kube-api-access-5ltxg\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:46.941719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:46.941600    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1185a711-3994-4129-a641-b93e62e8effe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:46.941719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:46.941638    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:46.941719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:46.941668    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:46.998264 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:46.998231    5508 scope.go:122] "RemoveContainer" containerID="e2d22024a69772fe778cd265f5935e2350d9c922a90e6d0cb3a1d165d991873d"
Nov 03 02:49:47.042876 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.042840    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.043059 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.042885    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.043059 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.042921    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.043059 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.042955    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.043059 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.042988    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.043059 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.043012    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1185a711-3994-4129-a641-b93e62e8effe-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.043324 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.043062    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1185a711-3994-4129-a641-b93e62e8effe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.043324 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.043092    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ltxg\" (UniqueName: \"kubernetes.io/projected/1185a711-3994-4129-a641-b93e62e8effe-kube-api-access-5ltxg\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.043466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.043411    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.043466 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.043444    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1185a711-3994-4129-a641-b93e62e8effe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.043577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.043477    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.043804 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.043783    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.043993 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.043831    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.044122 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.044101    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1185a711-3994-4129-a641-b93e62e8effe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.045620 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.045599    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1185a711-3994-4129-a641-b93e62e8effe-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.045986 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.045966    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1185a711-3994-4129-a641-b93e62e8effe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.101770 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.101699    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1185a711-3994-4129-a641-b93e62e8effe-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.101908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.101765    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ltxg\" (UniqueName: \"kubernetes.io/projected/1185a711-3994-4129-a641-b93e62e8effe-kube-api-access-5ltxg\") pod \"maas-default-gateway-openshift-default-59f957fdd7-xzjj9\" (UID: \"1185a711-3994-4129-a641-b93e62e8effe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.168420 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.168381    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:47.599853 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.599815    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"]
Nov 03 02:49:47.606756 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:49:47.606723    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1185a711_3994_4129_a641_b93e62e8effe.slice/crio-6c50ee1332053ddf13bb093f81f1177a23c3c0d025bcce281dd117384f13aa16 WatchSource:0}: Error finding container 6c50ee1332053ddf13bb093f81f1177a23c3c0d025bcce281dd117384f13aa16: Status 404 returned error can't find the container with id 6c50ee1332053ddf13bb093f81f1177a23c3c0d025bcce281dd117384f13aa16
Nov 03 02:49:47.609002 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.608967    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:49:47.609082 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.609041    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:49:47.609082 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.609075    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 02:49:47.952938 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.952894    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9" event={"ID":"1185a711-3994-4129-a641-b93e62e8effe","Type":"ContainerStarted","Data":"90973c3b5c1ccd3b4a3085e85f40ff6332a1941803c4c7bf652f2bda78e00933"}
Nov 03 02:49:47.953435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:47.952975    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9" event={"ID":"1185a711-3994-4129-a641-b93e62e8effe","Type":"ContainerStarted","Data":"6c50ee1332053ddf13bb093f81f1177a23c3c0d025bcce281dd117384f13aa16"}
Nov 03 02:49:48.072082 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:48.072032    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9" podStartSLOduration=2.072018829 podStartE2EDuration="2.072018829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:49:48.071951661 +0000 UTC m=+5467.497558126" watchObservedRunningTime="2026-11-03 02:49:48.072018829 +0000 UTC m=+5467.497625292"
Nov 03 02:49:48.171285 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:48.171248    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:48.172000 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:48.171681    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xzjj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused" start-of-body=
Nov 03 02:49:48.172000 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:48.171736    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9" podUID="1185a711-3994-4129-a641-b93e62e8effe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused"
Nov 03 02:49:48.307690 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:48.307603    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:49:49.170803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:49.170766    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xzjj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused" start-of-body=
Nov 03 02:49:49.171536 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:49.171504    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9" podUID="1185a711-3994-4129-a641-b93e62e8effe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused"
Nov 03 02:49:50.153373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.148461    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"]
Nov 03 02:49:50.173826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.173782    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xzjj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused" start-of-body=
Nov 03 02:49:50.174395 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.173849    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9" podUID="1185a711-3994-4129-a641-b93e62e8effe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused"
Nov 03 02:49:50.174888 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.174866    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"
Nov 03 02:49:50.190121 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.190094    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-cbwf5\" (UID: \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"
Nov 03 02:49:50.190242 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.190152    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-cbwf5\" (UID: \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"
Nov 03 02:49:50.190242 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.190223    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4krk\" (UniqueName: \"kubernetes.io/projected/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-kube-api-access-m4krk\") pod \"maas-api-revoke-sub-f03629db-cbwf5\" (UID: \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"
Nov 03 02:49:50.213179 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.213147    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"]
Nov 03 02:49:50.271373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.266654    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 02:49:50.271373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.266858    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:49:50.291315 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.291281    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-cbwf5\" (UID: \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"
Nov 03 02:49:50.291492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.291324    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4krk\" (UniqueName: \"kubernetes.io/projected/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-kube-api-access-m4krk\") pod \"maas-api-revoke-sub-f03629db-cbwf5\" (UID: \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"
Nov 03 02:49:50.291492 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.291442    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-cbwf5\" (UID: \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"
Nov 03 02:49:50.324275 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.324247    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:49:50.325654 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.325636    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:49:50.332261 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.332240    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-cbwf5\" (UID: \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"
Nov 03 02:49:50.334129 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.334107    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-cbwf5\" (UID: \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"
Nov 03 02:49:50.378394 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.378119    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4krk\" (UniqueName: \"kubernetes.io/projected/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-kube-api-access-m4krk\") pod \"maas-api-revoke-sub-f03629db-cbwf5\" (UID: \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"
Nov 03 02:49:50.488032 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.487940    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"
Nov 03 02:49:50.878709 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.878667    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"]
Nov 03 02:49:50.879101 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:49:50.879071    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19eb7fd_9a7c_48cc_ba64_bf4f8ca207c8.slice/crio-4744bac30b15c73351aba347eb8c3904a9c316d907ed04330369ad2f499cf909 WatchSource:0}: Error finding container 4744bac30b15c73351aba347eb8c3904a9c316d907ed04330369ad2f499cf909: Status 404 returned error can't find the container with id 4744bac30b15c73351aba347eb8c3904a9c316d907ed04330369ad2f499cf909
Nov 03 02:49:50.987696 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:50.987659    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5" event={"ID":"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8","Type":"ContainerStarted","Data":"4744bac30b15c73351aba347eb8c3904a9c316d907ed04330369ad2f499cf909"}
Nov 03 02:49:51.169204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:51.168756    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xzjj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused" start-of-body=
Nov 03 02:49:51.169204 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:51.168816    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9" podUID="1185a711-3994-4129-a641-b93e62e8effe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused"
Nov 03 02:49:51.994952 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:51.994918    5508 generic.go:356] "Generic (PLEG): container finished" podID="f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8" containerID="0681a5df0c261bc3702f7332dcb51852696d67418b506ad38c8b61f85333aeae" exitCode=0
Nov 03 02:49:51.995327 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:51.994970    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5" event={"ID":"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8","Type":"ContainerDied","Data":"0681a5df0c261bc3702f7332dcb51852696d67418b506ad38c8b61f85333aeae"}
Nov 03 02:49:52.173806 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:52.173717    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xzjj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused" start-of-body=
Nov 03 02:49:52.173806 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:52.173789    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9" podUID="1185a711-3994-4129-a641-b93e62e8effe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused"
Nov 03 02:49:53.173374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:53.172768    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xzjj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused" start-of-body=
Nov 03 02:49:53.173374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:53.172831    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9" podUID="1185a711-3994-4129-a641-b93e62e8effe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused"
Nov 03 02:49:53.181126 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:53.181101    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"
Nov 03 02:49:53.216140 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:53.216107    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-xks-ca\") pod \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\" (UID: \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\") "
Nov 03 02:49:53.216301 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:53.216154    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-openshift-ca\") pod \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\" (UID: \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\") "
Nov 03 02:49:53.216301 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:53.216221    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-kube-api-access-m4krk\" (UniqueName: \"kubernetes.io/projected/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-kube-api-access-m4krk\") pod \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\" (UID: \"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8\") "
Nov 03 02:49:53.216616 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:53.216587    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-openshift-ca" pod "f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8" (UID: "f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:49:53.218212 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:53.218187    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-xks-ca" pod "f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8" (UID: "f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:49:53.218282 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:53.218197    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-kube-api-access-m4krk" pod "f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8" (UID: "f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8"). InnerVolumeSpecName "kube-api-access-m4krk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:49:53.317290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:53.317194    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4krk\" (UniqueName: \"kubernetes.io/projected/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-kube-api-access-m4krk\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:49:53.317290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:53.317224    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:49:53.317290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:53.317233    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:49:54.011177 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:54.011143    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5" event={"ID":"f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8","Type":"ContainerDied","Data":"4744bac30b15c73351aba347eb8c3904a9c316d907ed04330369ad2f499cf909"}
Nov 03 02:49:54.011398 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:54.011199    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4744bac30b15c73351aba347eb8c3904a9c316d907ed04330369ad2f499cf909"
Nov 03 02:49:54.011398 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:54.011160    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"
Nov 03 02:49:54.047043 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:54.047015    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"]
Nov 03 02:49:54.050626 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:54.050604    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5faf4d8d-lfh9w"]
Nov 03 02:49:54.169635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:54.169603    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xzjj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused" start-of-body=
Nov 03 02:49:54.169826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:54.169656    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9" podUID="1185a711-3994-4129-a641-b93e62e8effe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.127:15021/healthz/ready\": dial tcp 10.133.1.127:15021: connect: connection refused"
Nov 03 02:49:54.691184 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:54.691152    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b095a04-cb9f-484d-a4b6-e9e963337d69" path="/var/lib/kubelet/pods/6b095a04-cb9f-484d-a4b6-e9e963337d69/volumes"
Nov 03 02:49:55.898664 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:55.898632    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:55.899186 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:55.899077    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:55.899872 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:55.899845    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9"
Nov 03 02:49:57.235172 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:49:57.235129    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:50:06.301869 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:06.301826    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:50:10.567923 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.567883    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"]
Nov 03 02:50:10.568339 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.568279    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8" containerName="revoke-keys"
Nov 03 02:50:10.568339 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.568291    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8" containerName="revoke-keys"
Nov 03 02:50:10.568460 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.568392    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8" containerName="revoke-keys"
Nov 03 02:50:10.572948 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.572926    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"
Nov 03 02:50:10.623227 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.623186    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:50:10.632144 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.632112    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-5dnfw\" (UID: \"a134e67a-9610-491a-93d8-6ac45346a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"
Nov 03 02:50:10.632314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.632159    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlvrx\" (UniqueName: \"kubernetes.io/projected/a134e67a-9610-491a-93d8-6ac45346a634-kube-api-access-tlvrx\") pod \"maas-api-revoke-sub-0165d7d2-5dnfw\" (UID: \"a134e67a-9610-491a-93d8-6ac45346a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"
Nov 03 02:50:10.632425 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.632372    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-5dnfw\" (UID: \"a134e67a-9610-491a-93d8-6ac45346a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"
Nov 03 02:50:10.669340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.669300    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:50:10.669340 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.669321    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:50:10.669641 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.669428    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 02:50:10.671473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.671448    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"]
Nov 03 02:50:10.734028 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.733991    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-5dnfw\" (UID: \"a134e67a-9610-491a-93d8-6ac45346a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"
Nov 03 02:50:10.734268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.734092    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-5dnfw\" (UID: \"a134e67a-9610-491a-93d8-6ac45346a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"
Nov 03 02:50:10.734268 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.734122    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlvrx\" (UniqueName: \"kubernetes.io/projected/a134e67a-9610-491a-93d8-6ac45346a634-kube-api-access-tlvrx\") pod \"maas-api-revoke-sub-0165d7d2-5dnfw\" (UID: \"a134e67a-9610-491a-93d8-6ac45346a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"
Nov 03 02:50:10.734836 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.734772    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-5dnfw\" (UID: \"a134e67a-9610-491a-93d8-6ac45346a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"
Nov 03 02:50:10.736437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.736419    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-5dnfw\" (UID: \"a134e67a-9610-491a-93d8-6ac45346a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"
Nov 03 02:50:10.793879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.793846    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlvrx\" (UniqueName: \"kubernetes.io/projected/a134e67a-9610-491a-93d8-6ac45346a634-kube-api-access-tlvrx\") pod \"maas-api-revoke-sub-0165d7d2-5dnfw\" (UID: \"a134e67a-9610-491a-93d8-6ac45346a634\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"
Nov 03 02:50:10.886028 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:10.885955    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"
Nov 03 02:50:11.262065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:11.262028    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"]
Nov 03 02:50:11.269218 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:50:11.269183    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda134e67a_9610_491a_93d8_6ac45346a634.slice/crio-55904195c87b8791ca537f14383e909d9218cff09bb4b986a797369047bd2c1a WatchSource:0}: Error finding container 55904195c87b8791ca537f14383e909d9218cff09bb4b986a797369047bd2c1a: Status 404 returned error can't find the container with id 55904195c87b8791ca537f14383e909d9218cff09bb4b986a797369047bd2c1a
Nov 03 02:50:12.144641 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:12.144603    5508 generic.go:356] "Generic (PLEG): container finished" podID="a134e67a-9610-491a-93d8-6ac45346a634" containerID="259b767b2c3963aee11027daeb8efcf0b509620dbfaf0c9c0ae97341d200b1a0" exitCode=0
Nov 03 02:50:12.145050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:12.144681    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw" event={"ID":"a134e67a-9610-491a-93d8-6ac45346a634","Type":"ContainerDied","Data":"259b767b2c3963aee11027daeb8efcf0b509620dbfaf0c9c0ae97341d200b1a0"}
Nov 03 02:50:12.145050 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:12.144751    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw" event={"ID":"a134e67a-9610-491a-93d8-6ac45346a634","Type":"ContainerStarted","Data":"55904195c87b8791ca537f14383e909d9218cff09bb4b986a797369047bd2c1a"}
Nov 03 02:50:13.281571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:13.281544    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"
Nov 03 02:50:13.360059 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:13.360020    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-xks-ca\") pod \"a134e67a-9610-491a-93d8-6ac45346a634\" (UID: \"a134e67a-9610-491a-93d8-6ac45346a634\") "
Nov 03 02:50:13.360247 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:13.360079    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a134e67a-9610-491a-93d8-6ac45346a634-kube-api-access-tlvrx\" (UniqueName: \"kubernetes.io/projected/a134e67a-9610-491a-93d8-6ac45346a634-kube-api-access-tlvrx\") pod \"a134e67a-9610-491a-93d8-6ac45346a634\" (UID: \"a134e67a-9610-491a-93d8-6ac45346a634\") "
Nov 03 02:50:13.360247 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:13.360134    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-openshift-ca\") pod \"a134e67a-9610-491a-93d8-6ac45346a634\" (UID: \"a134e67a-9610-491a-93d8-6ac45346a634\") "
Nov 03 02:50:13.360607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:13.360576    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-openshift-ca" pod "a134e67a-9610-491a-93d8-6ac45346a634" (UID: "a134e67a-9610-491a-93d8-6ac45346a634"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:50:13.362145 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:13.362120    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-xks-ca" pod "a134e67a-9610-491a-93d8-6ac45346a634" (UID: "a134e67a-9610-491a-93d8-6ac45346a634"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:50:13.362254 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:13.362231    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a134e67a-9610-491a-93d8-6ac45346a634-kube-api-access-tlvrx" pod "a134e67a-9610-491a-93d8-6ac45346a634" (UID: "a134e67a-9610-491a-93d8-6ac45346a634"). InnerVolumeSpecName "kube-api-access-tlvrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:50:13.461822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:13.461737    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:13.461822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:13.461774    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlvrx\" (UniqueName: \"kubernetes.io/projected/a134e67a-9610-491a-93d8-6ac45346a634-kube-api-access-tlvrx\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:13.461822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:13.461785    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a134e67a-9610-491a-93d8-6ac45346a634-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:13.812049 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:50:13.811965    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:50:14.156591 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:14.156557    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw" event={"ID":"a134e67a-9610-491a-93d8-6ac45346a634","Type":"ContainerDied","Data":"55904195c87b8791ca537f14383e909d9218cff09bb4b986a797369047bd2c1a"}
Nov 03 02:50:14.156591 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:14.156594    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"
Nov 03 02:50:14.156803 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:14.156616    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55904195c87b8791ca537f14383e909d9218cff09bb4b986a797369047bd2c1a"
Nov 03 02:50:16.114486 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:16.114400    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:50:22.537435 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.537396    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"]
Nov 03 02:50:22.537934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.537785    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a134e67a-9610-491a-93d8-6ac45346a634" containerName="revoke-keys"
Nov 03 02:50:22.537934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.537797    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a134e67a-9610-491a-93d8-6ac45346a634" containerName="revoke-keys"
Nov 03 02:50:22.537934 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.537869    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a134e67a-9610-491a-93d8-6ac45346a634" containerName="revoke-keys"
Nov 03 02:50:22.542142 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.542120    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"
Nov 03 02:50:22.592448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.592406    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:50:22.592448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.592406    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:50:22.592448 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.592416    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:50:22.597948 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.597919    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 02:50:22.640990 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.640954    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"]
Nov 03 02:50:22.657534 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.657504    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-58zxg\" (UID: \"b094c569-de76-4825-95ea-6d89ecffd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"
Nov 03 02:50:22.657773 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.657635    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgsqm\" (UniqueName: \"kubernetes.io/projected/b094c569-de76-4825-95ea-6d89ecffd3b9-kube-api-access-sgsqm\") pod \"maas-api-revoke-sub-6b19338f-58zxg\" (UID: \"b094c569-de76-4825-95ea-6d89ecffd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"
Nov 03 02:50:22.657773 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.657658    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-58zxg\" (UID: \"b094c569-de76-4825-95ea-6d89ecffd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"
Nov 03 02:50:22.758384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.758325    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgsqm\" (UniqueName: \"kubernetes.io/projected/b094c569-de76-4825-95ea-6d89ecffd3b9-kube-api-access-sgsqm\") pod \"maas-api-revoke-sub-6b19338f-58zxg\" (UID: \"b094c569-de76-4825-95ea-6d89ecffd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"
Nov 03 02:50:22.758577 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.758500    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-58zxg\" (UID: \"b094c569-de76-4825-95ea-6d89ecffd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"
Nov 03 02:50:22.758622 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.758594    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-58zxg\" (UID: \"b094c569-de76-4825-95ea-6d89ecffd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"
Nov 03 02:50:22.759290 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.759272    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-58zxg\" (UID: \"b094c569-de76-4825-95ea-6d89ecffd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"
Nov 03 02:50:22.760831 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.760811    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-58zxg\" (UID: \"b094c569-de76-4825-95ea-6d89ecffd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"
Nov 03 02:50:22.816319 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.816236    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgsqm\" (UniqueName: \"kubernetes.io/projected/b094c569-de76-4825-95ea-6d89ecffd3b9-kube-api-access-sgsqm\") pod \"maas-api-revoke-sub-6b19338f-58zxg\" (UID: \"b094c569-de76-4825-95ea-6d89ecffd3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"
Nov 03 02:50:22.855194 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:22.855159    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"
Nov 03 02:50:23.236224 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:23.236192    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"]
Nov 03 02:50:23.243086 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:50:23.243056    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb094c569_de76_4825_95ea_6d89ecffd3b9.slice/crio-5fcc66207cd84adf9ed13e9f228bf473e0c7f3f389eedc0f7f3787d468b54f73 WatchSource:0}: Error finding container 5fcc66207cd84adf9ed13e9f228bf473e0c7f3f389eedc0f7f3787d468b54f73: Status 404 returned error can't find the container with id 5fcc66207cd84adf9ed13e9f228bf473e0c7f3f389eedc0f7f3787d468b54f73
Nov 03 02:50:24.214712 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:24.214674    5508 generic.go:356] "Generic (PLEG): container finished" podID="b094c569-de76-4825-95ea-6d89ecffd3b9" containerID="6ede20bd7f1591b951cff8c88b8aaeef6e860a24af7186ee663f2372e9dc371f" exitCode=0
Nov 03 02:50:24.215129 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:24.214758    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg" event={"ID":"b094c569-de76-4825-95ea-6d89ecffd3b9","Type":"ContainerDied","Data":"6ede20bd7f1591b951cff8c88b8aaeef6e860a24af7186ee663f2372e9dc371f"}
Nov 03 02:50:24.215129 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:24.214817    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg" event={"ID":"b094c569-de76-4825-95ea-6d89ecffd3b9","Type":"ContainerStarted","Data":"5fcc66207cd84adf9ed13e9f228bf473e0c7f3f389eedc0f7f3787d468b54f73"}
Nov 03 02:50:25.353743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:25.353718    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"
Nov 03 02:50:25.486694 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:25.486598    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-xks-ca\") pod \"b094c569-de76-4825-95ea-6d89ecffd3b9\" (UID: \"b094c569-de76-4825-95ea-6d89ecffd3b9\") "
Nov 03 02:50:25.486888 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:25.486788    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-openshift-ca\") pod \"b094c569-de76-4825-95ea-6d89ecffd3b9\" (UID: \"b094c569-de76-4825-95ea-6d89ecffd3b9\") "
Nov 03 02:50:25.486888 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:25.486854    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b094c569-de76-4825-95ea-6d89ecffd3b9-kube-api-access-sgsqm\" (UniqueName: \"kubernetes.io/projected/b094c569-de76-4825-95ea-6d89ecffd3b9-kube-api-access-sgsqm\") pod \"b094c569-de76-4825-95ea-6d89ecffd3b9\" (UID: \"b094c569-de76-4825-95ea-6d89ecffd3b9\") "
Nov 03 02:50:25.487168 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:25.487142    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-openshift-ca" pod "b094c569-de76-4825-95ea-6d89ecffd3b9" (UID: "b094c569-de76-4825-95ea-6d89ecffd3b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:50:25.488782 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:25.488759    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b094c569-de76-4825-95ea-6d89ecffd3b9-kube-api-access-sgsqm" pod "b094c569-de76-4825-95ea-6d89ecffd3b9" (UID: "b094c569-de76-4825-95ea-6d89ecffd3b9"). InnerVolumeSpecName "kube-api-access-sgsqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:50:25.488902 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:25.488863    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-xks-ca" pod "b094c569-de76-4825-95ea-6d89ecffd3b9" (UID: "b094c569-de76-4825-95ea-6d89ecffd3b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:50:25.588514 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:25.588473    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgsqm\" (UniqueName: \"kubernetes.io/projected/b094c569-de76-4825-95ea-6d89ecffd3b9-kube-api-access-sgsqm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:25.588514 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:25.588509    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:25.588514 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:25.588519    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b094c569-de76-4825-95ea-6d89ecffd3b9-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:50:26.227449 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:26.227418    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"
Nov 03 02:50:26.227651 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:26.227445    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg" event={"ID":"b094c569-de76-4825-95ea-6d89ecffd3b9","Type":"ContainerDied","Data":"5fcc66207cd84adf9ed13e9f228bf473e0c7f3f389eedc0f7f3787d468b54f73"}
Nov 03 02:50:26.227651 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:26.227504    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fcc66207cd84adf9ed13e9f228bf473e0c7f3f389eedc0f7f3787d468b54f73"
Nov 03 02:50:35.408791 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:35.408744    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:50:38.607371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.606418    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"]
Nov 03 02:50:38.607371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.607080    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b094c569-de76-4825-95ea-6d89ecffd3b9" containerName="revoke-keys"
Nov 03 02:50:38.607371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.607106    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b094c569-de76-4825-95ea-6d89ecffd3b9" containerName="revoke-keys"
Nov 03 02:50:38.607371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.607225    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b094c569-de76-4825-95ea-6d89ecffd3b9" containerName="revoke-keys"
Nov 03 02:50:38.616373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.610946    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"
Nov 03 02:50:38.650330 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.650298    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"]
Nov 03 02:50:38.709692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.709631    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-1bfe99e1-epp-sa-dockercfg-r4lct\""
Nov 03 02:50:38.709939 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.709747    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-1bfe99e1-kserve-self-signed-certs\""
Nov 03 02:50:38.788371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.788257    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p52pd\" (UniqueName: \"kubernetes.io/projected/80a44056-1c78-41e9-94b9-68746bbcbd2c-kube-api-access-p52pd\") pod \"e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv\" (UID: \"80a44056-1c78-41e9-94b9-68746bbcbd2c\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"
Nov 03 02:50:38.788371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.788307    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80a44056-1c78-41e9-94b9-68746bbcbd2c-tls-certs\") pod \"e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv\" (UID: \"80a44056-1c78-41e9-94b9-68746bbcbd2c\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"
Nov 03 02:50:38.889516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.889416    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p52pd\" (UniqueName: \"kubernetes.io/projected/80a44056-1c78-41e9-94b9-68746bbcbd2c-kube-api-access-p52pd\") pod \"e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv\" (UID: \"80a44056-1c78-41e9-94b9-68746bbcbd2c\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"
Nov 03 02:50:38.889516 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.889481    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80a44056-1c78-41e9-94b9-68746bbcbd2c-tls-certs\") pod \"e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv\" (UID: \"80a44056-1c78-41e9-94b9-68746bbcbd2c\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"
Nov 03 02:50:38.892024 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.892000    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80a44056-1c78-41e9-94b9-68746bbcbd2c-tls-certs\") pod \"e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv\" (UID: \"80a44056-1c78-41e9-94b9-68746bbcbd2c\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"
Nov 03 02:50:38.947473 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:38.947438    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p52pd\" (UniqueName: \"kubernetes.io/projected/80a44056-1c78-41e9-94b9-68746bbcbd2c-kube-api-access-p52pd\") pod \"e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv\" (UID: \"80a44056-1c78-41e9-94b9-68746bbcbd2c\") " pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"
Nov 03 02:50:39.233891 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:39.233848    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"
Nov 03 02:50:39.612618 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:39.612532    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"]
Nov 03 02:50:39.620888 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:50:39.620855    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a44056_1c78_41e9_94b9_68746bbcbd2c.slice/crio-545df2ba3118c95351e3179ea6eac2ec6a44c0cc5c53e5df095e42d9f612169f WatchSource:0}: Error finding container 545df2ba3118c95351e3179ea6eac2ec6a44c0cc5c53e5df095e42d9f612169f: Status 404 returned error can't find the container with id 545df2ba3118c95351e3179ea6eac2ec6a44c0cc5c53e5df095e42d9f612169f
Nov 03 02:50:40.328778 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:40.328737    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv" event={"ID":"80a44056-1c78-41e9-94b9-68746bbcbd2c","Type":"ContainerStarted","Data":"545df2ba3118c95351e3179ea6eac2ec6a44c0cc5c53e5df095e42d9f612169f"}
Nov 03 02:50:42.343599 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:42.343504    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv" event={"ID":"80a44056-1c78-41e9-94b9-68746bbcbd2c","Type":"ContainerStarted","Data":"8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5"}
Nov 03 02:50:42.343599 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:42.343572    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"
Nov 03 02:50:42.455195 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:42.455136    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv" podStartSLOduration=2.174511771 podStartE2EDuration="4.455119334s" totalImagesPullingTime="2.280607563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:50:42.453485747 +0000 UTC m=+5521.879092221" watchObservedRunningTime="2026-11-03 02:50:42.455119334 +0000 UTC m=+5521.880725793"
Nov 03 02:50:43.824095 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:50:43.824051    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:50:47.021618 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:47.021578    5508 scope.go:122] "RemoveContainer" containerID="6abb34e55f916dea8b12bf14c663136a9ecd1e6f60d5816e057f47bdf7857164"
Nov 03 02:50:47.031054 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:47.031030    5508 scope.go:122] "RemoveContainer" containerID="293b370692b10639c8c882daac8fd58be5e45eae25a9ff0e6d5019f898b86a8d"
Nov 03 02:50:47.039792 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:47.039771    5508 scope.go:122] "RemoveContainer" containerID="861e315b6cfc9c79977d20853c2334d60c10d17539541a1e9941d1d5a4721792"
Nov 03 02:50:49.901799 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:50:49.901757    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:51:02.053667 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:51:02.053635    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"]
Nov 03 02:51:02.099704 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:51:02.099667    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a4c203d-5b6kf"]
Nov 03 02:51:02.690193 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:51:02.690159    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f31f72-2947-435f-ae60-d429a765edd3" path="/var/lib/kubelet/pods/99f31f72-2947-435f-ae60-d429a765edd3/volumes"
Nov 03 02:51:13.356721 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:51:13.356692    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"
Nov 03 02:51:13.810946 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:51:13.810910    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:51:15.045963 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:51:15.045928    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"]
Nov 03 02:51:15.092398 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:51:15.092340    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c142b96-gpkml"]
Nov 03 02:51:16.691386 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:51:16.691333    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95f2e42-accd-4be5-bcdb-aec368ad11a0" path="/var/lib/kubelet/pods/d95f2e42-accd-4be5-bcdb-aec368ad11a0/volumes"
Nov 03 02:51:21.698760 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:51:21.698723    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:51:43.811501 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:51:43.811453    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:51:46.286710 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:51:46.286628    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:51:46.367848 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:51:46.367813    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"]
Nov 03 02:51:47.069673 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:51:47.069644    5508 scope.go:122] "RemoveContainer" containerID="d0c50fe9a7dcaad48ae16952586b746ec9cf70e1bd28e86e4d65f61cb529deff"
Nov 03 02:51:53.356748 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:51:53.356709    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv" podUID="80a44056-1c78-41e9-94b9-68746bbcbd2c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:51:58.083636 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:51:58.083598    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:52:01.405735 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:01.405693    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv" podUID="80a44056-1c78-41e9-94b9-68746bbcbd2c" containerName="main" containerID="cri-o://8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5" gracePeriod=45
Nov 03 02:52:03.357617 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:03.357578    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv" podUID="80a44056-1c78-41e9-94b9-68746bbcbd2c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:52:13.357327 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:13.357283    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv" podUID="80a44056-1c78-41e9-94b9-68746bbcbd2c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:52:13.357829 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:13.357447    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"
Nov 03 02:52:13.811961 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:52:13.811924    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:52:21.354798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:21.354738    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:52:23.356826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:23.356789    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv" podUID="80a44056-1c78-41e9-94b9-68746bbcbd2c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 02:52:31.438435 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:52:31.438394    5508 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a44056_1c78_41e9_94b9_68746bbcbd2c.slice/crio-8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 02:52:31.650334 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:31.650307    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"
Nov 03 02:52:31.762312 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:31.762214    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a44056-1c78-41e9-94b9-68746bbcbd2c-kube-api-access-p52pd\" (UniqueName: \"kubernetes.io/projected/80a44056-1c78-41e9-94b9-68746bbcbd2c-kube-api-access-p52pd\") pod \"80a44056-1c78-41e9-94b9-68746bbcbd2c\" (UID: \"80a44056-1c78-41e9-94b9-68746bbcbd2c\") "
Nov 03 02:52:31.762312 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:31.762269    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80a44056-1c78-41e9-94b9-68746bbcbd2c-tls-certs\" (UniqueName: \"kubernetes.io/secret/80a44056-1c78-41e9-94b9-68746bbcbd2c-tls-certs\") pod \"80a44056-1c78-41e9-94b9-68746bbcbd2c\" (UID: \"80a44056-1c78-41e9-94b9-68746bbcbd2c\") "
Nov 03 02:52:31.765821 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:31.765785    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80a44056-1c78-41e9-94b9-68746bbcbd2c-tls-certs" pod "80a44056-1c78-41e9-94b9-68746bbcbd2c" (UID: "80a44056-1c78-41e9-94b9-68746bbcbd2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:52:31.766855 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:31.766825    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a44056-1c78-41e9-94b9-68746bbcbd2c-kube-api-access-p52pd" pod "80a44056-1c78-41e9-94b9-68746bbcbd2c" (UID: "80a44056-1c78-41e9-94b9-68746bbcbd2c"). InnerVolumeSpecName "kube-api-access-p52pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:52:31.864008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:31.863969    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p52pd\" (UniqueName: \"kubernetes.io/projected/80a44056-1c78-41e9-94b9-68746bbcbd2c-kube-api-access-p52pd\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:52:31.864008 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:31.864002    5508 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80a44056-1c78-41e9-94b9-68746bbcbd2c-tls-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:52:32.068697 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:32.068609    5508 generic.go:356] "Generic (PLEG): container finished" podID="80a44056-1c78-41e9-94b9-68746bbcbd2c" containerID="8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5" exitCode=0
Nov 03 02:52:32.068697 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:32.068663    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"
Nov 03 02:52:32.068890 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:32.068689    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv" event={"ID":"80a44056-1c78-41e9-94b9-68746bbcbd2c","Type":"ContainerDied","Data":"8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5"}
Nov 03 02:52:32.068890 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:32.068745    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv" event={"ID":"80a44056-1c78-41e9-94b9-68746bbcbd2c","Type":"ContainerDied","Data":"545df2ba3118c95351e3179ea6eac2ec6a44c0cc5c53e5df095e42d9f612169f"}
Nov 03 02:52:32.068890 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:32.068767    5508 scope.go:122] "RemoveContainer" containerID="8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5"
Nov 03 02:52:32.082439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:32.082414    5508 scope.go:122] "RemoveContainer" containerID="8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5"
Nov 03 02:52:32.082732 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:52:32.082709    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5\": container with ID starting with 8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5 not found: ID does not exist" containerID="8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5"
Nov 03 02:52:32.082790 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:32.082743    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5"} err="failed to get container status \"8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5\": rpc error: code = NotFound desc = could not find container \"8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5\": container with ID starting with 8a1716591ae6ab0ffa2d7d4217aa7abfdefbea21bc1992cfaee801f50e70e9a5 not found: ID does not exist"
Nov 03 02:52:32.222339 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:32.222305    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"]
Nov 03 02:52:32.268909 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:32.268873    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-1bfe99e1-kserve-router-scheduler-77c779cb96-swpsv"]
Nov 03 02:52:32.690936 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:32.690902    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a44056-1c78-41e9-94b9-68746bbcbd2c" path="/var/lib/kubelet/pods/80a44056-1c78-41e9-94b9-68746bbcbd2c/volumes"
Nov 03 02:52:43.757036 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:43.756986    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:52:43.834243 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:52:43.834202    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:52:47.094847 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:47.094814    5508 scope.go:122] "RemoveContainer" containerID="6dd7231c896dfad31837574fd4b95348795a85c133888ed1d5179b5dfa71c4a2"
Nov 03 02:52:47.103677 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:52:47.103654    5508 scope.go:122] "RemoveContainer" containerID="307b58b822c9feb13ecf028844f60459466c387ae8f9004d275c9d295092baad"
Nov 03 02:53:08.593911 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:08.593875    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:53:13.816565 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:53:13.816529    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:53:23.926224 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:23.926145    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"]
Nov 03 02:53:23.926683 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:23.926655    5508 cpu_manager.go:414] "containerMap: removing container" podUID="80a44056-1c78-41e9-94b9-68746bbcbd2c" containerName="main"
Nov 03 02:53:23.926683 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:23.926675    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a44056-1c78-41e9-94b9-68746bbcbd2c" containerName="main"
Nov 03 02:53:23.926822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:23.926809    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a44056-1c78-41e9-94b9-68746bbcbd2c" containerName="main"
Nov 03 02:53:23.930219 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:23.930201    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"
Nov 03 02:53:23.979371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:23.979310    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5abb6ab9-xvqx9\" (UID: \"ba6e7323-dc29-4919-a664-d8409d3e8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"
Nov 03 02:53:23.979537 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:23.979418    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5abb6ab9-xvqx9\" (UID: \"ba6e7323-dc29-4919-a664-d8409d3e8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"
Nov 03 02:53:23.979537 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:23.979453    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rg5w\" (UniqueName: \"kubernetes.io/projected/ba6e7323-dc29-4919-a664-d8409d3e8185-kube-api-access-2rg5w\") pod \"maas-api-revoke-sub-5abb6ab9-xvqx9\" (UID: \"ba6e7323-dc29-4919-a664-d8409d3e8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"
Nov 03 02:53:24.025344 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:24.025306    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:53:24.025344 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:24.025318    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:53:24.025631 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:24.025402    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:53:24.025631 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:24.025531    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 02:53:24.027849 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:24.027824    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"]
Nov 03 02:53:24.080048 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:24.080021    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5abb6ab9-xvqx9\" (UID: \"ba6e7323-dc29-4919-a664-d8409d3e8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"
Nov 03 02:53:24.080235 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:24.080058    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5abb6ab9-xvqx9\" (UID: \"ba6e7323-dc29-4919-a664-d8409d3e8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"
Nov 03 02:53:24.080235 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:24.080088    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rg5w\" (UniqueName: \"kubernetes.io/projected/ba6e7323-dc29-4919-a664-d8409d3e8185-kube-api-access-2rg5w\") pod \"maas-api-revoke-sub-5abb6ab9-xvqx9\" (UID: \"ba6e7323-dc29-4919-a664-d8409d3e8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"
Nov 03 02:53:24.080768 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:24.080743    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5abb6ab9-xvqx9\" (UID: \"ba6e7323-dc29-4919-a664-d8409d3e8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"
Nov 03 02:53:24.082815 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:24.082787    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5abb6ab9-xvqx9\" (UID: \"ba6e7323-dc29-4919-a664-d8409d3e8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"
Nov 03 02:53:24.138724 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:24.138693    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rg5w\" (UniqueName: \"kubernetes.io/projected/ba6e7323-dc29-4919-a664-d8409d3e8185-kube-api-access-2rg5w\") pod \"maas-api-revoke-sub-5abb6ab9-xvqx9\" (UID: \"ba6e7323-dc29-4919-a664-d8409d3e8185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"
Nov 03 02:53:24.242682 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:24.242588    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"
Nov 03 02:53:24.614635 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:24.614551    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"]
Nov 03 02:53:24.622325 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:53:24.622297    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6e7323_dc29_4919_a664_d8409d3e8185.slice/crio-e3955c07a1f6e184677018f8403dde605b23777113361ced0138f3f5e314c089 WatchSource:0}: Error finding container e3955c07a1f6e184677018f8403dde605b23777113361ced0138f3f5e314c089: Status 404 returned error can't find the container with id e3955c07a1f6e184677018f8403dde605b23777113361ced0138f3f5e314c089
Nov 03 02:53:25.339805 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:25.339758    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9" event={"ID":"ba6e7323-dc29-4919-a664-d8409d3e8185","Type":"ContainerStarted","Data":"82192e7ca440817efb1f092b77d22d35dcb5dce0cb334793001c047fee4ee899"}
Nov 03 02:53:25.340194 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:25.339821    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9" event={"ID":"ba6e7323-dc29-4919-a664-d8409d3e8185","Type":"ContainerStarted","Data":"e3955c07a1f6e184677018f8403dde605b23777113361ced0138f3f5e314c089"}
Nov 03 02:53:25.449761 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:25.449709    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9" podStartSLOduration=2.449693692 podStartE2EDuration="2.449693692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:53:25.448506385 +0000 UTC m=+5684.874112859" watchObservedRunningTime="2026-11-03 02:53:25.449693692 +0000 UTC m=+5684.875300155"
Nov 03 02:53:26.347040 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:26.347003    5508 generic.go:356] "Generic (PLEG): container finished" podID="ba6e7323-dc29-4919-a664-d8409d3e8185" containerID="82192e7ca440817efb1f092b77d22d35dcb5dce0cb334793001c047fee4ee899" exitCode=0
Nov 03 02:53:26.347601 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:26.347064    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9" event={"ID":"ba6e7323-dc29-4919-a664-d8409d3e8185","Type":"ContainerDied","Data":"82192e7ca440817efb1f092b77d22d35dcb5dce0cb334793001c047fee4ee899"}
Nov 03 02:53:27.491229 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:27.491205    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"
Nov 03 02:53:27.516205 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:27.516180    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6e7323-dc29-4919-a664-d8409d3e8185-kube-api-access-2rg5w\" (UniqueName: \"kubernetes.io/projected/ba6e7323-dc29-4919-a664-d8409d3e8185-kube-api-access-2rg5w\") pod \"ba6e7323-dc29-4919-a664-d8409d3e8185\" (UID: \"ba6e7323-dc29-4919-a664-d8409d3e8185\") "
Nov 03 02:53:27.516383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:27.516218    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-xks-ca\") pod \"ba6e7323-dc29-4919-a664-d8409d3e8185\" (UID: \"ba6e7323-dc29-4919-a664-d8409d3e8185\") "
Nov 03 02:53:27.516383 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:27.516268    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-openshift-ca\") pod \"ba6e7323-dc29-4919-a664-d8409d3e8185\" (UID: \"ba6e7323-dc29-4919-a664-d8409d3e8185\") "
Nov 03 02:53:27.516598 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:27.516577    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-openshift-ca" pod "ba6e7323-dc29-4919-a664-d8409d3e8185" (UID: "ba6e7323-dc29-4919-a664-d8409d3e8185"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:53:27.516819 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:27.516798    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:53:27.518275 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:27.518246    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-xks-ca" pod "ba6e7323-dc29-4919-a664-d8409d3e8185" (UID: "ba6e7323-dc29-4919-a664-d8409d3e8185"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:53:27.518402 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:27.518330    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6e7323-dc29-4919-a664-d8409d3e8185-kube-api-access-2rg5w" pod "ba6e7323-dc29-4919-a664-d8409d3e8185" (UID: "ba6e7323-dc29-4919-a664-d8409d3e8185"). InnerVolumeSpecName "kube-api-access-2rg5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:53:27.618361 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:27.618264    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rg5w\" (UniqueName: \"kubernetes.io/projected/ba6e7323-dc29-4919-a664-d8409d3e8185-kube-api-access-2rg5w\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:53:27.618361 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:27.618302    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6e7323-dc29-4919-a664-d8409d3e8185-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:53:28.359911 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:28.359882    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"
Nov 03 02:53:28.359911 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:28.359900    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9" event={"ID":"ba6e7323-dc29-4919-a664-d8409d3e8185","Type":"ContainerDied","Data":"e3955c07a1f6e184677018f8403dde605b23777113361ced0138f3f5e314c089"}
Nov 03 02:53:28.360183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:28.359957    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3955c07a1f6e184677018f8403dde605b23777113361ced0138f3f5e314c089"
Nov 03 02:53:32.047736 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:32.047697    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"]
Nov 03 02:53:32.093907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:32.093879    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-g87pl"]
Nov 03 02:53:32.691412 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:32.691376    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494fd1c5-6afe-4632-8064-6c000ec92389" path="/var/lib/kubelet/pods/494fd1c5-6afe-4632-8064-6c000ec92389/volumes"
Nov 03 02:53:36.880308 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:36.880271    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:53:41.750907 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:41.750881    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:53:41.876868 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:41.876843    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:53:43.811155 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:53:43.811119    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:53:56.062760 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:53:56.062726    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:54:13.811481 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:54:13.811444    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:54:19.051834 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:54:19.051793    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"]
Nov 03 02:54:19.096964 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:54:19.096931    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8tkgf"]
Nov 03 02:54:20.696138 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:54:20.696089    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa" path="/var/lib/kubelet/pods/5f64c47c-eb7b-4fbb-b6af-cd8b8a2aa5aa/volumes"
Nov 03 02:54:42.866670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:54:42.866633    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:54:43.824213 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:54:43.824164    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:54:47.145551 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:54:47.145473    5508 scope.go:122] "RemoveContainer" containerID="79a7e562f992275cce4c649e009aecff63ec08174e6814feff373001a015c5f5"
Nov 03 02:54:54.048366 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:54:54.048315    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"]
Nov 03 02:54:54.094847 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:54:54.094816    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cbwf5"]
Nov 03 02:54:54.690029 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:54:54.689994    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8" path="/var/lib/kubelet/pods/f19eb7fd-9a7c-48cc-ba64-bf4f8ca207c8/volumes"
Nov 03 02:55:09.658453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:09.658410    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:55:13.779373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:13.778508    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4jmtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:55:13.779373 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:13.778606    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" podUID="9d32d522-440d-42da-84ed-bb3b611a1a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:55:13.840372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:13.837903    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cnxmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:55:13.840372 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:13.837978    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podUID="f93283bb-8765-407f-86eb-af53fa0e399e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:55:13.936900 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:55:13.936794    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:55:14.044130 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:14.044039    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"]
Nov 03 02:55:14.054141 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:14.054110    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-5dnfw"]
Nov 03 02:55:14.690358 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:14.690312    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a134e67a-9610-491a-93d8-6ac45346a634" path="/var/lib/kubelet/pods/a134e67a-9610-491a-93d8-6ac45346a634/volumes"
Nov 03 02:55:26.046039 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:26.046007    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"]
Nov 03 02:55:26.092426 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:26.092390    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-58zxg"]
Nov 03 02:55:26.690508 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:26.690475    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b094c569-de76-4825-95ea-6d89ecffd3b9" path="/var/lib/kubelet/pods/b094c569-de76-4825-95ea-6d89ecffd3b9/volumes"
Nov 03 02:55:31.088311 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.088228    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"]
Nov 03 02:55:31.088718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.088605    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ba6e7323-dc29-4919-a664-d8409d3e8185" containerName="revoke-keys"
Nov 03 02:55:31.088718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.088616    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6e7323-dc29-4919-a664-d8409d3e8185" containerName="revoke-keys"
Nov 03 02:55:31.088718 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.088707    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6e7323-dc29-4919-a664-d8409d3e8185" containerName="revoke-keys"
Nov 03 02:55:31.091930 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.091906    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"
Nov 03 02:55:31.140674 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.140650    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:55:31.140674 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.140650    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:55:31.168416 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.168385    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7jzn\" (UniqueName: \"kubernetes.io/projected/db2648de-e241-4f6e-b94c-8a71583484d0-kube-api-access-n7jzn\") pod \"maas-api-revoke-sub-452b6d7b-ktzvn\" (UID: \"db2648de-e241-4f6e-b94c-8a71583484d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"
Nov 03 02:55:31.168571 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.168550    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-ktzvn\" (UID: \"db2648de-e241-4f6e-b94c-8a71583484d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"
Nov 03 02:55:31.168641 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.168616    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-ktzvn\" (UID: \"db2648de-e241-4f6e-b94c-8a71583484d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"
Nov 03 02:55:31.186924 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.186898    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 02:55:31.187037 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.187018    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:55:31.189063 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.189044    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"]
Nov 03 02:55:31.269800 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.269761    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-ktzvn\" (UID: \"db2648de-e241-4f6e-b94c-8a71583484d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"
Nov 03 02:55:31.270013 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.269819    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-ktzvn\" (UID: \"db2648de-e241-4f6e-b94c-8a71583484d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"
Nov 03 02:55:31.270013 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.269895    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7jzn\" (UniqueName: \"kubernetes.io/projected/db2648de-e241-4f6e-b94c-8a71583484d0-kube-api-access-n7jzn\") pod \"maas-api-revoke-sub-452b6d7b-ktzvn\" (UID: \"db2648de-e241-4f6e-b94c-8a71583484d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"
Nov 03 02:55:31.270527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.270501    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-ktzvn\" (UID: \"db2648de-e241-4f6e-b94c-8a71583484d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"
Nov 03 02:55:31.272151 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.272127    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-ktzvn\" (UID: \"db2648de-e241-4f6e-b94c-8a71583484d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"
Nov 03 02:55:31.327812 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.327778    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7jzn\" (UniqueName: \"kubernetes.io/projected/db2648de-e241-4f6e-b94c-8a71583484d0-kube-api-access-n7jzn\") pod \"maas-api-revoke-sub-452b6d7b-ktzvn\" (UID: \"db2648de-e241-4f6e-b94c-8a71583484d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"
Nov 03 02:55:31.404275 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.404093    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"
Nov 03 02:55:31.780837 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:31.780756    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"]
Nov 03 02:55:31.788663 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:55:31.788635    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2648de_e241_4f6e_b94c_8a71583484d0.slice/crio-e0f175f87652f13ce29f9969c05d4eaf38a131b1ef47ee746c2a1e073548c530 WatchSource:0}: Error finding container e0f175f87652f13ce29f9969c05d4eaf38a131b1ef47ee746c2a1e073548c530: Status 404 returned error can't find the container with id e0f175f87652f13ce29f9969c05d4eaf38a131b1ef47ee746c2a1e073548c530
Nov 03 02:55:32.071685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:32.071657    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn" event={"ID":"db2648de-e241-4f6e-b94c-8a71583484d0","Type":"ContainerStarted","Data":"46957e1f894174c4836e2baa0b981c060c97ec8b4bcc7c16be16acef51e40177"}
Nov 03 02:55:32.071829 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:32.071711    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn" event={"ID":"db2648de-e241-4f6e-b94c-8a71583484d0","Type":"ContainerStarted","Data":"e0f175f87652f13ce29f9969c05d4eaf38a131b1ef47ee746c2a1e073548c530"}
Nov 03 02:55:32.181639 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:32.181579    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn" podStartSLOduration=1.181564175 podStartE2EDuration="1.181564175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:55:32.180327015 +0000 UTC m=+5811.605933479" watchObservedRunningTime="2026-11-03 02:55:32.181564175 +0000 UTC m=+5811.607170639"
Nov 03 02:55:32.264361 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:32.264312    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:55:33.082584 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:33.082546    5508 generic.go:356] "Generic (PLEG): container finished" podID="db2648de-e241-4f6e-b94c-8a71583484d0" containerID="46957e1f894174c4836e2baa0b981c060c97ec8b4bcc7c16be16acef51e40177" exitCode=0
Nov 03 02:55:33.082946 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:33.082630    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn" event={"ID":"db2648de-e241-4f6e-b94c-8a71583484d0","Type":"ContainerDied","Data":"46957e1f894174c4836e2baa0b981c060c97ec8b4bcc7c16be16acef51e40177"}
Nov 03 02:55:34.889374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:34.888768    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"
Nov 03 02:55:35.005371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:35.004368    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-xks-ca\") pod \"db2648de-e241-4f6e-b94c-8a71583484d0\" (UID: \"db2648de-e241-4f6e-b94c-8a71583484d0\") "
Nov 03 02:55:35.005371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:35.004465    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2648de-e241-4f6e-b94c-8a71583484d0-kube-api-access-n7jzn\" (UniqueName: \"kubernetes.io/projected/db2648de-e241-4f6e-b94c-8a71583484d0-kube-api-access-n7jzn\") pod \"db2648de-e241-4f6e-b94c-8a71583484d0\" (UID: \"db2648de-e241-4f6e-b94c-8a71583484d0\") "
Nov 03 02:55:35.005371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:35.004513    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-openshift-ca\") pod \"db2648de-e241-4f6e-b94c-8a71583484d0\" (UID: \"db2648de-e241-4f6e-b94c-8a71583484d0\") "
Nov 03 02:55:35.005714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:35.005468    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-openshift-ca" pod "db2648de-e241-4f6e-b94c-8a71583484d0" (UID: "db2648de-e241-4f6e-b94c-8a71583484d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:55:35.005784 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:35.005710    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:55:35.006853 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:35.006810    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-xks-ca" pod "db2648de-e241-4f6e-b94c-8a71583484d0" (UID: "db2648de-e241-4f6e-b94c-8a71583484d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:55:35.024374 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:35.021571    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2648de-e241-4f6e-b94c-8a71583484d0-kube-api-access-n7jzn" pod "db2648de-e241-4f6e-b94c-8a71583484d0" (UID: "db2648de-e241-4f6e-b94c-8a71583484d0"). InnerVolumeSpecName "kube-api-access-n7jzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:55:35.116439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:35.115997    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7jzn\" (UniqueName: \"kubernetes.io/projected/db2648de-e241-4f6e-b94c-8a71583484d0-kube-api-access-n7jzn\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:55:35.116439 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:35.116042    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db2648de-e241-4f6e-b94c-8a71583484d0-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:55:35.149798 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:35.149760    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn" event={"ID":"db2648de-e241-4f6e-b94c-8a71583484d0","Type":"ContainerDied","Data":"e0f175f87652f13ce29f9969c05d4eaf38a131b1ef47ee746c2a1e073548c530"}
Nov 03 02:55:35.150276 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:35.150257    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0f175f87652f13ce29f9969c05d4eaf38a131b1ef47ee746c2a1e073548c530"
Nov 03 02:55:35.150453 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:35.150095    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"
Nov 03 02:55:43.811580 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:55:43.811541    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:55:47.167624 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:47.167589    5508 scope.go:122] "RemoveContainer" containerID="30c7d0901c18f56206d150259310864b7d36e7d486fc79b2939effebd55dc58b"
Nov 03 02:55:47.676497 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:47.676464    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:55:55.393156 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.393122    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"]
Nov 03 02:55:55.393590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.393535    5508 cpu_manager.go:414] "containerMap: removing container" podUID="db2648de-e241-4f6e-b94c-8a71583484d0" containerName="revoke-keys"
Nov 03 02:55:55.393590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.393548    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2648de-e241-4f6e-b94c-8a71583484d0" containerName="revoke-keys"
Nov 03 02:55:55.393669 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.393617    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2648de-e241-4f6e-b94c-8a71583484d0" containerName="revoke-keys"
Nov 03 02:55:55.396696 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.396675    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"
Nov 03 02:55:55.445936 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.445906    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:55:55.463419 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.463382    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-n2jhz\" (UID: \"c69de6aa-fb03-407d-b3af-f4ded6d56381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"
Nov 03 02:55:55.463584 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.463509    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-n2jhz\" (UID: \"c69de6aa-fb03-407d-b3af-f4ded6d56381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"
Nov 03 02:55:55.463707 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.463686    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64r49\" (UniqueName: \"kubernetes.io/projected/c69de6aa-fb03-407d-b3af-f4ded6d56381-kube-api-access-64r49\") pod \"maas-api-revoke-sub-3e31a506-n2jhz\" (UID: \"c69de6aa-fb03-407d-b3af-f4ded6d56381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"
Nov 03 02:55:55.492014 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.491981    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:55:55.492199 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.492021    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:55:55.492199 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.492010    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 02:55:55.494065 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.494046    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"]
Nov 03 02:55:55.564461 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.564419    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64r49\" (UniqueName: \"kubernetes.io/projected/c69de6aa-fb03-407d-b3af-f4ded6d56381-kube-api-access-64r49\") pod \"maas-api-revoke-sub-3e31a506-n2jhz\" (UID: \"c69de6aa-fb03-407d-b3af-f4ded6d56381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"
Nov 03 02:55:55.564680 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.564480    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-n2jhz\" (UID: \"c69de6aa-fb03-407d-b3af-f4ded6d56381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"
Nov 03 02:55:55.564680 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.564533    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-n2jhz\" (UID: \"c69de6aa-fb03-407d-b3af-f4ded6d56381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"
Nov 03 02:55:55.565231 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.565206    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-n2jhz\" (UID: \"c69de6aa-fb03-407d-b3af-f4ded6d56381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"
Nov 03 02:55:55.567084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.567062    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-n2jhz\" (UID: \"c69de6aa-fb03-407d-b3af-f4ded6d56381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"
Nov 03 02:55:55.622522 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.622492    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64r49\" (UniqueName: \"kubernetes.io/projected/c69de6aa-fb03-407d-b3af-f4ded6d56381-kube-api-access-64r49\") pod \"maas-api-revoke-sub-3e31a506-n2jhz\" (UID: \"c69de6aa-fb03-407d-b3af-f4ded6d56381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"
Nov 03 02:55:55.709627 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:55.709594    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"
Nov 03 02:55:56.091776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:56.091694    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"]
Nov 03 02:55:56.100253 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:55:56.100225    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69de6aa_fb03_407d_b3af_f4ded6d56381.slice/crio-c927b9021cb2eeae260f6070af7604fca535441d0c3f59ca35df2ee40d356e8b WatchSource:0}: Error finding container c927b9021cb2eeae260f6070af7604fca535441d0c3f59ca35df2ee40d356e8b: Status 404 returned error can't find the container with id c927b9021cb2eeae260f6070af7604fca535441d0c3f59ca35df2ee40d356e8b
Nov 03 02:55:56.300910 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:56.300874    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz" event={"ID":"c69de6aa-fb03-407d-b3af-f4ded6d56381","Type":"ContainerStarted","Data":"c927b9021cb2eeae260f6070af7604fca535441d0c3f59ca35df2ee40d356e8b"}
Nov 03 02:55:57.311063 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:57.310956    5508 generic.go:356] "Generic (PLEG): container finished" podID="c69de6aa-fb03-407d-b3af-f4ded6d56381" containerID="d2df673198d4042bbd000d640994e75fdaf37efa2c7aadfb1a27057315d79bef" exitCode=0
Nov 03 02:55:57.311712 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:57.311685    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz" event={"ID":"c69de6aa-fb03-407d-b3af-f4ded6d56381","Type":"ContainerDied","Data":"d2df673198d4042bbd000d640994e75fdaf37efa2c7aadfb1a27057315d79bef"}
Nov 03 02:55:58.948264 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:58.948174    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cnxmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:55:59.009384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:58.958378    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podUID="f93283bb-8765-407f-86eb-af53fa0e399e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:55:59.009384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:58.990472    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bh6ts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:55:59.009384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:58.990530    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" podUID="9b416926-1f08-4197-80e5-5c3c82166abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:55:59.009384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:58.991927    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4jmtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:55:59.009384 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:58.991991    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" podUID="9d32d522-440d-42da-84ed-bb3b611a1a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:55:59.313692 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:59.313661    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"
Nov 03 02:55:59.347648 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:59.347610    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz" event={"ID":"c69de6aa-fb03-407d-b3af-f4ded6d56381","Type":"ContainerDied","Data":"c927b9021cb2eeae260f6070af7604fca535441d0c3f59ca35df2ee40d356e8b"}
Nov 03 02:55:59.347949 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:59.347928    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c927b9021cb2eeae260f6070af7604fca535441d0c3f59ca35df2ee40d356e8b"
Nov 03 02:55:59.348139 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:59.348124    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"
Nov 03 02:55:59.402590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:59.402546    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-openshift-ca\") pod \"c69de6aa-fb03-407d-b3af-f4ded6d56381\" (UID: \"c69de6aa-fb03-407d-b3af-f4ded6d56381\") "
Nov 03 02:55:59.403630 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:59.403605    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69de6aa-fb03-407d-b3af-f4ded6d56381-kube-api-access-64r49\" (UniqueName: \"kubernetes.io/projected/c69de6aa-fb03-407d-b3af-f4ded6d56381-kube-api-access-64r49\") pod \"c69de6aa-fb03-407d-b3af-f4ded6d56381\" (UID: \"c69de6aa-fb03-407d-b3af-f4ded6d56381\") "
Nov 03 02:55:59.405096 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:59.403878    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-xks-ca\") pod \"c69de6aa-fb03-407d-b3af-f4ded6d56381\" (UID: \"c69de6aa-fb03-407d-b3af-f4ded6d56381\") "
Nov 03 02:55:59.405965 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:59.403306    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-openshift-ca" pod "c69de6aa-fb03-407d-b3af-f4ded6d56381" (UID: "c69de6aa-fb03-407d-b3af-f4ded6d56381"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:55:59.439534 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:59.439494    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-xks-ca" pod "c69de6aa-fb03-407d-b3af-f4ded6d56381" (UID: "c69de6aa-fb03-407d-b3af-f4ded6d56381"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:55:59.442567 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:59.442505    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69de6aa-fb03-407d-b3af-f4ded6d56381-kube-api-access-64r49" pod "c69de6aa-fb03-407d-b3af-f4ded6d56381" (UID: "c69de6aa-fb03-407d-b3af-f4ded6d56381"). InnerVolumeSpecName "kube-api-access-64r49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:55:59.509155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:59.509002    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:55:59.509155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:59.509043    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64r49\" (UniqueName: \"kubernetes.io/projected/c69de6aa-fb03-407d-b3af-f4ded6d56381-kube-api-access-64r49\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:55:59.509155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:55:59.509062    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69de6aa-fb03-407d-b3af-f4ded6d56381-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:56:03.375830 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:03.375797    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:56:11.947607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:11.947545    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xzjj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:56:11.948314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:11.948270    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9" podUID="1185a711-3994-4129-a641-b93e62e8effe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:56:13.811172 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:56:13.811139    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:56:14.796158 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:14.796120    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:56:20.571643 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.571572    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"]
Nov 03 02:56:20.572145 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.572004    5508 cpu_manager.go:414] "containerMap: removing container" podUID="c69de6aa-fb03-407d-b3af-f4ded6d56381" containerName="revoke-keys"
Nov 03 02:56:20.572145 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.572018    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69de6aa-fb03-407d-b3af-f4ded6d56381" containerName="revoke-keys"
Nov 03 02:56:20.572145 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.572074    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69de6aa-fb03-407d-b3af-f4ded6d56381" containerName="revoke-keys"
Nov 03 02:56:20.576496 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.576475    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"
Nov 03 02:56:20.625609 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.625579    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:56:20.625980 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.625956    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 02:56:20.625980 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.625956    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:56:20.626159 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.625985    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:56:20.674133 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.674094    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"]
Nov 03 02:56:20.708550 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.708519    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5954b227-lwlkp\" (UID: \"3a16c942-c711-4960-908d-9cd4f8793e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"
Nov 03 02:56:20.708731 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.708555    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5954b227-lwlkp\" (UID: \"3a16c942-c711-4960-908d-9cd4f8793e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"
Nov 03 02:56:20.708731 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.708665    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qrf6\" (UniqueName: \"kubernetes.io/projected/3a16c942-c711-4960-908d-9cd4f8793e3d-kube-api-access-5qrf6\") pod \"maas-api-revoke-sub-5954b227-lwlkp\" (UID: \"3a16c942-c711-4960-908d-9cd4f8793e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"
Nov 03 02:56:20.809586 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.809548    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5954b227-lwlkp\" (UID: \"3a16c942-c711-4960-908d-9cd4f8793e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"
Nov 03 02:56:20.809787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.809590    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5954b227-lwlkp\" (UID: \"3a16c942-c711-4960-908d-9cd4f8793e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"
Nov 03 02:56:20.809787 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.809676    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qrf6\" (UniqueName: \"kubernetes.io/projected/3a16c942-c711-4960-908d-9cd4f8793e3d-kube-api-access-5qrf6\") pod \"maas-api-revoke-sub-5954b227-lwlkp\" (UID: \"3a16c942-c711-4960-908d-9cd4f8793e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"
Nov 03 02:56:20.810341 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.810316    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5954b227-lwlkp\" (UID: \"3a16c942-c711-4960-908d-9cd4f8793e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"
Nov 03 02:56:20.812083 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.812053    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5954b227-lwlkp\" (UID: \"3a16c942-c711-4960-908d-9cd4f8793e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"
Nov 03 02:56:20.867565 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.867500    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qrf6\" (UniqueName: \"kubernetes.io/projected/3a16c942-c711-4960-908d-9cd4f8793e3d-kube-api-access-5qrf6\") pod \"maas-api-revoke-sub-5954b227-lwlkp\" (UID: \"3a16c942-c711-4960-908d-9cd4f8793e3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"
Nov 03 02:56:20.889377 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:20.889338    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"
Nov 03 02:56:21.280740 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:21.280709    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"]
Nov 03 02:56:21.287786 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:56:21.287757    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a16c942_c711_4960_908d_9cd4f8793e3d.slice/crio-842adc90ac77ab626f164dee198eb98f55fbb8738cd806b8822f0ed68ab9e462 WatchSource:0}: Error finding container 842adc90ac77ab626f164dee198eb98f55fbb8738cd806b8822f0ed68ab9e462: Status 404 returned error can't find the container with id 842adc90ac77ab626f164dee198eb98f55fbb8738cd806b8822f0ed68ab9e462
Nov 03 02:56:21.544079 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:21.543985    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp" event={"ID":"3a16c942-c711-4960-908d-9cd4f8793e3d","Type":"ContainerStarted","Data":"842adc90ac77ab626f164dee198eb98f55fbb8738cd806b8822f0ed68ab9e462"}
Nov 03 02:56:22.551051 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:22.551014    5508 generic.go:356] "Generic (PLEG): container finished" podID="3a16c942-c711-4960-908d-9cd4f8793e3d" containerID="a5d89a6cdae09bc9ff8cb69ea30da4284366559f5e54f9de1dedcb7f1ce3b32b" exitCode=0
Nov 03 02:56:22.551498 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:22.551100    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp" event={"ID":"3a16c942-c711-4960-908d-9cd4f8793e3d","Type":"ContainerDied","Data":"a5d89a6cdae09bc9ff8cb69ea30da4284366559f5e54f9de1dedcb7f1ce3b32b"}
Nov 03 02:56:23.698428 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:23.698398    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"
Nov 03 02:56:23.841277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:23.841180    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a16c942-c711-4960-908d-9cd4f8793e3d-kube-api-access-5qrf6\" (UniqueName: \"kubernetes.io/projected/3a16c942-c711-4960-908d-9cd4f8793e3d-kube-api-access-5qrf6\") pod \"3a16c942-c711-4960-908d-9cd4f8793e3d\" (UID: \"3a16c942-c711-4960-908d-9cd4f8793e3d\") "
Nov 03 02:56:23.841277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:23.841239    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-openshift-ca\") pod \"3a16c942-c711-4960-908d-9cd4f8793e3d\" (UID: \"3a16c942-c711-4960-908d-9cd4f8793e3d\") "
Nov 03 02:56:23.841277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:23.841271    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-xks-ca\") pod \"3a16c942-c711-4960-908d-9cd4f8793e3d\" (UID: \"3a16c942-c711-4960-908d-9cd4f8793e3d\") "
Nov 03 02:56:23.841730 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:23.841698    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-openshift-ca" pod "3a16c942-c711-4960-908d-9cd4f8793e3d" (UID: "3a16c942-c711-4960-908d-9cd4f8793e3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:56:23.843610 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:23.843568    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a16c942-c711-4960-908d-9cd4f8793e3d-kube-api-access-5qrf6" pod "3a16c942-c711-4960-908d-9cd4f8793e3d" (UID: "3a16c942-c711-4960-908d-9cd4f8793e3d"). InnerVolumeSpecName "kube-api-access-5qrf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:56:23.843723 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:23.843659    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-xks-ca" pod "3a16c942-c711-4960-908d-9cd4f8793e3d" (UID: "3a16c942-c711-4960-908d-9cd4f8793e3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:56:23.942544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:23.942509    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qrf6\" (UniqueName: \"kubernetes.io/projected/3a16c942-c711-4960-908d-9cd4f8793e3d-kube-api-access-5qrf6\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:56:23.942544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:23.942538    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:56:23.942544 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:23.942554    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a16c942-c711-4960-908d-9cd4f8793e3d-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:56:24.564521 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:24.564489    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"
Nov 03 02:56:24.564521 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:24.564503    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp" event={"ID":"3a16c942-c711-4960-908d-9cd4f8793e3d","Type":"ContainerDied","Data":"842adc90ac77ab626f164dee198eb98f55fbb8738cd806b8822f0ed68ab9e462"}
Nov 03 02:56:24.564766 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:24.564548    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="842adc90ac77ab626f164dee198eb98f55fbb8738cd806b8822f0ed68ab9e462"
Nov 03 02:56:33.377893 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:33.377858    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:56:43.811020 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:56:43.810980    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:56:47.190736 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:47.190707    5508 scope.go:122] "RemoveContainer" containerID="259b767b2c3963aee11027daeb8efcf0b509620dbfaf0c9c0ae97341d200b1a0"
Nov 03 02:56:47.199578 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:47.199560    5508 scope.go:122] "RemoveContainer" containerID="0681a5df0c261bc3702f7332dcb51852696d67418b506ad38c8b61f85333aeae"
Nov 03 02:56:47.207573 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:47.207548    5508 scope.go:122] "RemoveContainer" containerID="6ede20bd7f1591b951cff8c88b8aaeef6e860a24af7186ee663f2372e9dc371f"
Nov 03 02:56:52.369806 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.369764    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-d6685899d-n6m99"]
Nov 03 02:56:52.370450 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.370381    5508 cpu_manager.go:414] "containerMap: removing container" podUID="3a16c942-c711-4960-908d-9cd4f8793e3d" containerName="revoke-keys"
Nov 03 02:56:52.370450 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.370404    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a16c942-c711-4960-908d-9cd4f8793e3d" containerName="revoke-keys"
Nov 03 02:56:52.370593 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.370509    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a16c942-c711-4960-908d-9cd4f8793e3d" containerName="revoke-keys"
Nov 03 02:56:52.375154 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.375131    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-d6685899d-n6m99"
Nov 03 02:56:52.394289 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.394262    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swszf\" (UniqueName: \"kubernetes.io/projected/bde9ce17-c7a4-41a4-92ba-445d3270e82c-kube-api-access-swszf\") pod \"payload-processing-d6685899d-n6m99\" (UID: \"bde9ce17-c7a4-41a4-92ba-445d3270e82c\") " pod="openshift-ingress/payload-processing-d6685899d-n6m99"
Nov 03 02:56:52.394464 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.394325    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde9ce17-c7a4-41a4-92ba-445d3270e82c-plugins-config-volume\") pod \"payload-processing-d6685899d-n6m99\" (UID: \"bde9ce17-c7a4-41a4-92ba-445d3270e82c\") " pod="openshift-ingress/payload-processing-d6685899d-n6m99"
Nov 03 02:56:52.470651 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.470619    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8fd52\""
Nov 03 02:56:52.470651 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.470649    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 02:56:52.472743 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.472721    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-d6685899d-n6m99"]
Nov 03 02:56:52.494916 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.494888    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swszf\" (UniqueName: \"kubernetes.io/projected/bde9ce17-c7a4-41a4-92ba-445d3270e82c-kube-api-access-swszf\") pod \"payload-processing-d6685899d-n6m99\" (UID: \"bde9ce17-c7a4-41a4-92ba-445d3270e82c\") " pod="openshift-ingress/payload-processing-d6685899d-n6m99"
Nov 03 02:56:52.495081 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.494954    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde9ce17-c7a4-41a4-92ba-445d3270e82c-plugins-config-volume\") pod \"payload-processing-d6685899d-n6m99\" (UID: \"bde9ce17-c7a4-41a4-92ba-445d3270e82c\") " pod="openshift-ingress/payload-processing-d6685899d-n6m99"
Nov 03 02:56:52.495689 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.495668    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde9ce17-c7a4-41a4-92ba-445d3270e82c-plugins-config-volume\") pod \"payload-processing-d6685899d-n6m99\" (UID: \"bde9ce17-c7a4-41a4-92ba-445d3270e82c\") " pod="openshift-ingress/payload-processing-d6685899d-n6m99"
Nov 03 02:56:52.552697 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.552669    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swszf\" (UniqueName: \"kubernetes.io/projected/bde9ce17-c7a4-41a4-92ba-445d3270e82c-kube-api-access-swszf\") pod \"payload-processing-d6685899d-n6m99\" (UID: \"bde9ce17-c7a4-41a4-92ba-445d3270e82c\") " pod="openshift-ingress/payload-processing-d6685899d-n6m99"
Nov 03 02:56:52.692299 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:52.692257    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-d6685899d-n6m99"
Nov 03 02:56:53.062363 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:53.062266    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-d6685899d-n6m99"]
Nov 03 02:56:53.069653 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:56:53.069615    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde9ce17_c7a4_41a4_92ba_445d3270e82c.slice/crio-3b911e7e9c65ce4880d09afbf490250df05c6072e51eb7e7a9b7bc5c6be52616 WatchSource:0}: Error finding container 3b911e7e9c65ce4880d09afbf490250df05c6072e51eb7e7a9b7bc5c6be52616: Status 404 returned error can't find the container with id 3b911e7e9c65ce4880d09afbf490250df05c6072e51eb7e7a9b7bc5c6be52616
Nov 03 02:56:53.715012 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:53.714977    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-d6685899d-n6m99" event={"ID":"bde9ce17-c7a4-41a4-92ba-445d3270e82c","Type":"ContainerStarted","Data":"f61b5d99186231e3fddb7bc8e2557ef386526438a40a119a4dfdd4f2cf1d327e"}
Nov 03 02:56:53.715442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:53.715036    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-d6685899d-n6m99" event={"ID":"bde9ce17-c7a4-41a4-92ba-445d3270e82c","Type":"ContainerStarted","Data":"3b911e7e9c65ce4880d09afbf490250df05c6072e51eb7e7a9b7bc5c6be52616"}
Nov 03 02:56:53.715442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:53.715143    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-d6685899d-n6m99"
Nov 03 02:56:53.824806 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:56:53.824754    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-d6685899d-n6m99" podStartSLOduration=1.824738403 podStartE2EDuration="1.824738403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:56:53.823581129 +0000 UTC m=+5893.249187594" watchObservedRunningTime="2026-11-03 02:56:53.824738403 +0000 UTC m=+5893.250344867"
Nov 03 02:57:04.723084 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:04.723057    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-d6685899d-n6m99"
Nov 03 02:57:07.181562 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:07.181527    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74cf4f96d-l2rms"]
Nov 03 02:57:07.185091 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:07.185069    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74cf4f96d-l2rms"
Nov 03 02:57:07.225233 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:07.225197    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1978ac08-1656-4999-96cc-7f84264728c1-plugins-config-volume\") pod \"payload-pre-processing-74cf4f96d-l2rms\" (UID: \"1978ac08-1656-4999-96cc-7f84264728c1\") " pod="openshift-ingress/payload-pre-processing-74cf4f96d-l2rms"
Nov 03 02:57:07.225405 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:07.225326    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxxct\" (UniqueName: \"kubernetes.io/projected/1978ac08-1656-4999-96cc-7f84264728c1-kube-api-access-mxxct\") pod \"payload-pre-processing-74cf4f96d-l2rms\" (UID: \"1978ac08-1656-4999-96cc-7f84264728c1\") " pod="openshift-ingress/payload-pre-processing-74cf4f96d-l2rms"
Nov 03 02:57:07.243489 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:07.243461    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74cf4f96d-l2rms"]
Nov 03 02:57:07.326660 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:07.326621    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1978ac08-1656-4999-96cc-7f84264728c1-plugins-config-volume\") pod \"payload-pre-processing-74cf4f96d-l2rms\" (UID: \"1978ac08-1656-4999-96cc-7f84264728c1\") " pod="openshift-ingress/payload-pre-processing-74cf4f96d-l2rms"
Nov 03 02:57:07.326874 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:07.326719    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxxct\" (UniqueName: \"kubernetes.io/projected/1978ac08-1656-4999-96cc-7f84264728c1-kube-api-access-mxxct\") pod \"payload-pre-processing-74cf4f96d-l2rms\" (UID: \"1978ac08-1656-4999-96cc-7f84264728c1\") " pod="openshift-ingress/payload-pre-processing-74cf4f96d-l2rms"
Nov 03 02:57:07.327422 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:07.327398    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1978ac08-1656-4999-96cc-7f84264728c1-plugins-config-volume\") pod \"payload-pre-processing-74cf4f96d-l2rms\" (UID: \"1978ac08-1656-4999-96cc-7f84264728c1\") " pod="openshift-ingress/payload-pre-processing-74cf4f96d-l2rms"
Nov 03 02:57:07.384858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:07.384831    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxxct\" (UniqueName: \"kubernetes.io/projected/1978ac08-1656-4999-96cc-7f84264728c1-kube-api-access-mxxct\") pod \"payload-pre-processing-74cf4f96d-l2rms\" (UID: \"1978ac08-1656-4999-96cc-7f84264728c1\") " pod="openshift-ingress/payload-pre-processing-74cf4f96d-l2rms"
Nov 03 02:57:07.497568 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:07.497490    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74cf4f96d-l2rms"
Nov 03 02:57:07.867995 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:07.867918    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74cf4f96d-l2rms"]
Nov 03 02:57:07.875933 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:57:07.875905    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1978ac08_1656_4999_96cc_7f84264728c1.slice/crio-6fea4ac8056386a411909bc23ed63e40475be0e96f73eee6525ffdbd77033aff WatchSource:0}: Error finding container 6fea4ac8056386a411909bc23ed63e40475be0e96f73eee6525ffdbd77033aff: Status 404 returned error can't find the container with id 6fea4ac8056386a411909bc23ed63e40475be0e96f73eee6525ffdbd77033aff
Nov 03 02:57:08.798714 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:08.798680    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74cf4f96d-l2rms" event={"ID":"1978ac08-1656-4999-96cc-7f84264728c1","Type":"ContainerStarted","Data":"8cf007bf642b76aaba8f4199b8d6b028e23c6e567e7b000ecc6e95830c24a44b"}
Nov 03 02:57:08.799140 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:08.798730    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74cf4f96d-l2rms" event={"ID":"1978ac08-1656-4999-96cc-7f84264728c1","Type":"ContainerStarted","Data":"6fea4ac8056386a411909bc23ed63e40475be0e96f73eee6525ffdbd77033aff"}
Nov 03 02:57:08.799140 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:08.798771    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74cf4f96d-l2rms"
Nov 03 02:57:08.910976 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:08.910923    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74cf4f96d-l2rms" podStartSLOduration=1.9109090850000001 podStartE2EDuration="1.910909085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:57:08.908725502 +0000 UTC m=+5908.334331984" watchObservedRunningTime="2026-11-03 02:57:08.910909085 +0000 UTC m=+5908.336515548"
Nov 03 02:57:13.811908 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:57:13.811865    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:57:19.454785 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:19.454725    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-d6685899d-n6m99"]
Nov 03 02:57:19.807319 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:19.807238    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74cf4f96d-l2rms"
Nov 03 02:57:24.455482 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.455414    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-d6685899d-n6m99" podUID="bde9ce17-c7a4-41a4-92ba-445d3270e82c" containerName="payload-processing" containerID="cri-o://f61b5d99186231e3fddb7bc8e2557ef386526438a40a119a4dfdd4f2cf1d327e" gracePeriod=25
Nov 03 02:57:24.698139 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.698113    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-d6685899d-n6m99"
Nov 03 02:57:24.808771 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.808691    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bde9ce17-c7a4-41a4-92ba-445d3270e82c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde9ce17-c7a4-41a4-92ba-445d3270e82c-plugins-config-volume\") pod \"bde9ce17-c7a4-41a4-92ba-445d3270e82c\" (UID: \"bde9ce17-c7a4-41a4-92ba-445d3270e82c\") "
Nov 03 02:57:24.808929 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.808844    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde9ce17-c7a4-41a4-92ba-445d3270e82c-kube-api-access-swszf\" (UniqueName: \"kubernetes.io/projected/bde9ce17-c7a4-41a4-92ba-445d3270e82c-kube-api-access-swszf\") pod \"bde9ce17-c7a4-41a4-92ba-445d3270e82c\" (UID: \"bde9ce17-c7a4-41a4-92ba-445d3270e82c\") "
Nov 03 02:57:24.809090 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.809068    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde9ce17-c7a4-41a4-92ba-445d3270e82c-plugins-config-volume" pod "bde9ce17-c7a4-41a4-92ba-445d3270e82c" (UID: "bde9ce17-c7a4-41a4-92ba-445d3270e82c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:57:24.811387 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.811360    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde9ce17-c7a4-41a4-92ba-445d3270e82c-kube-api-access-swszf" pod "bde9ce17-c7a4-41a4-92ba-445d3270e82c" (UID: "bde9ce17-c7a4-41a4-92ba-445d3270e82c"). InnerVolumeSpecName "kube-api-access-swszf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:57:24.885155 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.885122    5508 generic.go:356] "Generic (PLEG): container finished" podID="bde9ce17-c7a4-41a4-92ba-445d3270e82c" containerID="f61b5d99186231e3fddb7bc8e2557ef386526438a40a119a4dfdd4f2cf1d327e" exitCode=0
Nov 03 02:57:24.885407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.885179    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-d6685899d-n6m99"
Nov 03 02:57:24.885407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.885185    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-d6685899d-n6m99" event={"ID":"bde9ce17-c7a4-41a4-92ba-445d3270e82c","Type":"ContainerDied","Data":"f61b5d99186231e3fddb7bc8e2557ef386526438a40a119a4dfdd4f2cf1d327e"}
Nov 03 02:57:24.885407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.885296    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-d6685899d-n6m99" event={"ID":"bde9ce17-c7a4-41a4-92ba-445d3270e82c","Type":"ContainerDied","Data":"3b911e7e9c65ce4880d09afbf490250df05c6072e51eb7e7a9b7bc5c6be52616"}
Nov 03 02:57:24.885407 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.885320    5508 scope.go:122] "RemoveContainer" containerID="f61b5d99186231e3fddb7bc8e2557ef386526438a40a119a4dfdd4f2cf1d327e"
Nov 03 02:57:24.894670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.894653    5508 scope.go:122] "RemoveContainer" containerID="f61b5d99186231e3fddb7bc8e2557ef386526438a40a119a4dfdd4f2cf1d327e"
Nov 03 02:57:24.894960 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:57:24.894934    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61b5d99186231e3fddb7bc8e2557ef386526438a40a119a4dfdd4f2cf1d327e\": container with ID starting with f61b5d99186231e3fddb7bc8e2557ef386526438a40a119a4dfdd4f2cf1d327e not found: ID does not exist" containerID="f61b5d99186231e3fddb7bc8e2557ef386526438a40a119a4dfdd4f2cf1d327e"
Nov 03 02:57:24.895027 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.894973    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61b5d99186231e3fddb7bc8e2557ef386526438a40a119a4dfdd4f2cf1d327e"} err="failed to get container status \"f61b5d99186231e3fddb7bc8e2557ef386526438a40a119a4dfdd4f2cf1d327e\": rpc error: code = NotFound desc = could not find container \"f61b5d99186231e3fddb7bc8e2557ef386526438a40a119a4dfdd4f2cf1d327e\": container with ID starting with f61b5d99186231e3fddb7bc8e2557ef386526438a40a119a4dfdd4f2cf1d327e not found: ID does not exist"
Nov 03 02:57:24.910572 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.910543    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swszf\" (UniqueName: \"kubernetes.io/projected/bde9ce17-c7a4-41a4-92ba-445d3270e82c-kube-api-access-swszf\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:57:24.910572 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:24.910567    5508 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bde9ce17-c7a4-41a4-92ba-445d3270e82c-plugins-config-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:57:25.040670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:25.040633    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-d6685899d-n6m99"]
Nov 03 02:57:25.046944 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:25.046915    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-d6685899d-n6m99"]
Nov 03 02:57:26.691520 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:26.691488    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde9ce17-c7a4-41a4-92ba-445d3270e82c" path="/var/lib/kubelet/pods/bde9ce17-c7a4-41a4-92ba-445d3270e82c/volumes"
Nov 03 02:57:27.286932 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:57:27.286896    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:57:43.810254 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:57:43.810214    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:58:13.810875 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:58:13.810835    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:58:28.047663 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:28.047623    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"]
Nov 03 02:58:28.094128 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:28.094098    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5abb6ab9-xvqx9"]
Nov 03 02:58:28.690359 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:28.690315    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6e7323-dc29-4919-a664-d8409d3e8185" path="/var/lib/kubelet/pods/ba6e7323-dc29-4919-a664-d8409d3e8185/volumes"
Nov 03 02:58:41.786765 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:41.786738    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:58:41.914161 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:41.914137    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 02:58:43.810140 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:58:43.810094    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:58:50.176807 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.176771    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"]
Nov 03 02:58:50.177337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.177170    5508 cpu_manager.go:414] "containerMap: removing container" podUID="bde9ce17-c7a4-41a4-92ba-445d3270e82c" containerName="payload-processing"
Nov 03 02:58:50.177337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.177180    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde9ce17-c7a4-41a4-92ba-445d3270e82c" containerName="payload-processing"
Nov 03 02:58:50.177337 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.177244    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde9ce17-c7a4-41a4-92ba-445d3270e82c" containerName="payload-processing"
Nov 03 02:58:50.180650 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.180628    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"
Nov 03 02:58:50.230376 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.230324    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:58:50.276719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.276675    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:58:50.276719 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.276685    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 02:58:50.276988 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.276687    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:58:50.278922 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.278899    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"]
Nov 03 02:58:50.287243 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.287223    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dg8j\" (UniqueName: \"kubernetes.io/projected/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-kube-api-access-2dg8j\") pod \"maas-api-revoke-sub-e2547622-86rh4\" (UID: \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"
Nov 03 02:58:50.287389 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.287282    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-86rh4\" (UID: \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"
Nov 03 02:58:50.287445 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.287386    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-86rh4\" (UID: \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"
Nov 03 02:58:50.388863 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.388825    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dg8j\" (UniqueName: \"kubernetes.io/projected/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-kube-api-access-2dg8j\") pod \"maas-api-revoke-sub-e2547622-86rh4\" (UID: \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"
Nov 03 02:58:50.389069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.388925    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-86rh4\" (UID: \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"
Nov 03 02:58:50.389069 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.388969    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-86rh4\" (UID: \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"
Nov 03 02:58:50.389706 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.389680    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-86rh4\" (UID: \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"
Nov 03 02:58:50.391342 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.391311    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-86rh4\" (UID: \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"
Nov 03 02:58:50.447052 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.446983    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dg8j\" (UniqueName: \"kubernetes.io/projected/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-kube-api-access-2dg8j\") pod \"maas-api-revoke-sub-e2547622-86rh4\" (UID: \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"
Nov 03 02:58:50.492956 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.492926    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"
Nov 03 02:58:50.861170 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:50.861084    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"]
Nov 03 02:58:50.868856 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:58:50.868827    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb0b90a_1693_4b5e_b42b_e4ee8189d179.slice/crio-6490fe57b7e041a249fd0aa23d72bf3855f91485c27bb72384377bfcdeaf1678 WatchSource:0}: Error finding container 6490fe57b7e041a249fd0aa23d72bf3855f91485c27bb72384377bfcdeaf1678: Status 404 returned error can't find the container with id 6490fe57b7e041a249fd0aa23d72bf3855f91485c27bb72384377bfcdeaf1678
Nov 03 02:58:51.334200 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:51.334146    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4" event={"ID":"6bb0b90a-1693-4b5e-b42b-e4ee8189d179","Type":"ContainerStarted","Data":"438b2f60b08093d3b09c4e37460f39817757cad677cb977998e42cbe0d05610b"}
Nov 03 02:58:51.334659 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:51.334253    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4" event={"ID":"6bb0b90a-1693-4b5e-b42b-e4ee8189d179","Type":"ContainerStarted","Data":"6490fe57b7e041a249fd0aa23d72bf3855f91485c27bb72384377bfcdeaf1678"}
Nov 03 02:58:51.443953 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:51.443895    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4" podStartSLOduration=1.4438830280000001 podStartE2EDuration="1.443883028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:58:51.443200164 +0000 UTC m=+6010.868806638" watchObservedRunningTime="2026-11-03 02:58:51.443883028 +0000 UTC m=+6010.869489492"
Nov 03 02:58:52.341164 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:52.341131    5508 generic.go:356] "Generic (PLEG): container finished" podID="6bb0b90a-1693-4b5e-b42b-e4ee8189d179" containerID="438b2f60b08093d3b09c4e37460f39817757cad677cb977998e42cbe0d05610b" exitCode=0
Nov 03 02:58:52.341663 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:52.341214    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4" event={"ID":"6bb0b90a-1693-4b5e-b42b-e4ee8189d179","Type":"ContainerDied","Data":"438b2f60b08093d3b09c4e37460f39817757cad677cb977998e42cbe0d05610b"}
Nov 03 02:58:53.476618 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:53.476594    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"
Nov 03 02:58:53.625276 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:53.625183    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-kube-api-access-2dg8j\" (UniqueName: \"kubernetes.io/projected/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-kube-api-access-2dg8j\") pod \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\" (UID: \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\") "
Nov 03 02:58:53.625276 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:53.625223    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-xks-ca\") pod \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\" (UID: \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\") "
Nov 03 02:58:53.625547 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:53.625397    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-openshift-ca\") pod \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\" (UID: \"6bb0b90a-1693-4b5e-b42b-e4ee8189d179\") "
Nov 03 02:58:53.625759 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:53.625733    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-openshift-ca" pod "6bb0b90a-1693-4b5e-b42b-e4ee8189d179" (UID: "6bb0b90a-1693-4b5e-b42b-e4ee8189d179"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:58:53.627314 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:53.627286    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-xks-ca" pod "6bb0b90a-1693-4b5e-b42b-e4ee8189d179" (UID: "6bb0b90a-1693-4b5e-b42b-e4ee8189d179"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:58:53.627433 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:53.627310    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-kube-api-access-2dg8j" pod "6bb0b90a-1693-4b5e-b42b-e4ee8189d179" (UID: "6bb0b90a-1693-4b5e-b42b-e4ee8189d179"). InnerVolumeSpecName "kube-api-access-2dg8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:58:53.726686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:53.726648    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:58:53.726686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:53.726679    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dg8j\" (UniqueName: \"kubernetes.io/projected/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-kube-api-access-2dg8j\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:58:53.726686 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:53.726690    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb0b90a-1693-4b5e-b42b-e4ee8189d179-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:58:54.351685 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:54.351656    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"
Nov 03 02:58:54.351883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:54.351662    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4" event={"ID":"6bb0b90a-1693-4b5e-b42b-e4ee8189d179","Type":"ContainerDied","Data":"6490fe57b7e041a249fd0aa23d72bf3855f91485c27bb72384377bfcdeaf1678"}
Nov 03 02:58:54.351883 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:58:54.351775    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6490fe57b7e041a249fd0aa23d72bf3855f91485c27bb72384377bfcdeaf1678"
Nov 03 02:59:04.800037 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:04.799993    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:59:13.811070 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:59:13.811024    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:59:15.504921 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:15.504891    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:59:21.192005 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.191971    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"]
Nov 03 02:59:21.192423 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.192328    5508 cpu_manager.go:414] "containerMap: removing container" podUID="6bb0b90a-1693-4b5e-b42b-e4ee8189d179" containerName="revoke-keys"
Nov 03 02:59:21.192423 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.192338    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb0b90a-1693-4b5e-b42b-e4ee8189d179" containerName="revoke-keys"
Nov 03 02:59:21.192502 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.192428    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb0b90a-1693-4b5e-b42b-e4ee8189d179" containerName="revoke-keys"
Nov 03 02:59:21.195725 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.195707    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"
Nov 03 02:59:21.244674 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.244645    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:59:21.290729 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.290675    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:59:21.290729 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.290692    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:59:21.290925 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.290832    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 02:59:21.293032 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.293010    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"]
Nov 03 02:59:21.356996 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.356968    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6s7x\" (UniqueName: \"kubernetes.io/projected/5806ea91-8e4d-48d0-96a4-75eaea67f942-kube-api-access-x6s7x\") pod \"maas-api-revoke-sub-c1e271a9-q27pt\" (UID: \"5806ea91-8e4d-48d0-96a4-75eaea67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"
Nov 03 02:59:21.357152 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.357092    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-q27pt\" (UID: \"5806ea91-8e4d-48d0-96a4-75eaea67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"
Nov 03 02:59:21.357198 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.357148    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-q27pt\" (UID: \"5806ea91-8e4d-48d0-96a4-75eaea67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"
Nov 03 02:59:21.458191 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.458103    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-q27pt\" (UID: \"5806ea91-8e4d-48d0-96a4-75eaea67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"
Nov 03 02:59:21.458191 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.458156    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-q27pt\" (UID: \"5806ea91-8e4d-48d0-96a4-75eaea67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"
Nov 03 02:59:21.458409 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.458224    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6s7x\" (UniqueName: \"kubernetes.io/projected/5806ea91-8e4d-48d0-96a4-75eaea67f942-kube-api-access-x6s7x\") pod \"maas-api-revoke-sub-c1e271a9-q27pt\" (UID: \"5806ea91-8e4d-48d0-96a4-75eaea67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"
Nov 03 02:59:21.458776 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.458752    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-q27pt\" (UID: \"5806ea91-8e4d-48d0-96a4-75eaea67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"
Nov 03 02:59:21.460607 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.460584    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-q27pt\" (UID: \"5806ea91-8e4d-48d0-96a4-75eaea67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"
Nov 03 02:59:21.515889 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.515860    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6s7x\" (UniqueName: \"kubernetes.io/projected/5806ea91-8e4d-48d0-96a4-75eaea67f942-kube-api-access-x6s7x\") pod \"maas-api-revoke-sub-c1e271a9-q27pt\" (UID: \"5806ea91-8e4d-48d0-96a4-75eaea67f942\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"
Nov 03 02:59:21.809273 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:21.809203    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"
Nov 03 02:59:22.179111 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:22.179086    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"]
Nov 03 02:59:22.186809 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:59:22.186780    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5806ea91_8e4d_48d0_96a4_75eaea67f942.slice/crio-aa8ddbd794208c5ff63fa432a4aa536cebab93a1aac38041364e3a6271f1cf21 WatchSource:0}: Error finding container aa8ddbd794208c5ff63fa432a4aa536cebab93a1aac38041364e3a6271f1cf21: Status 404 returned error can't find the container with id aa8ddbd794208c5ff63fa432a4aa536cebab93a1aac38041364e3a6271f1cf21
Nov 03 02:59:22.567367 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:22.567324    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt" event={"ID":"5806ea91-8e4d-48d0-96a4-75eaea67f942","Type":"ContainerStarted","Data":"08bd7ff2cefa930cf0cf8a0c67a26a11f4c2b95cbbea877fe526e28771ed07df"}
Nov 03 02:59:22.567754 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:22.567388    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt" event={"ID":"5806ea91-8e4d-48d0-96a4-75eaea67f942","Type":"ContainerStarted","Data":"aa8ddbd794208c5ff63fa432a4aa536cebab93a1aac38041364e3a6271f1cf21"}
Nov 03 02:59:22.678326 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:22.678276    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt" podStartSLOduration=1.6782635909999999 podStartE2EDuration="1.678263591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:59:22.676581403 +0000 UTC m=+6042.102187867" watchObservedRunningTime="2026-11-03 02:59:22.678263591 +0000 UTC m=+6042.103870054"
Nov 03 02:59:23.574277 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:23.574246    5508 generic.go:356] "Generic (PLEG): container finished" podID="5806ea91-8e4d-48d0-96a4-75eaea67f942" containerID="08bd7ff2cefa930cf0cf8a0c67a26a11f4c2b95cbbea877fe526e28771ed07df" exitCode=0
Nov 03 02:59:23.574711 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:23.574320    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt" event={"ID":"5806ea91-8e4d-48d0-96a4-75eaea67f942","Type":"ContainerDied","Data":"08bd7ff2cefa930cf0cf8a0c67a26a11f4c2b95cbbea877fe526e28771ed07df"}
Nov 03 02:59:24.716905 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:24.716882    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"
Nov 03 02:59:24.887748 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:24.887667    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5806ea91-8e4d-48d0-96a4-75eaea67f942-kube-api-access-x6s7x\" (UniqueName: \"kubernetes.io/projected/5806ea91-8e4d-48d0-96a4-75eaea67f942-kube-api-access-x6s7x\") pod \"5806ea91-8e4d-48d0-96a4-75eaea67f942\" (UID: \"5806ea91-8e4d-48d0-96a4-75eaea67f942\") "
Nov 03 02:59:24.887748 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:24.887733    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-openshift-ca\") pod \"5806ea91-8e4d-48d0-96a4-75eaea67f942\" (UID: \"5806ea91-8e4d-48d0-96a4-75eaea67f942\") "
Nov 03 02:59:24.887948 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:24.887807    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-xks-ca\") pod \"5806ea91-8e4d-48d0-96a4-75eaea67f942\" (UID: \"5806ea91-8e4d-48d0-96a4-75eaea67f942\") "
Nov 03 02:59:24.888082 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:24.888060    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-openshift-ca" pod "5806ea91-8e4d-48d0-96a4-75eaea67f942" (UID: "5806ea91-8e4d-48d0-96a4-75eaea67f942"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:59:24.889858 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:24.889829    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-xks-ca" pod "5806ea91-8e4d-48d0-96a4-75eaea67f942" (UID: "5806ea91-8e4d-48d0-96a4-75eaea67f942"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:59:24.889986 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:24.889888    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5806ea91-8e4d-48d0-96a4-75eaea67f942-kube-api-access-x6s7x" pod "5806ea91-8e4d-48d0-96a4-75eaea67f942" (UID: "5806ea91-8e4d-48d0-96a4-75eaea67f942"). InnerVolumeSpecName "kube-api-access-x6s7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:59:24.989333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:24.989301    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6s7x\" (UniqueName: \"kubernetes.io/projected/5806ea91-8e4d-48d0-96a4-75eaea67f942-kube-api-access-x6s7x\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:59:24.989333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:24.989328    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:59:24.989333 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:24.989338    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5806ea91-8e4d-48d0-96a4-75eaea67f942-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:59:25.585362 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:25.585325    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt" event={"ID":"5806ea91-8e4d-48d0-96a4-75eaea67f942","Type":"ContainerDied","Data":"aa8ddbd794208c5ff63fa432a4aa536cebab93a1aac38041364e3a6271f1cf21"}
Nov 03 02:59:25.585581 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:25.585403    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa8ddbd794208c5ff63fa432a4aa536cebab93a1aac38041364e3a6271f1cf21"
Nov 03 02:59:25.585581 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:25.585335    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"
Nov 03 02:59:34.102845 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:34.102807    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:59:41.949009 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:41.948465    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xzjj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 02:59:41.949009 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:41.948541    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9" podUID="1185a711-3994-4129-a641-b93e62e8effe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 02:59:43.811217 ip-10-0-140-255 kubenswrapper[5508]: E1103 02:59:43.811180    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 02:59:44.056790 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:44.056753    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:59:45.466670 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.466635    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"]
Nov 03 02:59:45.467100 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.467019    5508 cpu_manager.go:414] "containerMap: removing container" podUID="5806ea91-8e4d-48d0-96a4-75eaea67f942" containerName="revoke-keys"
Nov 03 02:59:45.467100 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.467033    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5806ea91-8e4d-48d0-96a4-75eaea67f942" containerName="revoke-keys"
Nov 03 02:59:45.467183 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.467111    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="5806ea91-8e4d-48d0-96a4-75eaea67f942" containerName="revoke-keys"
Nov 03 02:59:45.470413 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.470392    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"
Nov 03 02:59:45.519289 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.519259    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:59:45.519289 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.519284    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:59:45.533074 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.533037    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-l2z59\" (UID: \"4b355e0b-03c5-4184-aa4f-930730894895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"
Nov 03 02:59:45.533266 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.533212    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfm6j\" (UniqueName: \"kubernetes.io/projected/4b355e0b-03c5-4184-aa4f-930730894895-kube-api-access-mfm6j\") pod \"maas-api-revoke-sub-2e1f5b5f-l2z59\" (UID: \"4b355e0b-03c5-4184-aa4f-930730894895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"
Nov 03 02:59:45.533336 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.533270    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-l2z59\" (UID: \"4b355e0b-03c5-4184-aa4f-930730894895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"
Nov 03 02:59:45.565423 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.565386    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:59:45.565590 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.565452    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 02:59:45.567967 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.567944    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"]
Nov 03 02:59:45.634169 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.634137    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-l2z59\" (UID: \"4b355e0b-03c5-4184-aa4f-930730894895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"
Nov 03 02:59:45.634371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.634213    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfm6j\" (UniqueName: \"kubernetes.io/projected/4b355e0b-03c5-4184-aa4f-930730894895-kube-api-access-mfm6j\") pod \"maas-api-revoke-sub-2e1f5b5f-l2z59\" (UID: \"4b355e0b-03c5-4184-aa4f-930730894895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"
Nov 03 02:59:45.634371 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.634242    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-l2z59\" (UID: \"4b355e0b-03c5-4184-aa4f-930730894895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"
Nov 03 02:59:45.634927 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.634907    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-l2z59\" (UID: \"4b355e0b-03c5-4184-aa4f-930730894895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"
Nov 03 02:59:45.636527 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.636510    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-l2z59\" (UID: \"4b355e0b-03c5-4184-aa4f-930730894895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"
Nov 03 02:59:45.692129 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.692095    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfm6j\" (UniqueName: \"kubernetes.io/projected/4b355e0b-03c5-4184-aa4f-930730894895-kube-api-access-mfm6j\") pod \"maas-api-revoke-sub-2e1f5b5f-l2z59\" (UID: \"4b355e0b-03c5-4184-aa4f-930730894895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"
Nov 03 02:59:45.782826 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:45.782748    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"
Nov 03 02:59:46.157060 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:46.157027    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"]
Nov 03 02:59:46.165206 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:59:46.165175    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b355e0b_03c5_4184_aa4f_930730894895.slice/crio-7bbf85f0756aa5990de7d994ffea107eacab791c5c5d1064d2f18b79575ea7c3 WatchSource:0}: Error finding container 7bbf85f0756aa5990de7d994ffea107eacab791c5c5d1064d2f18b79575ea7c3: Status 404 returned error can't find the container with id 7bbf85f0756aa5990de7d994ffea107eacab791c5c5d1064d2f18b79575ea7c3
Nov 03 02:59:46.739818 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:46.739729    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59" event={"ID":"4b355e0b-03c5-4184-aa4f-930730894895","Type":"ContainerStarted","Data":"536f8e466849bd301390fce246eb612f0d167f4041a08e7e765c7e22cfc464ab"}
Nov 03 02:59:46.739818 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:46.739781    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59" event={"ID":"4b355e0b-03c5-4184-aa4f-930730894895","Type":"ContainerStarted","Data":"7bbf85f0756aa5990de7d994ffea107eacab791c5c5d1064d2f18b79575ea7c3"}
Nov 03 02:59:46.850072 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:46.850018    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59" podStartSLOduration=1.850004016 podStartE2EDuration="1.850004016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:59:46.847808441 +0000 UTC m=+6066.273414915" watchObservedRunningTime="2026-11-03 02:59:46.850004016 +0000 UTC m=+6066.275610532"
Nov 03 02:59:47.265334 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:47.265303    5508 scope.go:122] "RemoveContainer" containerID="82192e7ca440817efb1f092b77d22d35dcb5dce0cb334793001c047fee4ee899"
Nov 03 02:59:47.746304 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:47.746262    5508 generic.go:356] "Generic (PLEG): container finished" podID="4b355e0b-03c5-4184-aa4f-930730894895" containerID="536f8e466849bd301390fce246eb612f0d167f4041a08e7e765c7e22cfc464ab" exitCode=0
Nov 03 02:59:47.746762 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:47.746361    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59" event={"ID":"4b355e0b-03c5-4184-aa4f-930730894895","Type":"ContainerDied","Data":"536f8e466849bd301390fce246eb612f0d167f4041a08e7e765c7e22cfc464ab"}
Nov 03 02:59:48.890332 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:48.890299    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"
Nov 03 02:59:48.968725 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:48.968694    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b355e0b-03c5-4184-aa4f-930730894895-kube-api-access-mfm6j\" (UniqueName: \"kubernetes.io/projected/4b355e0b-03c5-4184-aa4f-930730894895-kube-api-access-mfm6j\") pod \"4b355e0b-03c5-4184-aa4f-930730894895\" (UID: \"4b355e0b-03c5-4184-aa4f-930730894895\") "
Nov 03 02:59:48.968914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:48.968741    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-openshift-ca\") pod \"4b355e0b-03c5-4184-aa4f-930730894895\" (UID: \"4b355e0b-03c5-4184-aa4f-930730894895\") "
Nov 03 02:59:48.968914 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:48.968809    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-xks-ca\") pod \"4b355e0b-03c5-4184-aa4f-930730894895\" (UID: \"4b355e0b-03c5-4184-aa4f-930730894895\") "
Nov 03 02:59:48.969158 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:48.969127    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-openshift-ca" pod "4b355e0b-03c5-4184-aa4f-930730894895" (UID: "4b355e0b-03c5-4184-aa4f-930730894895"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 02:59:48.970908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:48.970881    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-xks-ca" pod "4b355e0b-03c5-4184-aa4f-930730894895" (UID: "4b355e0b-03c5-4184-aa4f-930730894895"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 02:59:48.971056 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:48.971031    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b355e0b-03c5-4184-aa4f-930730894895-kube-api-access-mfm6j" pod "4b355e0b-03c5-4184-aa4f-930730894895" (UID: "4b355e0b-03c5-4184-aa4f-930730894895"). InnerVolumeSpecName "kube-api-access-mfm6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 02:59:49.069822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:49.069739    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfm6j\" (UniqueName: \"kubernetes.io/projected/4b355e0b-03c5-4184-aa4f-930730894895-kube-api-access-mfm6j\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:59:49.069822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:49.069770    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:59:49.069822 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:49.069781    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b355e0b-03c5-4184-aa4f-930730894895-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 02:59:49.757879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:49.757850    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"
Nov 03 02:59:49.757879 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:49.757873    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59" event={"ID":"4b355e0b-03c5-4184-aa4f-930730894895","Type":"ContainerDied","Data":"7bbf85f0756aa5990de7d994ffea107eacab791c5c5d1064d2f18b79575ea7c3"}
Nov 03 02:59:49.758110 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:49.757917    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bbf85f0756aa5990de7d994ffea107eacab791c5c5d1064d2f18b79575ea7c3"
Nov 03 02:59:58.070476 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.070440    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"]
Nov 03 02:59:58.071057 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.071035    5508 cpu_manager.go:414] "containerMap: removing container" podUID="4b355e0b-03c5-4184-aa4f-930730894895" containerName="revoke-keys"
Nov 03 02:59:58.071143 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.071061    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b355e0b-03c5-4184-aa4f-930730894895" containerName="revoke-keys"
Nov 03 02:59:58.071205 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.071179    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b355e0b-03c5-4184-aa4f-930730894895" containerName="revoke-keys"
Nov 03 02:59:58.074593 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.074563    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"
Nov 03 02:59:58.123481 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.123447    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 02:59:58.169602 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.169567    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 02:59:58.169602 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.169600    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 02:59:58.169908 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.169735    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 02:59:58.171982 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.171953    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"]
Nov 03 02:59:58.261648 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.261620    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4642113-07f4-426e-877d-f0443a206990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jgdrd\" (UID: \"b4642113-07f4-426e-877d-f0443a206990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"
Nov 03 02:59:58.261869 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.261672    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4642113-07f4-426e-877d-f0443a206990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jgdrd\" (UID: \"b4642113-07f4-426e-877d-f0443a206990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"
Nov 03 02:59:58.261869 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.261827    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7zhs\" (UniqueName: \"kubernetes.io/projected/b4642113-07f4-426e-877d-f0443a206990-kube-api-access-h7zhs\") pod \"maas-api-revoke-sub-ec843018-jgdrd\" (UID: \"b4642113-07f4-426e-877d-f0443a206990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"
Nov 03 02:59:58.303648 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.303607    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 02:59:58.362871 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.362782    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4642113-07f4-426e-877d-f0443a206990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jgdrd\" (UID: \"b4642113-07f4-426e-877d-f0443a206990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"
Nov 03 02:59:58.363062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.362890    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7zhs\" (UniqueName: \"kubernetes.io/projected/b4642113-07f4-426e-877d-f0443a206990-kube-api-access-h7zhs\") pod \"maas-api-revoke-sub-ec843018-jgdrd\" (UID: \"b4642113-07f4-426e-877d-f0443a206990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"
Nov 03 02:59:58.363062 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.362978    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4642113-07f4-426e-877d-f0443a206990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jgdrd\" (UID: \"b4642113-07f4-426e-877d-f0443a206990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"
Nov 03 02:59:58.363437 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.363413    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4642113-07f4-426e-877d-f0443a206990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-jgdrd\" (UID: \"b4642113-07f4-426e-877d-f0443a206990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"
Nov 03 02:59:58.365319 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.365298    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4642113-07f4-426e-877d-f0443a206990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-jgdrd\" (UID: \"b4642113-07f4-426e-877d-f0443a206990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"
Nov 03 02:59:58.420772 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.420735    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7zhs\" (UniqueName: \"kubernetes.io/projected/b4642113-07f4-426e-877d-f0443a206990-kube-api-access-h7zhs\") pod \"maas-api-revoke-sub-ec843018-jgdrd\" (UID: \"b4642113-07f4-426e-877d-f0443a206990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"
Nov 03 02:59:58.687442 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:58.687407    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"
Nov 03 02:59:59.070556 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:59.070473    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"]
Nov 03 02:59:59.077807 ip-10-0-140-255 kubenswrapper[5508]: W1103 02:59:59.077775    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4642113_07f4_426e_877d_f0443a206990.slice/crio-619082bd92b54172f843258155a90be2abd662daf6a4d46e04f19152a6e1f27d WatchSource:0}: Error finding container 619082bd92b54172f843258155a90be2abd662daf6a4d46e04f19152a6e1f27d: Status 404 returned error can't find the container with id 619082bd92b54172f843258155a90be2abd662daf6a4d46e04f19152a6e1f27d
Nov 03 02:59:59.809722 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:59.809683    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd" event={"ID":"b4642113-07f4-426e-877d-f0443a206990","Type":"ContainerStarted","Data":"33dc06b5cc1145e7ad95b5588250295ee5c9ef8af4bc0c5d608c27b9e3b4b1d4"}
Nov 03 02:59:59.809924 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:59.809744    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd" event={"ID":"b4642113-07f4-426e-877d-f0443a206990","Type":"ContainerStarted","Data":"619082bd92b54172f843258155a90be2abd662daf6a4d46e04f19152a6e1f27d"}
Nov 03 02:59:59.919251 ip-10-0-140-255 kubenswrapper[5508]: I1103 02:59:59.919195    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd" podStartSLOduration=1.919179941 podStartE2EDuration="1.919179941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 02:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 02:59:59.917515671 +0000 UTC m=+6079.343122135" watchObservedRunningTime="2026-11-03 02:59:59.919179941 +0000 UTC m=+6079.344786405"
Nov 03 03:00:00.815958 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:00.815924    5508 generic.go:356] "Generic (PLEG): container finished" podID="b4642113-07f4-426e-877d-f0443a206990" containerID="33dc06b5cc1145e7ad95b5588250295ee5c9ef8af4bc0c5d608c27b9e3b4b1d4" exitCode=0
Nov 03 03:00:00.816379 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:00.816015    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd" event={"ID":"b4642113-07f4-426e-877d-f0443a206990","Type":"ContainerDied","Data":"33dc06b5cc1145e7ad95b5588250295ee5c9ef8af4bc0c5d608c27b9e3b4b1d4"}
Nov 03 03:00:01.956400 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:01.956377    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"
Nov 03 03:00:02.100764 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:02.100678    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4642113-07f4-426e-877d-f0443a206990-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4642113-07f4-426e-877d-f0443a206990-maas-api-xks-ca\") pod \"b4642113-07f4-426e-877d-f0443a206990\" (UID: \"b4642113-07f4-426e-877d-f0443a206990\") "
Nov 03 03:00:02.100764 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:02.100736    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4642113-07f4-426e-877d-f0443a206990-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4642113-07f4-426e-877d-f0443a206990-maas-api-openshift-ca\") pod \"b4642113-07f4-426e-877d-f0443a206990\" (UID: \"b4642113-07f4-426e-877d-f0443a206990\") "
Nov 03 03:00:02.101018 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:02.100777    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4642113-07f4-426e-877d-f0443a206990-kube-api-access-h7zhs\" (UniqueName: \"kubernetes.io/projected/b4642113-07f4-426e-877d-f0443a206990-kube-api-access-h7zhs\") pod \"b4642113-07f4-426e-877d-f0443a206990\" (UID: \"b4642113-07f4-426e-877d-f0443a206990\") "
Nov 03 03:00:02.101173 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:02.101145    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4642113-07f4-426e-877d-f0443a206990-maas-api-openshift-ca" pod "b4642113-07f4-426e-877d-f0443a206990" (UID: "b4642113-07f4-426e-877d-f0443a206990"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:00:02.102834 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:02.102804    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4642113-07f4-426e-877d-f0443a206990-maas-api-xks-ca" pod "b4642113-07f4-426e-877d-f0443a206990" (UID: "b4642113-07f4-426e-877d-f0443a206990"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:00:02.102946 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:02.102925    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4642113-07f4-426e-877d-f0443a206990-kube-api-access-h7zhs" pod "b4642113-07f4-426e-877d-f0443a206990" (UID: "b4642113-07f4-426e-877d-f0443a206990"). InnerVolumeSpecName "kube-api-access-h7zhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:00:02.201982 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:02.201939    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4642113-07f4-426e-877d-f0443a206990-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:02.201982 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:02.201978    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4642113-07f4-426e-877d-f0443a206990-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:02.201982 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:02.201988    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7zhs\" (UniqueName: \"kubernetes.io/projected/b4642113-07f4-426e-877d-f0443a206990-kube-api-access-h7zhs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:02.830686 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:02.830643    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd" event={"ID":"b4642113-07f4-426e-877d-f0443a206990","Type":"ContainerDied","Data":"619082bd92b54172f843258155a90be2abd662daf6a4d46e04f19152a6e1f27d"}
Nov 03 03:00:02.830686 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:02.830666    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"
Nov 03 03:00:02.830900 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:02.830718    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="619082bd92b54172f843258155a90be2abd662daf6a4d46e04f19152a6e1f27d"
Nov 03 03:00:13.811188 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:00:13.811154    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:00:15.400534 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:15.400493    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:00:33.280212 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.280177    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"]
Nov 03 03:00:33.280693 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.280582    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b4642113-07f4-426e-877d-f0443a206990" containerName="revoke-keys"
Nov 03 03:00:33.280693 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.280594    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4642113-07f4-426e-877d-f0443a206990" containerName="revoke-keys"
Nov 03 03:00:33.280693 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.280678    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4642113-07f4-426e-877d-f0443a206990" containerName="revoke-keys"
Nov 03 03:00:33.283648 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.283618    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"
Nov 03 03:00:33.334613 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.334582    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:00:33.371189 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.371152    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9nhdg\" (UID: \"b66c3d24-bb50-4e81-a2be-1da246e203d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"
Nov 03 03:00:33.371392 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.371216    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2vk6\" (UniqueName: \"kubernetes.io/projected/b66c3d24-bb50-4e81-a2be-1da246e203d7-kube-api-access-j2vk6\") pod \"maas-api-revoke-sub-046e1343-9nhdg\" (UID: \"b66c3d24-bb50-4e81-a2be-1da246e203d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"
Nov 03 03:00:33.371392 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.371331    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9nhdg\" (UID: \"b66c3d24-bb50-4e81-a2be-1da246e203d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"
Nov 03 03:00:33.380740 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.380713    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:00:33.380892 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.380791    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:00:33.380964 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.380876    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:00:33.383747 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.383726    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"]
Nov 03 03:00:33.472430 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.472398    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9nhdg\" (UID: \"b66c3d24-bb50-4e81-a2be-1da246e203d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"
Nov 03 03:00:33.472595 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.472456    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2vk6\" (UniqueName: \"kubernetes.io/projected/b66c3d24-bb50-4e81-a2be-1da246e203d7-kube-api-access-j2vk6\") pod \"maas-api-revoke-sub-046e1343-9nhdg\" (UID: \"b66c3d24-bb50-4e81-a2be-1da246e203d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"
Nov 03 03:00:33.472595 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.472508    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9nhdg\" (UID: \"b66c3d24-bb50-4e81-a2be-1da246e203d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"
Nov 03 03:00:33.473127 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.473104    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9nhdg\" (UID: \"b66c3d24-bb50-4e81-a2be-1da246e203d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"
Nov 03 03:00:33.474865 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.474847    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9nhdg\" (UID: \"b66c3d24-bb50-4e81-a2be-1da246e203d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"
Nov 03 03:00:33.531060 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.530993    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2vk6\" (UniqueName: \"kubernetes.io/projected/b66c3d24-bb50-4e81-a2be-1da246e203d7-kube-api-access-j2vk6\") pod \"maas-api-revoke-sub-046e1343-9nhdg\" (UID: \"b66c3d24-bb50-4e81-a2be-1da246e203d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"
Nov 03 03:00:33.596112 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.596078    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"
Nov 03 03:00:33.965197 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:33.965168    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"]
Nov 03 03:00:33.972572 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:00:33.972547    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66c3d24_bb50_4e81_a2be_1da246e203d7.slice/crio-d23551a61499a8646152cc3441d8a10e529c25273b5e8b3a5ea9cdb549ba67c7 WatchSource:0}: Error finding container d23551a61499a8646152cc3441d8a10e529c25273b5e8b3a5ea9cdb549ba67c7: Status 404 returned error can't find the container with id d23551a61499a8646152cc3441d8a10e529c25273b5e8b3a5ea9cdb549ba67c7
Nov 03 03:00:34.049854 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:34.049811    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg" event={"ID":"b66c3d24-bb50-4e81-a2be-1da246e203d7","Type":"ContainerStarted","Data":"d23551a61499a8646152cc3441d8a10e529c25273b5e8b3a5ea9cdb549ba67c7"}
Nov 03 03:00:35.046403 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:35.046302    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"]
Nov 03 03:00:35.056849 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:35.056817    5508 generic.go:356] "Generic (PLEG): container finished" podID="b66c3d24-bb50-4e81-a2be-1da246e203d7" containerID="2b8111df5e9b793cb39b6aaa059e41de5e7809730a621e4ffb51a7cae7c975ee" exitCode=0
Nov 03 03:00:35.057005 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:35.056854    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg" event={"ID":"b66c3d24-bb50-4e81-a2be-1da246e203d7","Type":"ContainerDied","Data":"2b8111df5e9b793cb39b6aaa059e41de5e7809730a621e4ffb51a7cae7c975ee"}
Nov 03 03:00:35.092625 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:35.092597    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ktzvn"]
Nov 03 03:00:36.209607 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:36.209582    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"
Nov 03 03:00:36.303617 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:36.303583    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-openshift-ca\") pod \"b66c3d24-bb50-4e81-a2be-1da246e203d7\" (UID: \"b66c3d24-bb50-4e81-a2be-1da246e203d7\") "
Nov 03 03:00:36.303617 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:36.303624    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-xks-ca\") pod \"b66c3d24-bb50-4e81-a2be-1da246e203d7\" (UID: \"b66c3d24-bb50-4e81-a2be-1da246e203d7\") "
Nov 03 03:00:36.303873 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:36.303679    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66c3d24-bb50-4e81-a2be-1da246e203d7-kube-api-access-j2vk6\" (UniqueName: \"kubernetes.io/projected/b66c3d24-bb50-4e81-a2be-1da246e203d7-kube-api-access-j2vk6\") pod \"b66c3d24-bb50-4e81-a2be-1da246e203d7\" (UID: \"b66c3d24-bb50-4e81-a2be-1da246e203d7\") "
Nov 03 03:00:36.303985 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:36.303962    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-openshift-ca" pod "b66c3d24-bb50-4e81-a2be-1da246e203d7" (UID: "b66c3d24-bb50-4e81-a2be-1da246e203d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:00:36.305736 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:36.305710    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66c3d24-bb50-4e81-a2be-1da246e203d7-kube-api-access-j2vk6" pod "b66c3d24-bb50-4e81-a2be-1da246e203d7" (UID: "b66c3d24-bb50-4e81-a2be-1da246e203d7"). InnerVolumeSpecName "kube-api-access-j2vk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:00:36.305831 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:36.305774    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-xks-ca" pod "b66c3d24-bb50-4e81-a2be-1da246e203d7" (UID: "b66c3d24-bb50-4e81-a2be-1da246e203d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:00:36.405479 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:36.405445    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:36.405479 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:36.405481    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66c3d24-bb50-4e81-a2be-1da246e203d7-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:36.405694 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:36.405491    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2vk6\" (UniqueName: \"kubernetes.io/projected/b66c3d24-bb50-4e81-a2be-1da246e203d7-kube-api-access-j2vk6\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:36.690602 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:36.690519    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db2648de-e241-4f6e-b94c-8a71583484d0" path="/var/lib/kubelet/pods/db2648de-e241-4f6e-b94c-8a71583484d0/volumes"
Nov 03 03:00:37.072877 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:37.072798    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"
Nov 03 03:00:37.072877 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:37.072807    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg" event={"ID":"b66c3d24-bb50-4e81-a2be-1da246e203d7","Type":"ContainerDied","Data":"d23551a61499a8646152cc3441d8a10e529c25273b5e8b3a5ea9cdb549ba67c7"}
Nov 03 03:00:37.072877 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:37.072859    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d23551a61499a8646152cc3441d8a10e529c25273b5e8b3a5ea9cdb549ba67c7"
Nov 03 03:00:43.811334 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:00:43.811300    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:00:44.137482 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.137399    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"]
Nov 03 03:00:44.137787 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.137775    5508 cpu_manager.go:414] "containerMap: removing container" podUID="b66c3d24-bb50-4e81-a2be-1da246e203d7" containerName="revoke-keys"
Nov 03 03:00:44.137832 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.137788    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66c3d24-bb50-4e81-a2be-1da246e203d7" containerName="revoke-keys"
Nov 03 03:00:44.137874 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.137865    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66c3d24-bb50-4e81-a2be-1da246e203d7" containerName="revoke-keys"
Nov 03 03:00:44.141379 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.141336    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"
Nov 03 03:00:44.176296 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.176256    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f62v\" (UniqueName: \"kubernetes.io/projected/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-kube-api-access-5f62v\") pod \"maas-api-revoke-sub-d3fba056-8zm94\" (UID: \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"
Nov 03 03:00:44.176472 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.176393    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8zm94\" (UID: \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"
Nov 03 03:00:44.176472 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.176433    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8zm94\" (UID: \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"
Nov 03 03:00:44.236720 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.236652    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:00:44.236720 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.236664    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:00:44.236720 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.236709    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:00:44.237081 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.236776    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:00:44.240308 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.240283    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"]
Nov 03 03:00:44.277557 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.277515    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f62v\" (UniqueName: \"kubernetes.io/projected/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-kube-api-access-5f62v\") pod \"maas-api-revoke-sub-d3fba056-8zm94\" (UID: \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"
Nov 03 03:00:44.277763 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.277596    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8zm94\" (UID: \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"
Nov 03 03:00:44.277763 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.277644    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8zm94\" (UID: \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"
Nov 03 03:00:44.278445 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.278406    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8zm94\" (UID: \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"
Nov 03 03:00:44.280028 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.280007    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8zm94\" (UID: \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"
Nov 03 03:00:44.335812 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.335769    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f62v\" (UniqueName: \"kubernetes.io/projected/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-kube-api-access-5f62v\") pod \"maas-api-revoke-sub-d3fba056-8zm94\" (UID: \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"
Nov 03 03:00:44.456687 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.456654    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"
Nov 03 03:00:44.836714 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:44.836618    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"]
Nov 03 03:00:44.846682 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:00:44.846650    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290d0eda_4ddb_49b3_8b3e_4d5495e2e3ac.slice/crio-9463edeef6aac320f0e5d71cbd8075632d01e94acd8411af66b23c8678b8685b WatchSource:0}: Error finding container 9463edeef6aac320f0e5d71cbd8075632d01e94acd8411af66b23c8678b8685b: Status 404 returned error can't find the container with id 9463edeef6aac320f0e5d71cbd8075632d01e94acd8411af66b23c8678b8685b
Nov 03 03:00:45.118585 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:45.118503    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94" event={"ID":"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac","Type":"ContainerStarted","Data":"9463edeef6aac320f0e5d71cbd8075632d01e94acd8411af66b23c8678b8685b"}
Nov 03 03:00:46.125737 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:46.125612    5508 generic.go:356] "Generic (PLEG): container finished" podID="290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac" containerID="5c17e49d538735a880a7c8c4c90c83c80e1b080f59605a1c501b195c34c750e2" exitCode=0
Nov 03 03:00:46.125737 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:46.125680    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94" event={"ID":"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac","Type":"ContainerDied","Data":"5c17e49d538735a880a7c8c4c90c83c80e1b080f59605a1c501b195c34c750e2"}
Nov 03 03:00:46.252268 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:46.252234    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:00:47.336990 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:47.336953    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"
Nov 03 03:00:47.410367 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:47.410276    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-xks-ca\") pod \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\" (UID: \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\") "
Nov 03 03:00:47.410589 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:47.410439    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-kube-api-access-5f62v\" (UniqueName: \"kubernetes.io/projected/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-kube-api-access-5f62v\") pod \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\" (UID: \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\") "
Nov 03 03:00:47.410589 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:47.410519    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-openshift-ca\") pod \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\" (UID: \"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac\") "
Nov 03 03:00:47.411247 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:47.411215    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-openshift-ca" pod "290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac" (UID: "290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:00:47.425790 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:47.415491    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-kube-api-access-5f62v" pod "290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac" (UID: "290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac"). InnerVolumeSpecName "kube-api-access-5f62v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:00:47.425790 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:47.425701    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-xks-ca" pod "290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac" (UID: "290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:00:47.512371 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:47.512258    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.512371 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:47.512298    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:47.512371 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:47.512316    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f62v\" (UniqueName: \"kubernetes.io/projected/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac-kube-api-access-5f62v\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:00:48.160372 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:48.153957    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94" event={"ID":"290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac","Type":"ContainerDied","Data":"9463edeef6aac320f0e5d71cbd8075632d01e94acd8411af66b23c8678b8685b"}
Nov 03 03:00:48.160372 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:48.154025    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9463edeef6aac320f0e5d71cbd8075632d01e94acd8411af66b23c8678b8685b"
Nov 03 03:00:48.160372 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:48.154099    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"
Nov 03 03:00:56.890888 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:00:56.890854    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:01:00.052299 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:00.052261    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"]
Nov 03 03:01:00.061565 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:00.061540    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-n2jhz"]
Nov 03 03:01:00.690735 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:00.690700    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c69de6aa-fb03-407d-b3af-f4ded6d56381" path="/var/lib/kubelet/pods/c69de6aa-fb03-407d-b3af-f4ded6d56381/volumes"
Nov 03 03:01:09.701573 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:09.701539    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:01:13.811323 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:01:13.811287    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:01:19.467487 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.467451    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"]
Nov 03 03:01:19.468042 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.468019    5508 cpu_manager.go:414] "containerMap: removing container" podUID="290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac" containerName="revoke-keys"
Nov 03 03:01:19.468129 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.468045    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac" containerName="revoke-keys"
Nov 03 03:01:19.468187 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.468171    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac" containerName="revoke-keys"
Nov 03 03:01:19.472675 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.472652    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"
Nov 03 03:01:19.528138 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.528109    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9c9\" (UniqueName: \"kubernetes.io/projected/14bb627a-b2a2-4f87-9556-d49475a1af2a-kube-api-access-rf9c9\") pod \"maas-api-revoke-sub-ae594979-d7lks\" (UID: \"14bb627a-b2a2-4f87-9556-d49475a1af2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"
Nov 03 03:01:19.528302 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.528157    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d7lks\" (UID: \"14bb627a-b2a2-4f87-9556-d49475a1af2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"
Nov 03 03:01:19.528302 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.528184    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d7lks\" (UID: \"14bb627a-b2a2-4f87-9556-d49475a1af2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"
Nov 03 03:01:19.567603 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.567573    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:01:19.567603 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.567580    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:01:19.567881 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.567603    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:01:19.567881 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.567612    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:01:19.570274 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.570252    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"]
Nov 03 03:01:19.629601 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.629565    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9c9\" (UniqueName: \"kubernetes.io/projected/14bb627a-b2a2-4f87-9556-d49475a1af2a-kube-api-access-rf9c9\") pod \"maas-api-revoke-sub-ae594979-d7lks\" (UID: \"14bb627a-b2a2-4f87-9556-d49475a1af2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"
Nov 03 03:01:19.629791 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.629627    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d7lks\" (UID: \"14bb627a-b2a2-4f87-9556-d49475a1af2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"
Nov 03 03:01:19.629791 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.629670    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d7lks\" (UID: \"14bb627a-b2a2-4f87-9556-d49475a1af2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"
Nov 03 03:01:19.630333 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.630312    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-d7lks\" (UID: \"14bb627a-b2a2-4f87-9556-d49475a1af2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"
Nov 03 03:01:19.631895 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.631877    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-d7lks\" (UID: \"14bb627a-b2a2-4f87-9556-d49475a1af2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"
Nov 03 03:01:19.687706 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.687671    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9c9\" (UniqueName: \"kubernetes.io/projected/14bb627a-b2a2-4f87-9556-d49475a1af2a-kube-api-access-rf9c9\") pod \"maas-api-revoke-sub-ae594979-d7lks\" (UID: \"14bb627a-b2a2-4f87-9556-d49475a1af2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"
Nov 03 03:01:19.785838 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:19.785751    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"
Nov 03 03:01:20.164109 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:20.164068    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"]
Nov 03 03:01:20.168683 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:01:20.168656    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14bb627a_b2a2_4f87_9556_d49475a1af2a.slice/crio-8cee6e3400951370112d66125ec2066ed78f631ba34a98a0c117001048f1d158 WatchSource:0}: Error finding container 8cee6e3400951370112d66125ec2066ed78f631ba34a98a0c117001048f1d158: Status 404 returned error can't find the container with id 8cee6e3400951370112d66125ec2066ed78f631ba34a98a0c117001048f1d158
Nov 03 03:01:20.386762 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:20.386729    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks" event={"ID":"14bb627a-b2a2-4f87-9556-d49475a1af2a","Type":"ContainerStarted","Data":"8cee6e3400951370112d66125ec2066ed78f631ba34a98a0c117001048f1d158"}
Nov 03 03:01:21.393681 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:21.393597    5508 generic.go:356] "Generic (PLEG): container finished" podID="14bb627a-b2a2-4f87-9556-d49475a1af2a" containerID="02fb355fb66adfdbd8e9beaa780fffb5de2432c04db9e383affc52beca73f016" exitCode=0
Nov 03 03:01:21.394067 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:21.393681    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks" event={"ID":"14bb627a-b2a2-4f87-9556-d49475a1af2a","Type":"ContainerDied","Data":"02fb355fb66adfdbd8e9beaa780fffb5de2432c04db9e383affc52beca73f016"}
Nov 03 03:01:22.534549 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:22.534519    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"
Nov 03 03:01:22.662328 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:22.662237    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14bb627a-b2a2-4f87-9556-d49475a1af2a-kube-api-access-rf9c9\" (UniqueName: \"kubernetes.io/projected/14bb627a-b2a2-4f87-9556-d49475a1af2a-kube-api-access-rf9c9\") pod \"14bb627a-b2a2-4f87-9556-d49475a1af2a\" (UID: \"14bb627a-b2a2-4f87-9556-d49475a1af2a\") "
Nov 03 03:01:22.662328 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:22.662292    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-openshift-ca\") pod \"14bb627a-b2a2-4f87-9556-d49475a1af2a\" (UID: \"14bb627a-b2a2-4f87-9556-d49475a1af2a\") "
Nov 03 03:01:22.662630 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:22.662428    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-xks-ca\") pod \"14bb627a-b2a2-4f87-9556-d49475a1af2a\" (UID: \"14bb627a-b2a2-4f87-9556-d49475a1af2a\") "
Nov 03 03:01:22.662695 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:22.662678    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-openshift-ca" pod "14bb627a-b2a2-4f87-9556-d49475a1af2a" (UID: "14bb627a-b2a2-4f87-9556-d49475a1af2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:01:22.662839 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:22.662818    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:01:22.664487 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:22.664460    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-xks-ca" pod "14bb627a-b2a2-4f87-9556-d49475a1af2a" (UID: "14bb627a-b2a2-4f87-9556-d49475a1af2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:01:22.664926 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:22.664906    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14bb627a-b2a2-4f87-9556-d49475a1af2a-kube-api-access-rf9c9" pod "14bb627a-b2a2-4f87-9556-d49475a1af2a" (UID: "14bb627a-b2a2-4f87-9556-d49475a1af2a"). InnerVolumeSpecName "kube-api-access-rf9c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:01:22.763790 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:22.763756    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf9c9\" (UniqueName: \"kubernetes.io/projected/14bb627a-b2a2-4f87-9556-d49475a1af2a-kube-api-access-rf9c9\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:01:22.763790 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:22.763786    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14bb627a-b2a2-4f87-9556-d49475a1af2a-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:01:23.405221 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:23.405183    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"
Nov 03 03:01:23.405221 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:23.405214    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks" event={"ID":"14bb627a-b2a2-4f87-9556-d49475a1af2a","Type":"ContainerDied","Data":"8cee6e3400951370112d66125ec2066ed78f631ba34a98a0c117001048f1d158"}
Nov 03 03:01:23.405499 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:23.405286    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cee6e3400951370112d66125ec2066ed78f631ba34a98a0c117001048f1d158"
Nov 03 03:01:24.046387 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:24.046340    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"]
Nov 03 03:01:24.092541 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:24.092511    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5954b227-lwlkp"]
Nov 03 03:01:24.690129 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:24.690097    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a16c942-c711-4960-908d-9cd4f8793e3d" path="/var/lib/kubelet/pods/3a16c942-c711-4960-908d-9cd4f8793e3d/volumes"
Nov 03 03:01:32.302628 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:32.302583    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:01:38.291958 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.291913    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"]
Nov 03 03:01:38.292596 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.292573    5508 cpu_manager.go:414] "containerMap: removing container" podUID="14bb627a-b2a2-4f87-9556-d49475a1af2a" containerName="revoke-keys"
Nov 03 03:01:38.292686 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.292598    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14bb627a-b2a2-4f87-9556-d49475a1af2a" containerName="revoke-keys"
Nov 03 03:01:38.292759 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.292705    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="14bb627a-b2a2-4f87-9556-d49475a1af2a" containerName="revoke-keys"
Nov 03 03:01:38.296496 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.296472    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"
Nov 03 03:01:38.328039 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.327055    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"]
Nov 03 03:01:38.328039 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.327627    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2qrnn\" (UID: \"8fabb21e-dcfe-4608-8730-9382f32d772c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"
Nov 03 03:01:38.328039 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.327747    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwbgw\" (UniqueName: \"kubernetes.io/projected/8fabb21e-dcfe-4608-8730-9382f32d772c-kube-api-access-xwbgw\") pod \"maas-api-revoke-sub-a4aef7d7-2qrnn\" (UID: \"8fabb21e-dcfe-4608-8730-9382f32d772c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"
Nov 03 03:01:38.328039 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.327879    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2qrnn\" (UID: \"8fabb21e-dcfe-4608-8730-9382f32d772c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"
Nov 03 03:01:38.348454 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.348405    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:01:38.348646 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.348465    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:01:38.348646 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.348465    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:01:38.348646 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.348567    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:01:38.433374 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.429883    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2qrnn\" (UID: \"8fabb21e-dcfe-4608-8730-9382f32d772c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"
Nov 03 03:01:38.438662 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.438614    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwbgw\" (UniqueName: \"kubernetes.io/projected/8fabb21e-dcfe-4608-8730-9382f32d772c-kube-api-access-xwbgw\") pod \"maas-api-revoke-sub-a4aef7d7-2qrnn\" (UID: \"8fabb21e-dcfe-4608-8730-9382f32d772c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"
Nov 03 03:01:38.442454 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.442412    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2qrnn\" (UID: \"8fabb21e-dcfe-4608-8730-9382f32d772c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"
Nov 03 03:01:38.445915 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.445883    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2qrnn\" (UID: \"8fabb21e-dcfe-4608-8730-9382f32d772c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"
Nov 03 03:01:38.478006 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.477925    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2qrnn\" (UID: \"8fabb21e-dcfe-4608-8730-9382f32d772c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"
Nov 03 03:01:38.555046 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.547804    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwbgw\" (UniqueName: \"kubernetes.io/projected/8fabb21e-dcfe-4608-8730-9382f32d772c-kube-api-access-xwbgw\") pod \"maas-api-revoke-sub-a4aef7d7-2qrnn\" (UID: \"8fabb21e-dcfe-4608-8730-9382f32d772c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"
Nov 03 03:01:38.616796 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:38.616522    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"
Nov 03 03:01:39.310844 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:39.310814    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"]
Nov 03 03:01:39.627116 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:39.626948    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn" event={"ID":"8fabb21e-dcfe-4608-8730-9382f32d772c","Type":"ContainerStarted","Data":"fbd113c0e1ac27010f8dcb01c1e3ca28f37315028eeef69fcca722e3af240b39"}
Nov 03 03:01:40.347920 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:40.347868    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:01:40.634374 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:40.634254    5508 generic.go:356] "Generic (PLEG): container finished" podID="8fabb21e-dcfe-4608-8730-9382f32d772c" containerID="b184479db64c33776230d9ca76b3ba9bd596f2ad156ff8c0958dbb2ce398190d" exitCode=0
Nov 03 03:01:40.634584 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:40.634386    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn" event={"ID":"8fabb21e-dcfe-4608-8730-9382f32d772c","Type":"ContainerDied","Data":"b184479db64c33776230d9ca76b3ba9bd596f2ad156ff8c0958dbb2ce398190d"}
Nov 03 03:01:41.771956 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:41.771929    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"
Nov 03 03:01:41.853167 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:41.853130    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-openshift-ca\") pod \"8fabb21e-dcfe-4608-8730-9382f32d772c\" (UID: \"8fabb21e-dcfe-4608-8730-9382f32d772c\") "
Nov 03 03:01:41.853380 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:41.853193    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fabb21e-dcfe-4608-8730-9382f32d772c-kube-api-access-xwbgw\" (UniqueName: \"kubernetes.io/projected/8fabb21e-dcfe-4608-8730-9382f32d772c-kube-api-access-xwbgw\") pod \"8fabb21e-dcfe-4608-8730-9382f32d772c\" (UID: \"8fabb21e-dcfe-4608-8730-9382f32d772c\") "
Nov 03 03:01:41.853380 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:41.853236    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-xks-ca\") pod \"8fabb21e-dcfe-4608-8730-9382f32d772c\" (UID: \"8fabb21e-dcfe-4608-8730-9382f32d772c\") "
Nov 03 03:01:41.853585 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:41.853563    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-openshift-ca" pod "8fabb21e-dcfe-4608-8730-9382f32d772c" (UID: "8fabb21e-dcfe-4608-8730-9382f32d772c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:01:41.855388 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:41.855362    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-xks-ca" pod "8fabb21e-dcfe-4608-8730-9382f32d772c" (UID: "8fabb21e-dcfe-4608-8730-9382f32d772c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:01:41.855513 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:41.855384    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fabb21e-dcfe-4608-8730-9382f32d772c-kube-api-access-xwbgw" pod "8fabb21e-dcfe-4608-8730-9382f32d772c" (UID: "8fabb21e-dcfe-4608-8730-9382f32d772c"). InnerVolumeSpecName "kube-api-access-xwbgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:01:41.954661 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:41.954623    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:01:41.954661 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:41.954653    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwbgw\" (UniqueName: \"kubernetes.io/projected/8fabb21e-dcfe-4608-8730-9382f32d772c-kube-api-access-xwbgw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:01:41.954661 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:41.954663    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fabb21e-dcfe-4608-8730-9382f32d772c-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:01:42.645761 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:42.645734    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"
Nov 03 03:01:42.645952 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:42.645764    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn" event={"ID":"8fabb21e-dcfe-4608-8730-9382f32d772c","Type":"ContainerDied","Data":"fbd113c0e1ac27010f8dcb01c1e3ca28f37315028eeef69fcca722e3af240b39"}
Nov 03 03:01:42.645952 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:42.645809    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbd113c0e1ac27010f8dcb01c1e3ca28f37315028eeef69fcca722e3af240b39"
Nov 03 03:01:43.811414 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:01:43.811375    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:01:47.300197 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:47.300153    5508 scope.go:122] "RemoveContainer" containerID="46957e1f894174c4836e2baa0b981c060c97ec8b4bcc7c16be16acef51e40177"
Nov 03 03:01:48.698757 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:48.698726    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:01:57.478802 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:57.478767    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:01:59.544606 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.544566    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"]
Nov 03 03:01:59.550946 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.550916    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8fabb21e-dcfe-4608-8730-9382f32d772c" containerName="revoke-keys"
Nov 03 03:01:59.551168 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.551149    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fabb21e-dcfe-4608-8730-9382f32d772c" containerName="revoke-keys"
Nov 03 03:01:59.551459 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.551442    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fabb21e-dcfe-4608-8730-9382f32d772c" containerName="revoke-keys"
Nov 03 03:01:59.555185 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.555167    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"
Nov 03 03:01:59.637781 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.637735    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"]
Nov 03 03:01:59.658383 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.652029    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rk7pg\" (UID: \"715f1e6a-c193-4985-b324-0865a16f5a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"
Nov 03 03:01:59.658724 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.658689    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rk7pg\" (UID: \"715f1e6a-c193-4985-b324-0865a16f5a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"
Nov 03 03:01:59.658910 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.658893    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8mm\" (UniqueName: \"kubernetes.io/projected/715f1e6a-c193-4985-b324-0865a16f5a87-kube-api-access-ch8mm\") pod \"maas-api-revoke-sub-79805ca9-rk7pg\" (UID: \"715f1e6a-c193-4985-b324-0865a16f5a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"
Nov 03 03:01:59.690860 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.690826    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:01:59.700582 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.700551    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:01:59.700940 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.700921    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:01:59.701241 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.701209    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:01:59.829238 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.829130    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8mm\" (UniqueName: \"kubernetes.io/projected/715f1e6a-c193-4985-b324-0865a16f5a87-kube-api-access-ch8mm\") pod \"maas-api-revoke-sub-79805ca9-rk7pg\" (UID: \"715f1e6a-c193-4985-b324-0865a16f5a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"
Nov 03 03:01:59.833721 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.833687    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rk7pg\" (UID: \"715f1e6a-c193-4985-b324-0865a16f5a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"
Nov 03 03:01:59.834516 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.834493    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rk7pg\" (UID: \"715f1e6a-c193-4985-b324-0865a16f5a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"
Nov 03 03:01:59.835500 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.835396    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rk7pg\" (UID: \"715f1e6a-c193-4985-b324-0865a16f5a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"
Nov 03 03:01:59.882372 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.869160    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rk7pg\" (UID: \"715f1e6a-c193-4985-b324-0865a16f5a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"
Nov 03 03:01:59.944372 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:01:59.944112    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8mm\" (UniqueName: \"kubernetes.io/projected/715f1e6a-c193-4985-b324-0865a16f5a87-kube-api-access-ch8mm\") pod \"maas-api-revoke-sub-79805ca9-rk7pg\" (UID: \"715f1e6a-c193-4985-b324-0865a16f5a87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"
Nov 03 03:02:00.194373 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:00.193525    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"
Nov 03 03:02:00.992131 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:00.992090    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"]
Nov 03 03:02:01.984455 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:01.984424    5508 generic.go:356] "Generic (PLEG): container finished" podID="715f1e6a-c193-4985-b324-0865a16f5a87" containerID="e941c9897830c5ccd417b924aee0ec738dcccee573f40063121205030c86b260" exitCode=0
Nov 03 03:02:01.984644 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:01.984470    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg" event={"ID":"715f1e6a-c193-4985-b324-0865a16f5a87","Type":"ContainerDied","Data":"e941c9897830c5ccd417b924aee0ec738dcccee573f40063121205030c86b260"}
Nov 03 03:02:01.984644 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:01.984531    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg" event={"ID":"715f1e6a-c193-4985-b324-0865a16f5a87","Type":"ContainerStarted","Data":"a79a2e26696ae052ba75c37fe2c08e0c26e7ab36a0d2cbfe0674e2062d757e22"}
Nov 03 03:02:03.127186 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:03.127153    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"
Nov 03 03:02:03.212673 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:03.212648    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-openshift-ca\") pod \"715f1e6a-c193-4985-b324-0865a16f5a87\" (UID: \"715f1e6a-c193-4985-b324-0865a16f5a87\") "
Nov 03 03:02:03.212860 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:03.212723    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-xks-ca\") pod \"715f1e6a-c193-4985-b324-0865a16f5a87\" (UID: \"715f1e6a-c193-4985-b324-0865a16f5a87\") "
Nov 03 03:02:03.212860 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:03.212749    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715f1e6a-c193-4985-b324-0865a16f5a87-kube-api-access-ch8mm\" (UniqueName: \"kubernetes.io/projected/715f1e6a-c193-4985-b324-0865a16f5a87-kube-api-access-ch8mm\") pod \"715f1e6a-c193-4985-b324-0865a16f5a87\" (UID: \"715f1e6a-c193-4985-b324-0865a16f5a87\") "
Nov 03 03:02:03.213050 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:03.213021    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-openshift-ca" pod "715f1e6a-c193-4985-b324-0865a16f5a87" (UID: "715f1e6a-c193-4985-b324-0865a16f5a87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:02:03.214902 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:03.214872    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-xks-ca" pod "715f1e6a-c193-4985-b324-0865a16f5a87" (UID: "715f1e6a-c193-4985-b324-0865a16f5a87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:02:03.215003 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:03.214935    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715f1e6a-c193-4985-b324-0865a16f5a87-kube-api-access-ch8mm" pod "715f1e6a-c193-4985-b324-0865a16f5a87" (UID: "715f1e6a-c193-4985-b324-0865a16f5a87"). InnerVolumeSpecName "kube-api-access-ch8mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:02:03.314164 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:03.314077    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:02:03.314164 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:03.314109    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch8mm\" (UniqueName: \"kubernetes.io/projected/715f1e6a-c193-4985-b324-0865a16f5a87-kube-api-access-ch8mm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:02:03.314164 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:03.314120    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715f1e6a-c193-4985-b324-0865a16f5a87-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:02:03.996556 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:03.996521    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"
Nov 03 03:02:03.996556 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:03.996534    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg" event={"ID":"715f1e6a-c193-4985-b324-0865a16f5a87","Type":"ContainerDied","Data":"a79a2e26696ae052ba75c37fe2c08e0c26e7ab36a0d2cbfe0674e2062d757e22"}
Nov 03 03:02:03.996780 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:03.996589    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a79a2e26696ae052ba75c37fe2c08e0c26e7ab36a0d2cbfe0674e2062d757e22"
Nov 03 03:02:07.090939 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:07.090906    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:02:13.811623 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:02:13.811586    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:02:16.061868 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:16.061785    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:02:17.050386 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.048096    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"]
Nov 03 03:02:17.050386 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.049558    5508 cpu_manager.go:414] "containerMap: removing container" podUID="715f1e6a-c193-4985-b324-0865a16f5a87" containerName="revoke-keys"
Nov 03 03:02:17.050386 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.049584    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715f1e6a-c193-4985-b324-0865a16f5a87" containerName="revoke-keys"
Nov 03 03:02:17.050386 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.049711    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="715f1e6a-c193-4985-b324-0865a16f5a87" containerName="revoke-keys"
Nov 03 03:02:17.057369 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.053710    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"
Nov 03 03:02:17.154404 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.151212    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:02:17.154404 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.151876    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:02:17.155178 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.155116    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:02:17.158647 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.158255    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9x5vj\" (UID: \"02eaa27c-70ad-480a-8b9f-710562426465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"
Nov 03 03:02:17.158647 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.158319    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwrqz\" (UniqueName: \"kubernetes.io/projected/02eaa27c-70ad-480a-8b9f-710562426465-kube-api-access-dwrqz\") pod \"maas-api-revoke-sub-0b7e8edb-9x5vj\" (UID: \"02eaa27c-70ad-480a-8b9f-710562426465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"
Nov 03 03:02:17.158647 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.158488    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9x5vj\" (UID: \"02eaa27c-70ad-480a-8b9f-710562426465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"
Nov 03 03:02:17.158951 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.157711    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"]
Nov 03 03:02:17.203057 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.203018    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:02:17.259628 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.259584    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9x5vj\" (UID: \"02eaa27c-70ad-480a-8b9f-710562426465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"
Nov 03 03:02:17.259864 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.259651    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwrqz\" (UniqueName: \"kubernetes.io/projected/02eaa27c-70ad-480a-8b9f-710562426465-kube-api-access-dwrqz\") pod \"maas-api-revoke-sub-0b7e8edb-9x5vj\" (UID: \"02eaa27c-70ad-480a-8b9f-710562426465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"
Nov 03 03:02:17.259864 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.259788    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9x5vj\" (UID: \"02eaa27c-70ad-480a-8b9f-710562426465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"
Nov 03 03:02:17.260690 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.260664    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9x5vj\" (UID: \"02eaa27c-70ad-480a-8b9f-710562426465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"
Nov 03 03:02:17.262838 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.262813    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-9x5vj\" (UID: \"02eaa27c-70ad-480a-8b9f-710562426465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"
Nov 03 03:02:17.318544 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.318414    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwrqz\" (UniqueName: \"kubernetes.io/projected/02eaa27c-70ad-480a-8b9f-710562426465-kube-api-access-dwrqz\") pod \"maas-api-revoke-sub-0b7e8edb-9x5vj\" (UID: \"02eaa27c-70ad-480a-8b9f-710562426465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"
Nov 03 03:02:17.387102 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:17.386783    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"
Nov 03 03:02:18.321317 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:18.317956    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"]
Nov 03 03:02:18.379223 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:02:18.379189    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02eaa27c_70ad_480a_8b9f_710562426465.slice/crio-a2c095f28cace1110491b47d730aba3923c4c4f4e238097f0ccf903e10df35a8 WatchSource:0}: Error finding container a2c095f28cace1110491b47d730aba3923c4c4f4e238097f0ccf903e10df35a8: Status 404 returned error can't find the container with id a2c095f28cace1110491b47d730aba3923c4c4f4e238097f0ccf903e10df35a8
Nov 03 03:02:19.367389 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:19.363525    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj" event={"ID":"02eaa27c-70ad-480a-8b9f-710562426465","Type":"ContainerStarted","Data":"9c120378626f62938987e5886bdcb16fc25eac792d481e9dcad009e62b90caae"}
Nov 03 03:02:19.367389 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:19.363636    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj" event={"ID":"02eaa27c-70ad-480a-8b9f-710562426465","Type":"ContainerStarted","Data":"a2c095f28cace1110491b47d730aba3923c4c4f4e238097f0ccf903e10df35a8"}
Nov 03 03:02:20.370263 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:20.370224    5508 generic.go:356] "Generic (PLEG): container finished" podID="02eaa27c-70ad-480a-8b9f-710562426465" containerID="9c120378626f62938987e5886bdcb16fc25eac792d481e9dcad009e62b90caae" exitCode=0
Nov 03 03:02:20.370776 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:20.370289    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj" event={"ID":"02eaa27c-70ad-480a-8b9f-710562426465","Type":"ContainerDied","Data":"9c120378626f62938987e5886bdcb16fc25eac792d481e9dcad009e62b90caae"}
Nov 03 03:02:21.521581 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:21.521553    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"
Nov 03 03:02:21.705019 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:21.704986    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02eaa27c-70ad-480a-8b9f-710562426465-kube-api-access-dwrqz\" (UniqueName: \"kubernetes.io/projected/02eaa27c-70ad-480a-8b9f-710562426465-kube-api-access-dwrqz\") pod \"02eaa27c-70ad-480a-8b9f-710562426465\" (UID: \"02eaa27c-70ad-480a-8b9f-710562426465\") "
Nov 03 03:02:21.705240 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:21.705122    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-openshift-ca\") pod \"02eaa27c-70ad-480a-8b9f-710562426465\" (UID: \"02eaa27c-70ad-480a-8b9f-710562426465\") "
Nov 03 03:02:21.705240 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:21.705148    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-xks-ca\") pod \"02eaa27c-70ad-480a-8b9f-710562426465\" (UID: \"02eaa27c-70ad-480a-8b9f-710562426465\") "
Nov 03 03:02:21.705582 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:21.705552    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-openshift-ca" pod "02eaa27c-70ad-480a-8b9f-710562426465" (UID: "02eaa27c-70ad-480a-8b9f-710562426465"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:02:21.707198 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:21.707175    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-xks-ca" pod "02eaa27c-70ad-480a-8b9f-710562426465" (UID: "02eaa27c-70ad-480a-8b9f-710562426465"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:02:21.707300 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:21.707275    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eaa27c-70ad-480a-8b9f-710562426465-kube-api-access-dwrqz" pod "02eaa27c-70ad-480a-8b9f-710562426465" (UID: "02eaa27c-70ad-480a-8b9f-710562426465"). InnerVolumeSpecName "kube-api-access-dwrqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:02:21.806868 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:21.806834    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwrqz\" (UniqueName: \"kubernetes.io/projected/02eaa27c-70ad-480a-8b9f-710562426465-kube-api-access-dwrqz\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:02:21.806868 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:21.806863    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:02:21.806868 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:21.806873    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02eaa27c-70ad-480a-8b9f-710562426465-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:02:22.382487 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:22.382442    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"
Nov 03 03:02:22.382687 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:22.382493    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj" event={"ID":"02eaa27c-70ad-480a-8b9f-710562426465","Type":"ContainerDied","Data":"a2c095f28cace1110491b47d730aba3923c4c4f4e238097f0ccf903e10df35a8"}
Nov 03 03:02:22.382687 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:22.382549    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2c095f28cace1110491b47d730aba3923c4c4f4e238097f0ccf903e10df35a8"
Nov 03 03:02:23.959530 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:23.959492    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:02:28.365086 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.365052    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"]
Nov 03 03:02:28.365650 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.365473    5508 cpu_manager.go:414] "containerMap: removing container" podUID="02eaa27c-70ad-480a-8b9f-710562426465" containerName="revoke-keys"
Nov 03 03:02:28.365650 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.365486    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02eaa27c-70ad-480a-8b9f-710562426465" containerName="revoke-keys"
Nov 03 03:02:28.365650 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.365572    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="02eaa27c-70ad-480a-8b9f-710562426465" containerName="revoke-keys"
Nov 03 03:02:28.372646 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.372622    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"
Nov 03 03:02:28.423970 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.423939    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:02:28.470392 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.470329    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:02:28.470392 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.470363    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:02:28.470392 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.470383    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:02:28.472851 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.472829    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"]
Nov 03 03:02:28.473752 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.473728    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c846139-1309-436d-bac0-840794a56c9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pw926\" (UID: \"9c846139-1309-436d-bac0-840794a56c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"
Nov 03 03:02:28.473864 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.473847    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c846139-1309-436d-bac0-840794a56c9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pw926\" (UID: \"9c846139-1309-436d-bac0-840794a56c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"
Nov 03 03:02:28.473932 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.473890    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfzh8\" (UniqueName: \"kubernetes.io/projected/9c846139-1309-436d-bac0-840794a56c9c-kube-api-access-zfzh8\") pod \"maas-api-revoke-sub-3d4156f1-pw926\" (UID: \"9c846139-1309-436d-bac0-840794a56c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"
Nov 03 03:02:28.575026 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.574991    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c846139-1309-436d-bac0-840794a56c9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pw926\" (UID: \"9c846139-1309-436d-bac0-840794a56c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"
Nov 03 03:02:28.575227 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.575069    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c846139-1309-436d-bac0-840794a56c9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pw926\" (UID: \"9c846139-1309-436d-bac0-840794a56c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"
Nov 03 03:02:28.575227 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.575099    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzh8\" (UniqueName: \"kubernetes.io/projected/9c846139-1309-436d-bac0-840794a56c9c-kube-api-access-zfzh8\") pod \"maas-api-revoke-sub-3d4156f1-pw926\" (UID: \"9c846139-1309-436d-bac0-840794a56c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"
Nov 03 03:02:28.575710 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.575685    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c846139-1309-436d-bac0-840794a56c9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pw926\" (UID: \"9c846139-1309-436d-bac0-840794a56c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"
Nov 03 03:02:28.577451 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.577429    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c846139-1309-436d-bac0-840794a56c9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pw926\" (UID: \"9c846139-1309-436d-bac0-840794a56c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"
Nov 03 03:02:28.632178 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.632114    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfzh8\" (UniqueName: \"kubernetes.io/projected/9c846139-1309-436d-bac0-840794a56c9c-kube-api-access-zfzh8\") pod \"maas-api-revoke-sub-3d4156f1-pw926\" (UID: \"9c846139-1309-436d-bac0-840794a56c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"
Nov 03 03:02:28.686098 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:28.686071    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"
Nov 03 03:02:29.059850 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:29.059819    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"]
Nov 03 03:02:29.067162 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:02:29.067130    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c846139_1309_436d_bac0_840794a56c9c.slice/crio-85b4632770d30d9198485c9b4c00ab8354b928d77558b0a569ac978b18f9b7e4 WatchSource:0}: Error finding container 85b4632770d30d9198485c9b4c00ab8354b928d77558b0a569ac978b18f9b7e4: Status 404 returned error can't find the container with id 85b4632770d30d9198485c9b4c00ab8354b928d77558b0a569ac978b18f9b7e4
Nov 03 03:02:29.427033 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:29.427000    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926" event={"ID":"9c846139-1309-436d-bac0-840794a56c9c","Type":"ContainerStarted","Data":"73354cbabb7da95b9aa1ab243f8b7cf14462c225ab524bee7da0da86d1606544"}
Nov 03 03:02:29.427465 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:29.427050    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926" event={"ID":"9c846139-1309-436d-bac0-840794a56c9c","Type":"ContainerStarted","Data":"85b4632770d30d9198485c9b4c00ab8354b928d77558b0a569ac978b18f9b7e4"}
Nov 03 03:02:29.534992 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:29.534930    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926" podStartSLOduration=1.534914334 podStartE2EDuration="1.534914334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:02:29.534652591 +0000 UTC m=+6228.960259055" watchObservedRunningTime="2026-11-03 03:02:29.534914334 +0000 UTC m=+6228.960520822"
Nov 03 03:02:30.433707 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:30.433674    5508 generic.go:356] "Generic (PLEG): container finished" podID="9c846139-1309-436d-bac0-840794a56c9c" containerID="73354cbabb7da95b9aa1ab243f8b7cf14462c225ab524bee7da0da86d1606544" exitCode=0
Nov 03 03:02:30.434260 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:30.433773    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926" event={"ID":"9c846139-1309-436d-bac0-840794a56c9c","Type":"ContainerDied","Data":"73354cbabb7da95b9aa1ab243f8b7cf14462c225ab524bee7da0da86d1606544"}
Nov 03 03:02:31.564899 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:31.564873    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"
Nov 03 03:02:31.607744 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:31.607710    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c846139-1309-436d-bac0-840794a56c9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c846139-1309-436d-bac0-840794a56c9c-maas-api-openshift-ca\") pod \"9c846139-1309-436d-bac0-840794a56c9c\" (UID: \"9c846139-1309-436d-bac0-840794a56c9c\") "
Nov 03 03:02:31.607919 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:31.607856    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c846139-1309-436d-bac0-840794a56c9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c846139-1309-436d-bac0-840794a56c9c-maas-api-xks-ca\") pod \"9c846139-1309-436d-bac0-840794a56c9c\" (UID: \"9c846139-1309-436d-bac0-840794a56c9c\") "
Nov 03 03:02:31.607919 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:31.607879    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c846139-1309-436d-bac0-840794a56c9c-kube-api-access-zfzh8\" (UniqueName: \"kubernetes.io/projected/9c846139-1309-436d-bac0-840794a56c9c-kube-api-access-zfzh8\") pod \"9c846139-1309-436d-bac0-840794a56c9c\" (UID: \"9c846139-1309-436d-bac0-840794a56c9c\") "
Nov 03 03:02:31.608138 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:31.608105    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c846139-1309-436d-bac0-840794a56c9c-maas-api-openshift-ca" pod "9c846139-1309-436d-bac0-840794a56c9c" (UID: "9c846139-1309-436d-bac0-840794a56c9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:02:31.609912 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:31.609888    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c846139-1309-436d-bac0-840794a56c9c-maas-api-xks-ca" pod "9c846139-1309-436d-bac0-840794a56c9c" (UID: "9c846139-1309-436d-bac0-840794a56c9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:02:31.610020 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:31.609998    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c846139-1309-436d-bac0-840794a56c9c-kube-api-access-zfzh8" pod "9c846139-1309-436d-bac0-840794a56c9c" (UID: "9c846139-1309-436d-bac0-840794a56c9c"). InnerVolumeSpecName "kube-api-access-zfzh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:02:31.709023 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:31.708942    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c846139-1309-436d-bac0-840794a56c9c-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:02:31.709023 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:31.708972    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfzh8\" (UniqueName: \"kubernetes.io/projected/9c846139-1309-436d-bac0-840794a56c9c-kube-api-access-zfzh8\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:02:31.709023 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:31.708983    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c846139-1309-436d-bac0-840794a56c9c-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:02:32.445928 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:32.445890    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926" event={"ID":"9c846139-1309-436d-bac0-840794a56c9c","Type":"ContainerDied","Data":"85b4632770d30d9198485c9b4c00ab8354b928d77558b0a569ac978b18f9b7e4"}
Nov 03 03:02:32.446114 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:32.445934    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"
Nov 03 03:02:32.446114 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:32.445952    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85b4632770d30d9198485c9b4c00ab8354b928d77558b0a569ac978b18f9b7e4"
Nov 03 03:02:42.890787 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:42.890751    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:02:43.831269 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:02:43.831224    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:02:46.849799 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:46.849763    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"]
Nov 03 03:02:46.886996 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:46.886963    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pw926"]
Nov 03 03:02:46.976778 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:46.976749    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"]
Nov 03 03:02:46.977181 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:46.977161    5508 cpu_manager.go:414] "containerMap: removing container" podUID="9c846139-1309-436d-bac0-840794a56c9c" containerName="revoke-keys"
Nov 03 03:02:46.977181 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:46.977179    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c846139-1309-436d-bac0-840794a56c9c" containerName="revoke-keys"
Nov 03 03:02:46.977383 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:46.977291    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c846139-1309-436d-bac0-840794a56c9c" containerName="revoke-keys"
Nov 03 03:02:46.980610 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:46.980591    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"
Nov 03 03:02:47.029387 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.029341    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:02:47.029549 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.029417    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:02:47.029549 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.029431    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:02:47.029549 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.029464    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:02:47.038569 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.038542    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"]
Nov 03 03:02:47.088528 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.088498    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fqsx\" (UniqueName: \"kubernetes.io/projected/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-kube-api-access-8fqsx\") pod \"maas-api-revoke-sub-3d4156f1-d72rc\" (UID: \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"
Nov 03 03:02:47.088711 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.088539    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d72rc\" (UID: \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"
Nov 03 03:02:47.088711 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.088615    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d72rc\" (UID: \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"
Nov 03 03:02:47.189949 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.189918    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fqsx\" (UniqueName: \"kubernetes.io/projected/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-kube-api-access-8fqsx\") pod \"maas-api-revoke-sub-3d4156f1-d72rc\" (UID: \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"
Nov 03 03:02:47.190138 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.189957    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d72rc\" (UID: \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"
Nov 03 03:02:47.190138 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.189992    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d72rc\" (UID: \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"
Nov 03 03:02:47.190681 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.190657    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d72rc\" (UID: \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"
Nov 03 03:02:47.192427 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.192402    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d72rc\" (UID: \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"
Nov 03 03:02:47.248164 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.248133    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fqsx\" (UniqueName: \"kubernetes.io/projected/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-kube-api-access-8fqsx\") pod \"maas-api-revoke-sub-3d4156f1-d72rc\" (UID: \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"
Nov 03 03:02:47.293742 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.293713    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"
Nov 03 03:02:47.323340 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.323314    5508 scope.go:122] "RemoveContainer" containerID="a5d89a6cdae09bc9ff8cb69ea30da4284366559f5e54f9de1dedcb7f1ce3b32b"
Nov 03 03:02:47.332343 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.332314    5508 scope.go:122] "RemoveContainer" containerID="d2df673198d4042bbd000d640994e75fdaf37efa2c7aadfb1a27057315d79bef"
Nov 03 03:02:47.671645 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:47.671606    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"]
Nov 03 03:02:47.679362 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:02:47.679314    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5681e87_fbf6_4a09_87cd_90d3d4d03a1c.slice/crio-45afd3d9451e4ed3ddfdfb37970977e6e3374a035a58ed1840940961d2e2fd5e WatchSource:0}: Error finding container 45afd3d9451e4ed3ddfdfb37970977e6e3374a035a58ed1840940961d2e2fd5e: Status 404 returned error can't find the container with id 45afd3d9451e4ed3ddfdfb37970977e6e3374a035a58ed1840940961d2e2fd5e
Nov 03 03:02:48.553752 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:48.553718    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc" event={"ID":"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c","Type":"ContainerStarted","Data":"50ca5788aaa9ca56a5fe880d8f7397c57dd45bb98e09b2dc8bfdc6604a64c42c"}
Nov 03 03:02:48.554211 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:48.553772    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc" event={"ID":"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c","Type":"ContainerStarted","Data":"45afd3d9451e4ed3ddfdfb37970977e6e3374a035a58ed1840940961d2e2fd5e"}
Nov 03 03:02:48.664803 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:48.664753    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc" podStartSLOduration=2.66473908 podStartE2EDuration="2.66473908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:02:48.662222124 +0000 UTC m=+6248.087828590" watchObservedRunningTime="2026-11-03 03:02:48.66473908 +0000 UTC m=+6248.090345546"
Nov 03 03:02:48.690942 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:48.690909    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c846139-1309-436d-bac0-840794a56c9c" path="/var/lib/kubelet/pods/9c846139-1309-436d-bac0-840794a56c9c/volumes"
Nov 03 03:02:49.560953 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:49.560920    5508 generic.go:356] "Generic (PLEG): container finished" podID="a5681e87-fbf6-4a09-87cd-90d3d4d03a1c" containerID="50ca5788aaa9ca56a5fe880d8f7397c57dd45bb98e09b2dc8bfdc6604a64c42c" exitCode=0
Nov 03 03:02:49.561410 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:49.561009    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc" event={"ID":"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c","Type":"ContainerDied","Data":"50ca5788aaa9ca56a5fe880d8f7397c57dd45bb98e09b2dc8bfdc6604a64c42c"}
Nov 03 03:02:50.712656 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:50.712626    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"
Nov 03 03:02:50.833099 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:50.833015    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-xks-ca\") pod \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\" (UID: \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\") "
Nov 03 03:02:50.833294 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:50.833119    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-openshift-ca\") pod \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\" (UID: \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\") "
Nov 03 03:02:50.833294 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:50.833148    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-kube-api-access-8fqsx\" (UniqueName: \"kubernetes.io/projected/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-kube-api-access-8fqsx\") pod \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\" (UID: \"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c\") "
Nov 03 03:02:50.833546 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:50.833522    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-openshift-ca" pod "a5681e87-fbf6-4a09-87cd-90d3d4d03a1c" (UID: "a5681e87-fbf6-4a09-87cd-90d3d4d03a1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:02:50.835283 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:50.835256    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-xks-ca" pod "a5681e87-fbf6-4a09-87cd-90d3d4d03a1c" (UID: "a5681e87-fbf6-4a09-87cd-90d3d4d03a1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:02:50.835381 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:50.835282    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-kube-api-access-8fqsx" pod "a5681e87-fbf6-4a09-87cd-90d3d4d03a1c" (UID: "a5681e87-fbf6-4a09-87cd-90d3d4d03a1c"). InnerVolumeSpecName "kube-api-access-8fqsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:02:50.934292 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:50.934256    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:02:50.934292 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:50.934285    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fqsx\" (UniqueName: \"kubernetes.io/projected/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-kube-api-access-8fqsx\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:02:50.934292 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:50.934297    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:02:51.572535 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:51.572504    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"
Nov 03 03:02:51.572755 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:51.572538    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc" event={"ID":"a5681e87-fbf6-4a09-87cd-90d3d4d03a1c","Type":"ContainerDied","Data":"45afd3d9451e4ed3ddfdfb37970977e6e3374a035a58ed1840940961d2e2fd5e"}
Nov 03 03:02:51.572755 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:02:51.572584    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45afd3d9451e4ed3ddfdfb37970977e6e3374a035a58ed1840940961d2e2fd5e"
Nov 03 03:03:13.813582 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:03:13.813543    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:03:14.492778 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:14.492745    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:03:24.902067 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:24.902030    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"]
Nov 03 03:03:24.902674 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:24.902654    5508 cpu_manager.go:414] "containerMap: removing container" podUID="a5681e87-fbf6-4a09-87cd-90d3d4d03a1c" containerName="revoke-keys"
Nov 03 03:03:24.902764 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:24.902677    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5681e87-fbf6-4a09-87cd-90d3d4d03a1c" containerName="revoke-keys"
Nov 03 03:03:24.902821 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:24.902782    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5681e87-fbf6-4a09-87cd-90d3d4d03a1c" containerName="revoke-keys"
Nov 03 03:03:24.906724 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:24.906700    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"
Nov 03 03:03:24.932623 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:24.932589    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22707a93-24c4-490e-8d79-f73610306d08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b5nlx\" (UID: \"22707a93-24c4-490e-8d79-f73610306d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"
Nov 03 03:03:24.932917 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:24.932752    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qzjx\" (UniqueName: \"kubernetes.io/projected/22707a93-24c4-490e-8d79-f73610306d08-kube-api-access-5qzjx\") pod \"maas-api-revoke-sub-e2fc9c5b-b5nlx\" (UID: \"22707a93-24c4-490e-8d79-f73610306d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"
Nov 03 03:03:24.932917 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:24.932799    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22707a93-24c4-490e-8d79-f73610306d08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b5nlx\" (UID: \"22707a93-24c4-490e-8d79-f73610306d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"
Nov 03 03:03:24.955453 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:24.955423    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:03:24.955453 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:24.955425    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:03:24.955680 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:24.955440    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:03:25.001751 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:25.001712    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:03:25.006165 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:25.006136    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"]
Nov 03 03:03:25.033618 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:25.033586    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22707a93-24c4-490e-8d79-f73610306d08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b5nlx\" (UID: \"22707a93-24c4-490e-8d79-f73610306d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"
Nov 03 03:03:25.033790 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:25.033668    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qzjx\" (UniqueName: \"kubernetes.io/projected/22707a93-24c4-490e-8d79-f73610306d08-kube-api-access-5qzjx\") pod \"maas-api-revoke-sub-e2fc9c5b-b5nlx\" (UID: \"22707a93-24c4-490e-8d79-f73610306d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"
Nov 03 03:03:25.033790 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:25.033704    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22707a93-24c4-490e-8d79-f73610306d08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b5nlx\" (UID: \"22707a93-24c4-490e-8d79-f73610306d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"
Nov 03 03:03:25.034374 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:25.034332    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22707a93-24c4-490e-8d79-f73610306d08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b5nlx\" (UID: \"22707a93-24c4-490e-8d79-f73610306d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"
Nov 03 03:03:25.038753 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:25.036032    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22707a93-24c4-490e-8d79-f73610306d08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-b5nlx\" (UID: \"22707a93-24c4-490e-8d79-f73610306d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"
Nov 03 03:03:25.091003 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:25.090966    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qzjx\" (UniqueName: \"kubernetes.io/projected/22707a93-24c4-490e-8d79-f73610306d08-kube-api-access-5qzjx\") pod \"maas-api-revoke-sub-e2fc9c5b-b5nlx\" (UID: \"22707a93-24c4-490e-8d79-f73610306d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"
Nov 03 03:03:25.219169 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:25.219084    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"
Nov 03 03:03:25.595601 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:25.595524    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"]
Nov 03 03:03:25.602942 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:03:25.602911    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22707a93_24c4_490e_8d79_f73610306d08.slice/crio-58051fb23fc8bff0a30b0cdf588e3054d733320db263fb29368a5d89cd30db43 WatchSource:0}: Error finding container 58051fb23fc8bff0a30b0cdf588e3054d733320db263fb29368a5d89cd30db43: Status 404 returned error can't find the container with id 58051fb23fc8bff0a30b0cdf588e3054d733320db263fb29368a5d89cd30db43
Nov 03 03:03:25.790783 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:25.790753    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx" event={"ID":"22707a93-24c4-490e-8d79-f73610306d08","Type":"ContainerStarted","Data":"58051fb23fc8bff0a30b0cdf588e3054d733320db263fb29368a5d89cd30db43"}
Nov 03 03:03:26.797334 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:26.797229    5508 generic.go:356] "Generic (PLEG): container finished" podID="22707a93-24c4-490e-8d79-f73610306d08" containerID="d4b7c3644c28e306c2f213e87fc04312d926fd89201a3ab0e81a7c84976821b2" exitCode=0
Nov 03 03:03:26.797737 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:26.797334    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx" event={"ID":"22707a93-24c4-490e-8d79-f73610306d08","Type":"ContainerDied","Data":"d4b7c3644c28e306c2f213e87fc04312d926fd89201a3ab0e81a7c84976821b2"}
Nov 03 03:03:27.933795 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:27.933769    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"
Nov 03 03:03:27.960989 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:27.960964    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22707a93-24c4-490e-8d79-f73610306d08-kube-api-access-5qzjx\" (UniqueName: \"kubernetes.io/projected/22707a93-24c4-490e-8d79-f73610306d08-kube-api-access-5qzjx\") pod \"22707a93-24c4-490e-8d79-f73610306d08\" (UID: \"22707a93-24c4-490e-8d79-f73610306d08\") "
Nov 03 03:03:27.961128 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:27.960998    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22707a93-24c4-490e-8d79-f73610306d08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22707a93-24c4-490e-8d79-f73610306d08-maas-api-openshift-ca\") pod \"22707a93-24c4-490e-8d79-f73610306d08\" (UID: \"22707a93-24c4-490e-8d79-f73610306d08\") "
Nov 03 03:03:27.961128 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:27.961055    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22707a93-24c4-490e-8d79-f73610306d08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22707a93-24c4-490e-8d79-f73610306d08-maas-api-xks-ca\") pod \"22707a93-24c4-490e-8d79-f73610306d08\" (UID: \"22707a93-24c4-490e-8d79-f73610306d08\") "
Nov 03 03:03:27.961418 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:27.961395    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22707a93-24c4-490e-8d79-f73610306d08-maas-api-openshift-ca" pod "22707a93-24c4-490e-8d79-f73610306d08" (UID: "22707a93-24c4-490e-8d79-f73610306d08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:03:27.961512 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:27.961500    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22707a93-24c4-490e-8d79-f73610306d08-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:03:27.963059 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:27.963039    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22707a93-24c4-490e-8d79-f73610306d08-maas-api-xks-ca" pod "22707a93-24c4-490e-8d79-f73610306d08" (UID: "22707a93-24c4-490e-8d79-f73610306d08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:03:27.963110 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:27.963064    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22707a93-24c4-490e-8d79-f73610306d08-kube-api-access-5qzjx" pod "22707a93-24c4-490e-8d79-f73610306d08" (UID: "22707a93-24c4-490e-8d79-f73610306d08"). InnerVolumeSpecName "kube-api-access-5qzjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:03:28.062898 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:28.062825    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qzjx\" (UniqueName: \"kubernetes.io/projected/22707a93-24c4-490e-8d79-f73610306d08-kube-api-access-5qzjx\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:03:28.062898 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:28.062857    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22707a93-24c4-490e-8d79-f73610306d08-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:03:28.810238 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:28.810200    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx" event={"ID":"22707a93-24c4-490e-8d79-f73610306d08","Type":"ContainerDied","Data":"58051fb23fc8bff0a30b0cdf588e3054d733320db263fb29368a5d89cd30db43"}
Nov 03 03:03:28.810238 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:28.810225    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"
Nov 03 03:03:28.810494 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:28.810254    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58051fb23fc8bff0a30b0cdf588e3054d733320db263fb29368a5d89cd30db43"
Nov 03 03:03:35.748316 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.748278    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"]
Nov 03 03:03:35.748788 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.748772    5508 cpu_manager.go:414] "containerMap: removing container" podUID="22707a93-24c4-490e-8d79-f73610306d08" containerName="revoke-keys"
Nov 03 03:03:35.748843 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.748790    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22707a93-24c4-490e-8d79-f73610306d08" containerName="revoke-keys"
Nov 03 03:03:35.748906 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.748897    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="22707a93-24c4-490e-8d79-f73610306d08" containerName="revoke-keys"
Nov 03 03:03:35.752402 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.752379    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"
Nov 03 03:03:35.801492 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.801461    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:03:35.838823 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.838790    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl7qj\" (UniqueName: \"kubernetes.io/projected/70559420-e3ed-4cc1-ac36-17ed98066b74-kube-api-access-sl7qj\") pod \"maas-api-revoke-sub-5a29efb3-j24kn\" (UID: \"70559420-e3ed-4cc1-ac36-17ed98066b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"
Nov 03 03:03:35.839010 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.838854    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-j24kn\" (UID: \"70559420-e3ed-4cc1-ac36-17ed98066b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"
Nov 03 03:03:35.839010 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.838969    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-j24kn\" (UID: \"70559420-e3ed-4cc1-ac36-17ed98066b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"
Nov 03 03:03:35.847715 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.847685    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:03:35.847933 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.847694    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:03:35.848047 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.847740    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:03:35.850452 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.850413    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"]
Nov 03 03:03:35.939819 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.939770    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-j24kn\" (UID: \"70559420-e3ed-4cc1-ac36-17ed98066b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"
Nov 03 03:03:35.940024 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.939843    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl7qj\" (UniqueName: \"kubernetes.io/projected/70559420-e3ed-4cc1-ac36-17ed98066b74-kube-api-access-sl7qj\") pod \"maas-api-revoke-sub-5a29efb3-j24kn\" (UID: \"70559420-e3ed-4cc1-ac36-17ed98066b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"
Nov 03 03:03:35.940024 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.939889    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-j24kn\" (UID: \"70559420-e3ed-4cc1-ac36-17ed98066b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"
Nov 03 03:03:35.940547 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.940526    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-j24kn\" (UID: \"70559420-e3ed-4cc1-ac36-17ed98066b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"
Nov 03 03:03:35.942079 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.942058    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-j24kn\" (UID: \"70559420-e3ed-4cc1-ac36-17ed98066b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"
Nov 03 03:03:35.997389 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:35.997358    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl7qj\" (UniqueName: \"kubernetes.io/projected/70559420-e3ed-4cc1-ac36-17ed98066b74-kube-api-access-sl7qj\") pod \"maas-api-revoke-sub-5a29efb3-j24kn\" (UID: \"70559420-e3ed-4cc1-ac36-17ed98066b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"
Nov 03 03:03:36.065641 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:36.065540    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"
Nov 03 03:03:36.444533 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:36.444499    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"]
Nov 03 03:03:36.453621 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:03:36.453596    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70559420_e3ed_4cc1_ac36_17ed98066b74.slice/crio-c052f7c2a292f6525dae3030033b691b59c5ca3c578fa99b6def0308b962361a WatchSource:0}: Error finding container c052f7c2a292f6525dae3030033b691b59c5ca3c578fa99b6def0308b962361a: Status 404 returned error can't find the container with id c052f7c2a292f6525dae3030033b691b59c5ca3c578fa99b6def0308b962361a
Nov 03 03:03:36.855433 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:36.855396    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn" event={"ID":"70559420-e3ed-4cc1-ac36-17ed98066b74","Type":"ContainerStarted","Data":"1f573fb6eb8d413a07b1ef2ae90861aef19a41b3de22182d7e05e61090e7e24b"}
Nov 03 03:03:36.855836 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:36.855453    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn" event={"ID":"70559420-e3ed-4cc1-ac36-17ed98066b74","Type":"ContainerStarted","Data":"c052f7c2a292f6525dae3030033b691b59c5ca3c578fa99b6def0308b962361a"}
Nov 03 03:03:36.964825 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:36.964772    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn" podStartSLOduration=1.96475932 podStartE2EDuration="1.96475932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:03:36.964594205 +0000 UTC m=+6296.390200668" watchObservedRunningTime="2026-11-03 03:03:36.96475932 +0000 UTC m=+6296.390365784"
Nov 03 03:03:37.862926 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:37.862895    5508 generic.go:356] "Generic (PLEG): container finished" podID="70559420-e3ed-4cc1-ac36-17ed98066b74" containerID="1f573fb6eb8d413a07b1ef2ae90861aef19a41b3de22182d7e05e61090e7e24b" exitCode=0
Nov 03 03:03:37.863367 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:37.862975    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn" event={"ID":"70559420-e3ed-4cc1-ac36-17ed98066b74","Type":"ContainerDied","Data":"1f573fb6eb8d413a07b1ef2ae90861aef19a41b3de22182d7e05e61090e7e24b"}
Nov 03 03:03:37.880917 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:37.880889    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:03:39.091411 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:39.091058    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"
Nov 03 03:03:39.180235 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:39.176716    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70559420-e3ed-4cc1-ac36-17ed98066b74-kube-api-access-sl7qj\" (UniqueName: \"kubernetes.io/projected/70559420-e3ed-4cc1-ac36-17ed98066b74-kube-api-access-sl7qj\") pod \"70559420-e3ed-4cc1-ac36-17ed98066b74\" (UID: \"70559420-e3ed-4cc1-ac36-17ed98066b74\") "
Nov 03 03:03:39.180235 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:39.176835    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-openshift-ca\") pod \"70559420-e3ed-4cc1-ac36-17ed98066b74\" (UID: \"70559420-e3ed-4cc1-ac36-17ed98066b74\") "
Nov 03 03:03:39.180235 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:39.176904    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-xks-ca\") pod \"70559420-e3ed-4cc1-ac36-17ed98066b74\" (UID: \"70559420-e3ed-4cc1-ac36-17ed98066b74\") "
Nov 03 03:03:39.199821 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:39.189811    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-xks-ca" pod "70559420-e3ed-4cc1-ac36-17ed98066b74" (UID: "70559420-e3ed-4cc1-ac36-17ed98066b74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:03:39.199821 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:39.191590    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-openshift-ca" pod "70559420-e3ed-4cc1-ac36-17ed98066b74" (UID: "70559420-e3ed-4cc1-ac36-17ed98066b74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:03:39.220236 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:39.220176    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70559420-e3ed-4cc1-ac36-17ed98066b74-kube-api-access-sl7qj" pod "70559420-e3ed-4cc1-ac36-17ed98066b74" (UID: "70559420-e3ed-4cc1-ac36-17ed98066b74"). InnerVolumeSpecName "kube-api-access-sl7qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:03:39.283117 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:39.283021    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl7qj\" (UniqueName: \"kubernetes.io/projected/70559420-e3ed-4cc1-ac36-17ed98066b74-kube-api-access-sl7qj\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:03:39.283117 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:39.283067    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:03:39.283117 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:39.283086    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70559420-e3ed-4cc1-ac36-17ed98066b74-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:03:39.888542 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:39.888493    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn" event={"ID":"70559420-e3ed-4cc1-ac36-17ed98066b74","Type":"ContainerDied","Data":"c052f7c2a292f6525dae3030033b691b59c5ca3c578fa99b6def0308b962361a"}
Nov 03 03:03:39.888858 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:39.888838    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c052f7c2a292f6525dae3030033b691b59c5ca3c578fa99b6def0308b962361a"
Nov 03 03:03:39.898431 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:39.895476    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"
Nov 03 03:03:41.823206 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:41.823176    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 03:03:41.952763 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:41.952739    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 03:03:43.811910 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:03:43.811859    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:03:48.492399 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:48.492289    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:03:53.001099 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.001068    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"]
Nov 03 03:03:53.047229 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.047201    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d72rc"]
Nov 03 03:03:53.127375 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.127330    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"]
Nov 03 03:03:53.127772 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.127760    5508 cpu_manager.go:414] "containerMap: removing container" podUID="70559420-e3ed-4cc1-ac36-17ed98066b74" containerName="revoke-keys"
Nov 03 03:03:53.127817 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.127774    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70559420-e3ed-4cc1-ac36-17ed98066b74" containerName="revoke-keys"
Nov 03 03:03:53.127859 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.127844    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="70559420-e3ed-4cc1-ac36-17ed98066b74" containerName="revoke-keys"
Nov 03 03:03:53.131723 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.131703    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"
Nov 03 03:03:53.180929 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.180891    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:03:53.180929 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.180896    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:03:53.181126 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.180937    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:03:53.181126 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.180942    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:03:53.189151 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.189127    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"]
Nov 03 03:03:53.275416 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.275313    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nll4v\" (UID: \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"
Nov 03 03:03:53.275590 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.275474    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wcwz\" (UniqueName: \"kubernetes.io/projected/f3784ffe-1276-4125-81ef-7eabc97f3b1d-kube-api-access-7wcwz\") pod \"maas-api-revoke-sub-3d4156f1-nll4v\" (UID: \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"
Nov 03 03:03:53.275590 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.275519    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nll4v\" (UID: \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"
Nov 03 03:03:53.376188 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.376148    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nll4v\" (UID: \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"
Nov 03 03:03:53.376417 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.376278    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nll4v\" (UID: \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"
Nov 03 03:03:53.376493 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.376433    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wcwz\" (UniqueName: \"kubernetes.io/projected/f3784ffe-1276-4125-81ef-7eabc97f3b1d-kube-api-access-7wcwz\") pod \"maas-api-revoke-sub-3d4156f1-nll4v\" (UID: \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"
Nov 03 03:03:53.377129 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.377080    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nll4v\" (UID: \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"
Nov 03 03:03:53.379156 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.379135    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nll4v\" (UID: \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"
Nov 03 03:03:53.434430 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.434401    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wcwz\" (UniqueName: \"kubernetes.io/projected/f3784ffe-1276-4125-81ef-7eabc97f3b1d-kube-api-access-7wcwz\") pod \"maas-api-revoke-sub-3d4156f1-nll4v\" (UID: \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"
Nov 03 03:03:53.445251 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.445211    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"
Nov 03 03:03:53.819274 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:53.819237    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"]
Nov 03 03:03:53.825537 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:03:53.825507    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3784ffe_1276_4125_81ef_7eabc97f3b1d.slice/crio-329033122b5178118e1c6d6a86c1ed5cf469404e9be3b35e93636159a68baf1c WatchSource:0}: Error finding container 329033122b5178118e1c6d6a86c1ed5cf469404e9be3b35e93636159a68baf1c: Status 404 returned error can't find the container with id 329033122b5178118e1c6d6a86c1ed5cf469404e9be3b35e93636159a68baf1c
Nov 03 03:03:54.007341 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:54.007305    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v" event={"ID":"f3784ffe-1276-4125-81ef-7eabc97f3b1d","Type":"ContainerStarted","Data":"329033122b5178118e1c6d6a86c1ed5cf469404e9be3b35e93636159a68baf1c"}
Nov 03 03:03:54.053145 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:54.053115    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"]
Nov 03 03:03:54.055651 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:54.055626    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-86rh4"]
Nov 03 03:03:54.690153 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:54.690124    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb0b90a-1693-4b5e-b42b-e4ee8189d179" path="/var/lib/kubelet/pods/6bb0b90a-1693-4b5e-b42b-e4ee8189d179/volumes"
Nov 03 03:03:54.690550 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:54.690536    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5681e87-fbf6-4a09-87cd-90d3d4d03a1c" path="/var/lib/kubelet/pods/a5681e87-fbf6-4a09-87cd-90d3d4d03a1c/volumes"
Nov 03 03:03:55.013843 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:55.013753    5508 generic.go:356] "Generic (PLEG): container finished" podID="f3784ffe-1276-4125-81ef-7eabc97f3b1d" containerID="bead43e8ddf871b5e5657b4311ba034be3e4c141cbd3a05dcd59d6222614d993" exitCode=0
Nov 03 03:03:55.013843 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:55.013798    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v" event={"ID":"f3784ffe-1276-4125-81ef-7eabc97f3b1d","Type":"ContainerDied","Data":"bead43e8ddf871b5e5657b4311ba034be3e4c141cbd3a05dcd59d6222614d993"}
Nov 03 03:03:56.159507 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:56.159480    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"
Nov 03 03:03:56.306376 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:56.306271    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3784ffe-1276-4125-81ef-7eabc97f3b1d-kube-api-access-7wcwz\" (UniqueName: \"kubernetes.io/projected/f3784ffe-1276-4125-81ef-7eabc97f3b1d-kube-api-access-7wcwz\") pod \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\" (UID: \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\") "
Nov 03 03:03:56.306578 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:56.306423    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-openshift-ca\") pod \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\" (UID: \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\") "
Nov 03 03:03:56.306578 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:56.306465    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-xks-ca\") pod \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\" (UID: \"f3784ffe-1276-4125-81ef-7eabc97f3b1d\") "
Nov 03 03:03:56.306786 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:56.306763    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-openshift-ca" pod "f3784ffe-1276-4125-81ef-7eabc97f3b1d" (UID: "f3784ffe-1276-4125-81ef-7eabc97f3b1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:03:56.308467 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:56.308444    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-xks-ca" pod "f3784ffe-1276-4125-81ef-7eabc97f3b1d" (UID: "f3784ffe-1276-4125-81ef-7eabc97f3b1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:03:56.308585 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:56.308558    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3784ffe-1276-4125-81ef-7eabc97f3b1d-kube-api-access-7wcwz" pod "f3784ffe-1276-4125-81ef-7eabc97f3b1d" (UID: "f3784ffe-1276-4125-81ef-7eabc97f3b1d"). InnerVolumeSpecName "kube-api-access-7wcwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:03:56.407853 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:56.407820    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:03:56.407853 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:56.407851    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3784ffe-1276-4125-81ef-7eabc97f3b1d-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:03:56.408061 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:56.407866    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wcwz\" (UniqueName: \"kubernetes.io/projected/f3784ffe-1276-4125-81ef-7eabc97f3b1d-kube-api-access-7wcwz\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:03:57.025775 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:57.025694    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"
Nov 03 03:03:57.025775 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:57.025736    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v" event={"ID":"f3784ffe-1276-4125-81ef-7eabc97f3b1d","Type":"ContainerDied","Data":"329033122b5178118e1c6d6a86c1ed5cf469404e9be3b35e93636159a68baf1c"}
Nov 03 03:03:57.025972 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:03:57.025789    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="329033122b5178118e1c6d6a86c1ed5cf469404e9be3b35e93636159a68baf1c"
Nov 03 03:04:09.746399 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.746361    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"]
Nov 03 03:04:09.747001 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.746853    5508 cpu_manager.go:414] "containerMap: removing container" podUID="f3784ffe-1276-4125-81ef-7eabc97f3b1d" containerName="revoke-keys"
Nov 03 03:04:09.747001 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.746869    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3784ffe-1276-4125-81ef-7eabc97f3b1d" containerName="revoke-keys"
Nov 03 03:04:09.747001 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.746960    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3784ffe-1276-4125-81ef-7eabc97f3b1d" containerName="revoke-keys"
Nov 03 03:04:09.750297 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.750280    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.799445 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.799397    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-9fbc1b-openshift-default-dockercfg-x9gvr\""
Nov 03 03:04:09.827206 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.827146    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-certs\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.827206 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.827192    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-socket\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.827451 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.827323    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-envoy\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.827451 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.827382    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-token\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.827572 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.827476    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istiod-ca-cert\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.827572 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.827552    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-data\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.827666 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.827599    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcjdc\" (UniqueName: \"kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-kube-api-access-hcjdc\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.827666 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.827645    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-credential-socket\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.827748 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.827673    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-podinfo\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.853031 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.852997    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"]
Nov 03 03:04:09.928545 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.928494    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-envoy\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.928754 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.928561    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-token\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.928754 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.928595    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istiod-ca-cert\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.928754 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.928635    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-data\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.928754 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.928667    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjdc\" (UniqueName: \"kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-kube-api-access-hcjdc\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.928754 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.928721    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-credential-socket\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.928754 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.928755    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-podinfo\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.929125 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.928860    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-certs\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.929125 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.928888    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-socket\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.931383 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.929416    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-socket\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.931589 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.931277    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-credential-socket\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.931755 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.931528    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-data\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.931872 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.931758    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-certs\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.931872 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.931696    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istiod-ca-cert\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.936910 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.936874    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-envoy\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.943709 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.943682    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-podinfo\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.987646 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.987614    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-token\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:09.987802 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:09.987648    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcjdc\" (UniqueName: \"kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-kube-api-access-hcjdc\") pod \"e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") " pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:10.063545 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:10.063454    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:10.492080 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:10.492044    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"]
Nov 03 03:04:10.499164 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:04:10.499134    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e5dd3c_21ec_4786_ae03_49a41fb8e9ac.slice/crio-6427433fe750f449234ab512b77d716ff0554ef48a5a94488d555487214e7c19 WatchSource:0}: Error finding container 6427433fe750f449234ab512b77d716ff0554ef48a5a94488d555487214e7c19: Status 404 returned error can't find the container with id 6427433fe750f449234ab512b77d716ff0554ef48a5a94488d555487214e7c19
Nov 03 03:04:10.501433 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:10.501403    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 03:04:10.501527 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:10.501466    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 03:04:10.501527 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:10.501494    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 03:04:11.108089 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:11.108050    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg" event={"ID":"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac","Type":"ContainerStarted","Data":"a3f2952c1431422339f54dd909ca8c31ea4dd65bf116bc4125e86f08a7a96639"}
Nov 03 03:04:11.108518 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:11.108108    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg" event={"ID":"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac","Type":"ContainerStarted","Data":"6427433fe750f449234ab512b77d716ff0554ef48a5a94488d555487214e7c19"}
Nov 03 03:04:11.222468 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:11.222399    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg" podStartSLOduration=2.222382726 podStartE2EDuration="2.222382726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:04:11.221205831 +0000 UTC m=+6330.646812313" watchObservedRunningTime="2026-11-03 03:04:11.222382726 +0000 UTC m=+6330.647989207"
Nov 03 03:04:12.063808 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:12.063774    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:12.068939 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:12.068916    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:12.114277 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:12.114247    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:12.115283 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:12.115261    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:13.811247 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:04:13.811216    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:04:21.085635 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.085602    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"]
Nov 03 03:04:21.089171 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.089150    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:21.138202 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.138165    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-9fbc1b-serving-cert\""
Nov 03 03:04:21.150908 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.150873    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:21.151085 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.151031    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:21.151085 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.151082    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661b3f55-0289-4ea1-8f85-3ca805f3a699-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:21.151232 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.151201    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8thg\" (UniqueName: \"kubernetes.io/projected/661b3f55-0289-4ea1-8f85-3ca805f3a699-kube-api-access-n8thg\") pod \"maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:21.184667 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.184637    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 03:04:21.184667 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.184637    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:04:21.184899 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.184685    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:04:21.184899 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.184704    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sxqdt\""
Nov 03 03:04:21.187489 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.187470    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"]
Nov 03 03:04:21.252548 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.252512    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:21.252745 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.252564    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661b3f55-0289-4ea1-8f85-3ca805f3a699-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:21.252745 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.252617    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8thg\" (UniqueName: \"kubernetes.io/projected/661b3f55-0289-4ea1-8f85-3ca805f3a699-kube-api-access-n8thg\") pod \"maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:21.252745 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:04:21.252670    5508 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-serving-cert: secret "maas-api-e2e-worker-single-main-9fbc1b-serving-cert" not found
Nov 03 03:04:21.252745 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.252685    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:21.252745 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:04:21.252741    5508 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-maas-api-tls podName:661b3f55-0289-4ea1-8f85-3ca805f3a699 nodeName:}" failed. No retries permitted until 2026-11-03 03:04:21.752718798 +0000 UTC m=+6341.178325254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-maas-api-tls") pod "maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc" (UID: "661b3f55-0289-4ea1-8f85-3ca805f3a699") : secret "maas-api-e2e-worker-single-main-9fbc1b-serving-cert" not found
Nov 03 03:04:21.253387 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.253368    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661b3f55-0289-4ea1-8f85-3ca805f3a699-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:21.255073 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.255052    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-metrics-certs\") pod \"maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:21.311095 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.311060    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8thg\" (UniqueName: \"kubernetes.io/projected/661b3f55-0289-4ea1-8f85-3ca805f3a699-kube-api-access-n8thg\") pod \"maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:21.759966 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.759929    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:21.762628 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:21.762601    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:22.003139 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:22.003108    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:22.375918 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:22.375830    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"]
Nov 03 03:04:22.383159 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:04:22.383130    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661b3f55_0289_4ea1_8f85_3ca805f3a699.slice/crio-61e7283ae8e784bd538b85cd513cfa44ca3c5bb1425d53118a567335e5c3f2b0 WatchSource:0}: Error finding container 61e7283ae8e784bd538b85cd513cfa44ca3c5bb1425d53118a567335e5c3f2b0: Status 404 returned error can't find the container with id 61e7283ae8e784bd538b85cd513cfa44ca3c5bb1425d53118a567335e5c3f2b0
Nov 03 03:04:22.384576 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:22.384559    5508 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 03:04:23.180992 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:23.180955    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc" event={"ID":"661b3f55-0289-4ea1-8f85-3ca805f3a699","Type":"ContainerStarted","Data":"61e7283ae8e784bd538b85cd513cfa44ca3c5bb1425d53118a567335e5c3f2b0"}
Nov 03 03:04:24.187938 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:24.187902    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc" event={"ID":"661b3f55-0289-4ea1-8f85-3ca805f3a699","Type":"ContainerStarted","Data":"8d2bb8ecc78ead1845fd8febb2d34c216921079a97b61c4995c94a3ce0161a6e"}
Nov 03 03:04:24.188454 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:24.187977    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:24.303289 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:24.303216    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc" podStartSLOduration=2.315843821 podStartE2EDuration="3.303199674s" totalImagesPullingTime="987.355853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:04:24.302663636 +0000 UTC m=+6343.728270119" watchObservedRunningTime="2026-11-03 03:04:24.303199674 +0000 UTC m=+6343.728806139"
Nov 03 03:04:25.047530 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:25.047492    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"]
Nov 03 03:04:25.101586 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:25.101544    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q27pt"]
Nov 03 03:04:26.690928 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:26.690898    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5806ea91-8e4d-48d0-96a4-75eaea67f942" path="/var/lib/kubelet/pods/5806ea91-8e4d-48d0-96a4-75eaea67f942/volumes"
Nov 03 03:04:29.280447 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:29.280413    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:04:30.201046 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:30.201015    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:04:38.272162 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:38.272124    5508 generic.go:356] "Generic (PLEG): container finished" podID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerID="a3f2952c1431422339f54dd909ca8c31ea4dd65bf116bc4125e86f08a7a96639" exitCode=0
Nov 03 03:04:38.272603 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:38.272195    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg" event={"ID":"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac","Type":"ContainerDied","Data":"a3f2952c1431422339f54dd909ca8c31ea4dd65bf116bc4125e86f08a7a96639"}
Nov 03 03:04:38.272603 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:38.272597    5508 scope.go:122] "RemoveContainer" containerID="a3f2952c1431422339f54dd909ca8c31ea4dd65bf116bc4125e86f08a7a96639"
Nov 03 03:04:38.273961 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:38.273922    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 03:04:38.274063 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:38.274015    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 03:04:38.274102 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:38.274064    5508 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 03:04:39.281410 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:39.281377    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg" event={"ID":"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac","Type":"ContainerStarted","Data":"47e89dcbf105adf12233c3ec39c84f4e55326a7781ebab2d9186cb5d75928f25"}
Nov 03 03:04:40.066373 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:40.065717    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:40.070507 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:40.070466    5508 patch_prober.go:29] interesting pod/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.153:15021/healthz/ready\": dial tcp 10.133.1.153:15021: connect: connection refused" start-of-body=
Nov 03 03:04:40.070808 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:40.070764    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg" podUID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.153:15021/healthz/ready\": dial tcp 10.133.1.153:15021: connect: connection refused"
Nov 03 03:04:41.069374 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:41.066790    5508 patch_prober.go:29] interesting pod/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.153:15021/healthz/ready\": dial tcp 10.133.1.153:15021: connect: connection refused" start-of-body=
Nov 03 03:04:41.069374 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:41.066856    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg" podUID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.153:15021/healthz/ready\": dial tcp 10.133.1.153:15021: connect: connection refused"
Nov 03 03:04:42.070376 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:42.065795    5508 patch_prober.go:29] interesting pod/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.153:15021/healthz/ready\": dial tcp 10.133.1.153:15021: connect: connection refused" start-of-body=
Nov 03 03:04:42.070376 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:42.065862    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg" podUID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.153:15021/healthz/ready\": dial tcp 10.133.1.153:15021: connect: connection refused"
Nov 03 03:04:42.114705 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:42.114673    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:43.067236 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:43.063873    5508 patch_prober.go:29] interesting pod/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.153:15021/healthz/ready\": dial tcp 10.133.1.153:15021: connect: connection refused" start-of-body=
Nov 03 03:04:43.067236 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:43.063937    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg" podUID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.153:15021/healthz/ready\": dial tcp 10.133.1.153:15021: connect: connection refused"
Nov 03 03:04:43.811620 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:04:43.811582    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:04:44.064248 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:44.064149    5508 patch_prober.go:29] interesting pod/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.153:15021/healthz/ready\": dial tcp 10.133.1.153:15021: connect: connection refused" start-of-body=
Nov 03 03:04:44.064248 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:44.064206    5508 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg" podUID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.153:15021/healthz/ready\": dial tcp 10.133.1.153:15021: connect: connection refused"
Nov 03 03:04:45.068842 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:45.068807    5508 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:45.069830 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:45.069808    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:04:47.012491 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:47.012451    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:04:49.092511 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:49.092481    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"]
Nov 03 03:04:49.138958 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:49.138925    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-l2z59"]
Nov 03 03:04:50.690904 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:04:50.690870    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b355e0b-03c5-4184-aa4f-930730894895" path="/var/lib/kubelet/pods/4b355e0b-03c5-4184-aa4f-930730894895/volumes"
Nov 03 03:05:02.057421 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:02.057387    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"]
Nov 03 03:05:02.102905 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:02.102875    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-jgdrd"]
Nov 03 03:05:02.690305 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:02.690268    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4642113-07f4-426e-877d-f0443a206990" path="/var/lib/kubelet/pods/b4642113-07f4-426e-877d-f0443a206990/volumes"
Nov 03 03:05:13.811626 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:05:13.811586    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:05:18.204301 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:18.204095    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:05:37.045483 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:37.045448    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"]
Nov 03 03:05:37.091792 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:37.091763    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9nhdg"]
Nov 03 03:05:38.690165 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:38.690128    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66c3d24-bb50-4e81-a2be-1da246e203d7" path="/var/lib/kubelet/pods/b66c3d24-bb50-4e81-a2be-1da246e203d7/volumes"
Nov 03 03:05:43.813075 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:05:43.813039    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:05:47.424776 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:47.424745    5508 scope.go:122] "RemoveContainer" containerID="08bd7ff2cefa930cf0cf8a0c67a26a11f4c2b95cbbea877fe526e28771ed07df"
Nov 03 03:05:47.434407 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:47.434386    5508 scope.go:122] "RemoveContainer" containerID="536f8e466849bd301390fce246eb612f0d167f4041a08e7e765c7e22cfc464ab"
Nov 03 03:05:47.443461 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:47.443440    5508 scope.go:122] "RemoveContainer" containerID="438b2f60b08093d3b09c4e37460f39817757cad677cb977998e42cbe0d05610b"
Nov 03 03:05:47.570128 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:47.570093    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:05:48.045848 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:48.045813    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"]
Nov 03 03:05:48.092847 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:48.092806    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8zm94"]
Nov 03 03:05:48.690943 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:48.690910    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac" path="/var/lib/kubelet/pods/290d0eda-4ddb-49b3-8b3e-4d5495e2e3ac/volumes"
Nov 03 03:05:56.804380 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:56.804334    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"]
Nov 03 03:05:56.804846 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:56.804565    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc" podUID="661b3f55-0289-4ea1-8f85-3ca805f3a699" containerName="maas-api" containerID="cri-o://8d2bb8ecc78ead1845fd8febb2d34c216921079a97b61c4995c94a3ce0161a6e" gracePeriod=30
Nov 03 03:05:58.052167 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.052141    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:05:58.094589 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.094557    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661b3f55-0289-4ea1-8f85-3ca805f3a699-kube-api-access-n8thg\" (UniqueName: \"kubernetes.io/projected/661b3f55-0289-4ea1-8f85-3ca805f3a699-kube-api-access-n8thg\") pod \"661b3f55-0289-4ea1-8f85-3ca805f3a699\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") "
Nov 03 03:05:58.094780 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.094672    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661b3f55-0289-4ea1-8f85-3ca805f3a699-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661b3f55-0289-4ea1-8f85-3ca805f3a699-openshift-service-ca\") pod \"661b3f55-0289-4ea1-8f85-3ca805f3a699\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") "
Nov 03 03:05:58.094780 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.094705    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-metrics-certs\" (UniqueName: \"kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-metrics-certs\") pod \"661b3f55-0289-4ea1-8f85-3ca805f3a699\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") "
Nov 03 03:05:58.094880 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.094785    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-maas-api-tls\") pod \"661b3f55-0289-4ea1-8f85-3ca805f3a699\" (UID: \"661b3f55-0289-4ea1-8f85-3ca805f3a699\") "
Nov 03 03:05:58.094970 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.094942    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661b3f55-0289-4ea1-8f85-3ca805f3a699-openshift-service-ca" pod "661b3f55-0289-4ea1-8f85-3ca805f3a699" (UID: "661b3f55-0289-4ea1-8f85-3ca805f3a699"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:05:58.095129 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.095112    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/661b3f55-0289-4ea1-8f85-3ca805f3a699-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:05:58.096849 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.096819    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-maas-api-tls" pod "661b3f55-0289-4ea1-8f85-3ca805f3a699" (UID: "661b3f55-0289-4ea1-8f85-3ca805f3a699"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:05:58.096960 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.096857    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661b3f55-0289-4ea1-8f85-3ca805f3a699-kube-api-access-n8thg" pod "661b3f55-0289-4ea1-8f85-3ca805f3a699" (UID: "661b3f55-0289-4ea1-8f85-3ca805f3a699"). InnerVolumeSpecName "kube-api-access-n8thg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:05:58.096960 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.096894    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-metrics-certs" pod "661b3f55-0289-4ea1-8f85-3ca805f3a699" (UID: "661b3f55-0289-4ea1-8f85-3ca805f3a699"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:05:58.195674 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.195638    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8thg\" (UniqueName: \"kubernetes.io/projected/661b3f55-0289-4ea1-8f85-3ca805f3a699-kube-api-access-n8thg\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:05:58.195674 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.195668    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:05:58.195674 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.195680    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/661b3f55-0289-4ea1-8f85-3ca805f3a699-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:05:58.766022 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.765990    5508 generic.go:356] "Generic (PLEG): container finished" podID="661b3f55-0289-4ea1-8f85-3ca805f3a699" containerID="8d2bb8ecc78ead1845fd8febb2d34c216921079a97b61c4995c94a3ce0161a6e" exitCode=0
Nov 03 03:05:58.766221 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.766035    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"
Nov 03 03:05:58.766221 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.766080    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc" event={"ID":"661b3f55-0289-4ea1-8f85-3ca805f3a699","Type":"ContainerDied","Data":"8d2bb8ecc78ead1845fd8febb2d34c216921079a97b61c4995c94a3ce0161a6e"}
Nov 03 03:05:58.766221 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.766144    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc" event={"ID":"661b3f55-0289-4ea1-8f85-3ca805f3a699","Type":"ContainerDied","Data":"61e7283ae8e784bd538b85cd513cfa44ca3c5bb1425d53118a567335e5c3f2b0"}
Nov 03 03:05:58.766221 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.766171    5508 scope.go:122] "RemoveContainer" containerID="8d2bb8ecc78ead1845fd8febb2d34c216921079a97b61c4995c94a3ce0161a6e"
Nov 03 03:05:58.776080 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.776054    5508 scope.go:122] "RemoveContainer" containerID="8d2bb8ecc78ead1845fd8febb2d34c216921079a97b61c4995c94a3ce0161a6e"
Nov 03 03:05:58.776374 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:05:58.776358    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d2bb8ecc78ead1845fd8febb2d34c216921079a97b61c4995c94a3ce0161a6e\": container with ID starting with 8d2bb8ecc78ead1845fd8febb2d34c216921079a97b61c4995c94a3ce0161a6e not found: ID does not exist" containerID="8d2bb8ecc78ead1845fd8febb2d34c216921079a97b61c4995c94a3ce0161a6e"
Nov 03 03:05:58.776440 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.776383    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d2bb8ecc78ead1845fd8febb2d34c216921079a97b61c4995c94a3ce0161a6e"} err="failed to get container status \"8d2bb8ecc78ead1845fd8febb2d34c216921079a97b61c4995c94a3ce0161a6e\": rpc error: code = NotFound desc = could not find container \"8d2bb8ecc78ead1845fd8febb2d34c216921079a97b61c4995c94a3ce0161a6e\": container with ID starting with 8d2bb8ecc78ead1845fd8febb2d34c216921079a97b61c4995c94a3ce0161a6e not found: ID does not exist"
Nov 03 03:05:58.919995 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.919963    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"]
Nov 03 03:05:58.966125 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:05:58.966042    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-9fbc1b-9587859dc-gxznc"]
Nov 03 03:06:00.690772 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:00.690735    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661b3f55-0289-4ea1-8f85-3ca805f3a699" path="/var/lib/kubelet/pods/661b3f55-0289-4ea1-8f85-3ca805f3a699/volumes"
Nov 03 03:06:01.598758 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:01.598714    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:06:13.811862 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:06:13.811815    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:06:23.047639 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:23.047604    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"]
Nov 03 03:06:23.093531 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:23.093497    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-d7lks"]
Nov 03 03:06:24.690835 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:24.690800    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14bb627a-b2a2-4f87-9556-d49475a1af2a" path="/var/lib/kubelet/pods/14bb627a-b2a2-4f87-9556-d49475a1af2a/volumes"
Nov 03 03:06:31.174562 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:31.174529    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"]
Nov 03 03:06:31.175002 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:31.174749    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg" podUID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerName="istio-proxy" containerID="cri-o://47e89dcbf105adf12233c3ec39c84f4e55326a7781ebab2d9186cb5d75928f25" gracePeriod=30
Nov 03 03:06:32.873800 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:32.873758    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"]
Nov 03 03:06:32.919833 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:32.919793    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nll4v"]
Nov 03 03:06:33.003759 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.003727    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"]
Nov 03 03:06:33.004192 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.004180    5508 cpu_manager.go:414] "containerMap: removing container" podUID="661b3f55-0289-4ea1-8f85-3ca805f3a699" containerName="maas-api"
Nov 03 03:06:33.004243 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.004194    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661b3f55-0289-4ea1-8f85-3ca805f3a699" containerName="maas-api"
Nov 03 03:06:33.004282 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.004271    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="661b3f55-0289-4ea1-8f85-3ca805f3a699" containerName="maas-api"
Nov 03 03:06:33.007384 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.007365    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"
Nov 03 03:06:33.056931 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.056902    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:06:33.057089 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.056959    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:06:33.057089 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.056987    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:06:33.057089 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.057023    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:06:33.066898 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.066872    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"]
Nov 03 03:06:33.077760 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.077734    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zslk6\" (UID: \"67489fa9-855b-4b1b-a118-368708d5a5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"
Nov 03 03:06:33.077910 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.077771    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nndzf\" (UniqueName: \"kubernetes.io/projected/67489fa9-855b-4b1b-a118-368708d5a5d1-kube-api-access-nndzf\") pod \"maas-api-revoke-sub-3d4156f1-zslk6\" (UID: \"67489fa9-855b-4b1b-a118-368708d5a5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"
Nov 03 03:06:33.078043 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.077995    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zslk6\" (UID: \"67489fa9-855b-4b1b-a118-368708d5a5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"
Nov 03 03:06:33.179624 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.179586    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zslk6\" (UID: \"67489fa9-855b-4b1b-a118-368708d5a5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"
Nov 03 03:06:33.179820 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.179689    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zslk6\" (UID: \"67489fa9-855b-4b1b-a118-368708d5a5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"
Nov 03 03:06:33.179820 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.179718    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nndzf\" (UniqueName: \"kubernetes.io/projected/67489fa9-855b-4b1b-a118-368708d5a5d1-kube-api-access-nndzf\") pod \"maas-api-revoke-sub-3d4156f1-zslk6\" (UID: \"67489fa9-855b-4b1b-a118-368708d5a5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"
Nov 03 03:06:33.180493 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.180464    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zslk6\" (UID: \"67489fa9-855b-4b1b-a118-368708d5a5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"
Nov 03 03:06:33.182701 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.182661    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zslk6\" (UID: \"67489fa9-855b-4b1b-a118-368708d5a5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"
Nov 03 03:06:33.240536 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.240505    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nndzf\" (UniqueName: \"kubernetes.io/projected/67489fa9-855b-4b1b-a118-368708d5a5d1-kube-api-access-nndzf\") pod \"maas-api-revoke-sub-3d4156f1-zslk6\" (UID: \"67489fa9-855b-4b1b-a118-368708d5a5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"
Nov 03 03:06:33.320526 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.320492    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"
Nov 03 03:06:33.694915 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.694883    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"]
Nov 03 03:06:33.702019 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:06:33.701989    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67489fa9_855b_4b1b_a118_368708d5a5d1.slice/crio-ab04c6205257afdbcbfdf10164bba2d98ecc8a87ed8ca1bc6a1d6946eb615751 WatchSource:0}: Error finding container ab04c6205257afdbcbfdf10164bba2d98ecc8a87ed8ca1bc6a1d6946eb615751: Status 404 returned error can't find the container with id ab04c6205257afdbcbfdf10164bba2d98ecc8a87ed8ca1bc6a1d6946eb615751
Nov 03 03:06:33.972666 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.972638    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6" event={"ID":"67489fa9-855b-4b1b-a118-368708d5a5d1","Type":"ContainerStarted","Data":"454fd3bce7e48896b6f0c71e40ad107ec44c97973b2784cc82a65a8cee407a01"}
Nov 03 03:06:33.972999 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:33.972687    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6" event={"ID":"67489fa9-855b-4b1b-a118-368708d5a5d1","Type":"ContainerStarted","Data":"ab04c6205257afdbcbfdf10164bba2d98ecc8a87ed8ca1bc6a1d6946eb615751"}
Nov 03 03:06:34.082309 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:34.082251    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6" podStartSLOduration=2.082235303 podStartE2EDuration="2.082235303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:06:34.080672092 +0000 UTC m=+6473.506278556" watchObservedRunningTime="2026-11-03 03:06:34.082235303 +0000 UTC m=+6473.507841768"
Nov 03 03:06:34.690659 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:34.690624    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3784ffe-1276-4125-81ef-7eabc97f3b1d" path="/var/lib/kubelet/pods/f3784ffe-1276-4125-81ef-7eabc97f3b1d/volumes"
Nov 03 03:06:34.981432 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:34.981398    5508 generic.go:356] "Generic (PLEG): container finished" podID="67489fa9-855b-4b1b-a118-368708d5a5d1" containerID="454fd3bce7e48896b6f0c71e40ad107ec44c97973b2784cc82a65a8cee407a01" exitCode=0
Nov 03 03:06:34.981902 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:34.981488    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6" event={"ID":"67489fa9-855b-4b1b-a118-368708d5a5d1","Type":"ContainerDied","Data":"454fd3bce7e48896b6f0c71e40ad107ec44c97973b2784cc82a65a8cee407a01"}
Nov 03 03:06:36.125997 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.125969    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"
Nov 03 03:06:36.210910 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.210882    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-xks-ca\") pod \"67489fa9-855b-4b1b-a118-368708d5a5d1\" (UID: \"67489fa9-855b-4b1b-a118-368708d5a5d1\") "
Nov 03 03:06:36.211044 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.210973    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-openshift-ca\") pod \"67489fa9-855b-4b1b-a118-368708d5a5d1\" (UID: \"67489fa9-855b-4b1b-a118-368708d5a5d1\") "
Nov 03 03:06:36.211044 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.211001    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67489fa9-855b-4b1b-a118-368708d5a5d1-kube-api-access-nndzf\" (UniqueName: \"kubernetes.io/projected/67489fa9-855b-4b1b-a118-368708d5a5d1-kube-api-access-nndzf\") pod \"67489fa9-855b-4b1b-a118-368708d5a5d1\" (UID: \"67489fa9-855b-4b1b-a118-368708d5a5d1\") "
Nov 03 03:06:36.211410 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.211378    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-openshift-ca" pod "67489fa9-855b-4b1b-a118-368708d5a5d1" (UID: "67489fa9-855b-4b1b-a118-368708d5a5d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:06:36.213052 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.213033    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-xks-ca" pod "67489fa9-855b-4b1b-a118-368708d5a5d1" (UID: "67489fa9-855b-4b1b-a118-368708d5a5d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:06:36.213490 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.213473    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67489fa9-855b-4b1b-a118-368708d5a5d1-kube-api-access-nndzf" pod "67489fa9-855b-4b1b-a118-368708d5a5d1" (UID: "67489fa9-855b-4b1b-a118-368708d5a5d1"). InnerVolumeSpecName "kube-api-access-nndzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:06:36.312738 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.312706    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:36.312738 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.312737    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67489fa9-855b-4b1b-a118-368708d5a5d1-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:36.312930 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.312751    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nndzf\" (UniqueName: \"kubernetes.io/projected/67489fa9-855b-4b1b-a118-368708d5a5d1-kube-api-access-nndzf\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:36.423789 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.423761    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:06:36.514591 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.514493    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-envoy\") pod \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") "
Nov 03 03:06:36.514591 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.514541    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-certs\") pod \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") "
Nov 03 03:06:36.514591 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.514573    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-kube-api-access-hcjdc\" (UniqueName: \"kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-kube-api-access-hcjdc\") pod \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") "
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.514623    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-data\") pod \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") "
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.514659    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-token\" (UniqueName: \"kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-token\") pod \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") "
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.514734    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-socket\") pod \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") "
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.514781    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istiod-ca-cert\") pod \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") "
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.514840    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-podinfo\") pod \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") "
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.514853    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-certs" pod "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" (UID: "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.514873    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-credential-socket\") pod \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\" (UID: \"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac\") "
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.515040    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-data" pod "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" (UID: "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.515051    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-socket" pod "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" (UID: "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.515174    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istiod-ca-cert" pod "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" (UID: "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.515262    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-credential-socket" pod "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" (UID: "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.515313    5508 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.515333    5508 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-data\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:36.515524 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.515362    5508 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-workload-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:36.516881 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.516862    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-envoy" pod "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" (UID: "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:06:36.517240 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.517214    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-podinfo" pod "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" (UID: "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 03:06:36.517338 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.517309    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-kube-api-access-hcjdc" pod "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" (UID: "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac"). InnerVolumeSpecName "kube-api-access-hcjdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:06:36.517338 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.517319    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-token" pod "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" (UID: "66e5dd3c-21ec-4786-ae03-49a41fb8e9ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:06:36.616718 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.616677    5508 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-envoy\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:36.616718 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.616713    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcjdc\" (UniqueName: \"kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-kube-api-access-hcjdc\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:36.616718 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.616723    5508 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-token\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:36.616984 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.616735    5508 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istiod-ca-cert\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:36.616984 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.616744    5508 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-istio-podinfo\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:36.616984 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.616753    5508 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac-credential-socket\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:36.994907 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.994869    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"
Nov 03 03:06:36.995126 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.994868    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6" event={"ID":"67489fa9-855b-4b1b-a118-368708d5a5d1","Type":"ContainerDied","Data":"ab04c6205257afdbcbfdf10164bba2d98ecc8a87ed8ca1bc6a1d6946eb615751"}
Nov 03 03:06:36.995126 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.995025    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab04c6205257afdbcbfdf10164bba2d98ecc8a87ed8ca1bc6a1d6946eb615751"
Nov 03 03:06:36.996727 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.996697    5508 generic.go:356] "Generic (PLEG): container finished" podID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerID="47e89dcbf105adf12233c3ec39c84f4e55326a7781ebab2d9186cb5d75928f25" exitCode=0
Nov 03 03:06:36.996888 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.996742    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg" event={"ID":"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac","Type":"ContainerDied","Data":"47e89dcbf105adf12233c3ec39c84f4e55326a7781ebab2d9186cb5d75928f25"}
Nov 03 03:06:36.996888 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.996770    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"
Nov 03 03:06:36.996888 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.996789    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg" event={"ID":"66e5dd3c-21ec-4786-ae03-49a41fb8e9ac","Type":"ContainerDied","Data":"6427433fe750f449234ab512b77d716ff0554ef48a5a94488d555487214e7c19"}
Nov 03 03:06:36.996888 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:36.996822    5508 scope.go:122] "RemoveContainer" containerID="47e89dcbf105adf12233c3ec39c84f4e55326a7781ebab2d9186cb5d75928f25"
Nov 03 03:06:37.006555 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:37.006536    5508 scope.go:122] "RemoveContainer" containerID="a3f2952c1431422339f54dd909ca8c31ea4dd65bf116bc4125e86f08a7a96639"
Nov 03 03:06:37.015819 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:37.015802    5508 scope.go:122] "RemoveContainer" containerID="47e89dcbf105adf12233c3ec39c84f4e55326a7781ebab2d9186cb5d75928f25"
Nov 03 03:06:37.016071 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:06:37.016049    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e89dcbf105adf12233c3ec39c84f4e55326a7781ebab2d9186cb5d75928f25\": container with ID starting with 47e89dcbf105adf12233c3ec39c84f4e55326a7781ebab2d9186cb5d75928f25 not found: ID does not exist" containerID="47e89dcbf105adf12233c3ec39c84f4e55326a7781ebab2d9186cb5d75928f25"
Nov 03 03:06:37.016122 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:37.016081    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e89dcbf105adf12233c3ec39c84f4e55326a7781ebab2d9186cb5d75928f25"} err="failed to get container status \"47e89dcbf105adf12233c3ec39c84f4e55326a7781ebab2d9186cb5d75928f25\": rpc error: code = NotFound desc = could not find container \"47e89dcbf105adf12233c3ec39c84f4e55326a7781ebab2d9186cb5d75928f25\": container with ID starting with 47e89dcbf105adf12233c3ec39c84f4e55326a7781ebab2d9186cb5d75928f25 not found: ID does not exist"
Nov 03 03:06:37.016122 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:37.016099    5508 scope.go:122] "RemoveContainer" containerID="a3f2952c1431422339f54dd909ca8c31ea4dd65bf116bc4125e86f08a7a96639"
Nov 03 03:06:37.016335 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:06:37.016315    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f2952c1431422339f54dd909ca8c31ea4dd65bf116bc4125e86f08a7a96639\": container with ID starting with a3f2952c1431422339f54dd909ca8c31ea4dd65bf116bc4125e86f08a7a96639 not found: ID does not exist" containerID="a3f2952c1431422339f54dd909ca8c31ea4dd65bf116bc4125e86f08a7a96639"
Nov 03 03:06:37.016399 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:37.016343    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f2952c1431422339f54dd909ca8c31ea4dd65bf116bc4125e86f08a7a96639"} err="failed to get container status \"a3f2952c1431422339f54dd909ca8c31ea4dd65bf116bc4125e86f08a7a96639\": rpc error: code = NotFound desc = could not find container \"a3f2952c1431422339f54dd909ca8c31ea4dd65bf116bc4125e86f08a7a96639\": container with ID starting with a3f2952c1431422339f54dd909ca8c31ea4dd65bf116bc4125e86f08a7a96639 not found: ID does not exist"
Nov 03 03:06:37.201162 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:37.201125    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"]
Nov 03 03:06:37.205194 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:37.205166    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9fbc1b-openshift-default-6774ddd86fggmkg"]
Nov 03 03:06:38.690891 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:38.690857    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" path="/var/lib/kubelet/pods/66e5dd3c-21ec-4786-ae03-49a41fb8e9ac/volumes"
Nov 03 03:06:42.045552 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:42.045519    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"]
Nov 03 03:06:42.092433 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:42.092400    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2qrnn"]
Nov 03 03:06:42.690200 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:42.690158    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fabb21e-dcfe-4608-8730-9382f32d772c" path="/var/lib/kubelet/pods/8fabb21e-dcfe-4608-8730-9382f32d772c/volumes"
Nov 03 03:06:43.812615 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:06:43.812574    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:06:47.478749 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:47.478657    5508 scope.go:122] "RemoveContainer" containerID="2b8111df5e9b793cb39b6aaa059e41de5e7809730a621e4ffb51a7cae7c975ee"
Nov 03 03:06:47.488998 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:47.488975    5508 scope.go:122] "RemoveContainer" containerID="5c17e49d538735a880a7c8c4c90c83c80e1b080f59605a1c501b195c34c750e2"
Nov 03 03:06:47.497902 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:47.497878    5508 scope.go:122] "RemoveContainer" containerID="33dc06b5cc1145e7ad95b5588250295ee5c9ef8af4bc0c5d608c27b9e3b4b1d4"
Nov 03 03:06:51.994518 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:51.994479    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"]
Nov 03 03:06:51.995021 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:51.994722    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp" podUID="6a99ffc0-6cc1-4188-8878-82a0dce662cc" containerName="manager" containerID="cri-o://258e9f52c546ab485f9cf08d5e79a5e64e41f9607d5e03ada49eda63f160ebb3" gracePeriod=10
Nov 03 03:06:52.249097 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:52.249037    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"
Nov 03 03:06:52.376450 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:52.376414    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a99ffc0-6cc1-4188-8878-82a0dce662cc-kube-api-access-g7smm\" (UniqueName: \"kubernetes.io/projected/6a99ffc0-6cc1-4188-8878-82a0dce662cc-kube-api-access-g7smm\") pod \"6a99ffc0-6cc1-4188-8878-82a0dce662cc\" (UID: \"6a99ffc0-6cc1-4188-8878-82a0dce662cc\") "
Nov 03 03:06:52.376633 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:52.376561    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a99ffc0-6cc1-4188-8878-82a0dce662cc-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a99ffc0-6cc1-4188-8878-82a0dce662cc-extensions-socket-volume\") pod \"6a99ffc0-6cc1-4188-8878-82a0dce662cc\" (UID: \"6a99ffc0-6cc1-4188-8878-82a0dce662cc\") "
Nov 03 03:06:52.377890 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:52.377857    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a99ffc0-6cc1-4188-8878-82a0dce662cc-extensions-socket-volume" pod "6a99ffc0-6cc1-4188-8878-82a0dce662cc" (UID: "6a99ffc0-6cc1-4188-8878-82a0dce662cc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 03:06:52.379488 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:52.379455    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a99ffc0-6cc1-4188-8878-82a0dce662cc-kube-api-access-g7smm" pod "6a99ffc0-6cc1-4188-8878-82a0dce662cc" (UID: "6a99ffc0-6cc1-4188-8878-82a0dce662cc"). InnerVolumeSpecName "kube-api-access-g7smm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:06:52.478085 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:52.478050    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7smm\" (UniqueName: \"kubernetes.io/projected/6a99ffc0-6cc1-4188-8878-82a0dce662cc-kube-api-access-g7smm\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:52.478085 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:52.478083    5508 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a99ffc0-6cc1-4188-8878-82a0dce662cc-extensions-socket-volume\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:06:53.084189 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:53.084087    5508 generic.go:356] "Generic (PLEG): container finished" podID="6a99ffc0-6cc1-4188-8878-82a0dce662cc" containerID="258e9f52c546ab485f9cf08d5e79a5e64e41f9607d5e03ada49eda63f160ebb3" exitCode=0
Nov 03 03:06:53.084189 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:53.084129    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"
Nov 03 03:06:53.084189 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:53.084167    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp" event={"ID":"6a99ffc0-6cc1-4188-8878-82a0dce662cc","Type":"ContainerDied","Data":"258e9f52c546ab485f9cf08d5e79a5e64e41f9607d5e03ada49eda63f160ebb3"}
Nov 03 03:06:53.084754 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:53.084222    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp" event={"ID":"6a99ffc0-6cc1-4188-8878-82a0dce662cc","Type":"ContainerDied","Data":"84362349396b3afeeb08a74479b06fd6a15b9496372878c9db86edeb1cf0af96"}
Nov 03 03:06:53.084754 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:53.084249    5508 scope.go:122] "RemoveContainer" containerID="258e9f52c546ab485f9cf08d5e79a5e64e41f9607d5e03ada49eda63f160ebb3"
Nov 03 03:06:53.098495 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:53.098475    5508 scope.go:122] "RemoveContainer" containerID="258e9f52c546ab485f9cf08d5e79a5e64e41f9607d5e03ada49eda63f160ebb3"
Nov 03 03:06:53.098810 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:06:53.098784    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258e9f52c546ab485f9cf08d5e79a5e64e41f9607d5e03ada49eda63f160ebb3\": container with ID starting with 258e9f52c546ab485f9cf08d5e79a5e64e41f9607d5e03ada49eda63f160ebb3 not found: ID does not exist" containerID="258e9f52c546ab485f9cf08d5e79a5e64e41f9607d5e03ada49eda63f160ebb3"
Nov 03 03:06:53.098920 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:53.098816    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258e9f52c546ab485f9cf08d5e79a5e64e41f9607d5e03ada49eda63f160ebb3"} err="failed to get container status \"258e9f52c546ab485f9cf08d5e79a5e64e41f9607d5e03ada49eda63f160ebb3\": rpc error: code = NotFound desc = could not find container \"258e9f52c546ab485f9cf08d5e79a5e64e41f9607d5e03ada49eda63f160ebb3\": container with ID starting with 258e9f52c546ab485f9cf08d5e79a5e64e41f9607d5e03ada49eda63f160ebb3 not found: ID does not exist"
Nov 03 03:06:53.286135 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:53.286098    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"]
Nov 03 03:06:53.332586 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:53.332551    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jr6bp"]
Nov 03 03:06:54.689989 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:06:54.689945    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a99ffc0-6cc1-4188-8878-82a0dce662cc" path="/var/lib/kubelet/pods/6a99ffc0-6cc1-4188-8878-82a0dce662cc/volumes"
Nov 03 03:07:04.054756 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:04.054716    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"]
Nov 03 03:07:04.092409 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:04.092372    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rk7pg"]
Nov 03 03:07:04.690527 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:04.690492    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715f1e6a-c193-4985-b324-0865a16f5a87" path="/var/lib/kubelet/pods/715f1e6a-c193-4985-b324-0865a16f5a87/volumes"
Nov 03 03:07:13.811776 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:07:13.811734    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:07:22.044805 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:22.044764    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"]
Nov 03 03:07:22.090668 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:22.090631    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-9x5vj"]
Nov 03 03:07:22.690269 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:22.690234    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02eaa27c-70ad-480a-8b9f-710562426465" path="/var/lib/kubelet/pods/02eaa27c-70ad-480a-8b9f-710562426465/volumes"
Nov 03 03:07:43.812186 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:07:43.812149    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:07:47.544462 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:47.544428    5508 scope.go:122] "RemoveContainer" containerID="b184479db64c33776230d9ca76b3ba9bd596f2ad156ff8c0958dbb2ce398190d"
Nov 03 03:07:47.557064 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:47.554521    5508 scope.go:122] "RemoveContainer" containerID="02fb355fb66adfdbd8e9beaa780fffb5de2432c04db9e383affc52beca73f016"
Nov 03 03:07:55.571919 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.571880    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw"]
Nov 03 03:07:55.572467 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.572283    5508 cpu_manager.go:414] "containerMap: removing container" podUID="6a99ffc0-6cc1-4188-8878-82a0dce662cc" containerName="manager"
Nov 03 03:07:55.572467 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.572295    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a99ffc0-6cc1-4188-8878-82a0dce662cc" containerName="manager"
Nov 03 03:07:55.572467 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.572307    5508 cpu_manager.go:414] "containerMap: removing container" podUID="67489fa9-855b-4b1b-a118-368708d5a5d1" containerName="revoke-keys"
Nov 03 03:07:55.572467 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.572314    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67489fa9-855b-4b1b-a118-368708d5a5d1" containerName="revoke-keys"
Nov 03 03:07:55.572467 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.572324    5508 cpu_manager.go:414] "containerMap: removing container" podUID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerName="istio-proxy"
Nov 03 03:07:55.572467 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.572330    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerName="istio-proxy"
Nov 03 03:07:55.572467 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.572417    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerName="istio-proxy"
Nov 03 03:07:55.572467 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.572430    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a99ffc0-6cc1-4188-8878-82a0dce662cc" containerName="manager"
Nov 03 03:07:55.572467 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.572438    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="67489fa9-855b-4b1b-a118-368708d5a5d1" containerName="revoke-keys"
Nov 03 03:07:55.576917 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.576896    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw"
Nov 03 03:07:55.625994 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.625958    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pkj7d\""
Nov 03 03:07:55.674844 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.674804    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw"]
Nov 03 03:07:55.706996 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.706957    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea2b4e9e-4a03-44e4-b089-c7a7e857562e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-whgfw\" (UID: \"ea2b4e9e-4a03-44e4-b089-c7a7e857562e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw"
Nov 03 03:07:55.707217 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.707011    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w58d\" (UniqueName: \"kubernetes.io/projected/ea2b4e9e-4a03-44e4-b089-c7a7e857562e-kube-api-access-7w58d\") pod \"kuadrant-operator-controller-manager-678d87766f-whgfw\" (UID: \"ea2b4e9e-4a03-44e4-b089-c7a7e857562e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw"
Nov 03 03:07:55.807921 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.807884    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea2b4e9e-4a03-44e4-b089-c7a7e857562e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-whgfw\" (UID: \"ea2b4e9e-4a03-44e4-b089-c7a7e857562e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw"
Nov 03 03:07:55.808116 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.807939    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w58d\" (UniqueName: \"kubernetes.io/projected/ea2b4e9e-4a03-44e4-b089-c7a7e857562e-kube-api-access-7w58d\") pod \"kuadrant-operator-controller-manager-678d87766f-whgfw\" (UID: \"ea2b4e9e-4a03-44e4-b089-c7a7e857562e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw"
Nov 03 03:07:55.808298 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.808276    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea2b4e9e-4a03-44e4-b089-c7a7e857562e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-whgfw\" (UID: \"ea2b4e9e-4a03-44e4-b089-c7a7e857562e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw"
Nov 03 03:07:55.871702 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.871623    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w58d\" (UniqueName: \"kubernetes.io/projected/ea2b4e9e-4a03-44e4-b089-c7a7e857562e-kube-api-access-7w58d\") pod \"kuadrant-operator-controller-manager-678d87766f-whgfw\" (UID: \"ea2b4e9e-4a03-44e4-b089-c7a7e857562e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw"
Nov 03 03:07:55.889482 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:55.889454    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw"
Nov 03 03:07:56.317002 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:56.316967    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw"]
Nov 03 03:07:56.362817 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:07:56.362785    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2b4e9e_4a03_44e4_b089_c7a7e857562e.slice/crio-846677595e62fd705891fd55f647b1c2fa5472ee8fff19d51b1998584d5bb576 WatchSource:0}: Error finding container 846677595e62fd705891fd55f647b1c2fa5472ee8fff19d51b1998584d5bb576: Status 404 returned error can't find the container with id 846677595e62fd705891fd55f647b1c2fa5472ee8fff19d51b1998584d5bb576
Nov 03 03:07:56.428575 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:56.428543    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw" event={"ID":"ea2b4e9e-4a03-44e4-b089-c7a7e857562e","Type":"ContainerStarted","Data":"846677595e62fd705891fd55f647b1c2fa5472ee8fff19d51b1998584d5bb576"}
Nov 03 03:07:57.439070 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:57.439028    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw" event={"ID":"ea2b4e9e-4a03-44e4-b089-c7a7e857562e","Type":"ContainerStarted","Data":"c743328a83f2be119bfa9b7ee176594aad8958bc47a9e0e680aff5b494cf3d19"}
Nov 03 03:07:57.439588 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:57.439091    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw"
Nov 03 03:07:57.645001 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:07:57.644946    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw" podStartSLOduration=2.644932597 podStartE2EDuration="2.644932597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:07:57.597405557 +0000 UTC m=+6557.023012022" watchObservedRunningTime="2026-11-03 03:07:57.644932597 +0000 UTC m=+6557.070539062"
Nov 03 03:08:08.447982 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:08.447947    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-whgfw"
Nov 03 03:08:13.812383 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:08:13.812331    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:08:15.509578 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:15.509540    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:08:20.283579 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.283540    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"]
Nov 03 03:08:20.284001 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.283950    5508 cpu_manager.go:414] "containerMap: removing container" podUID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerName="istio-proxy"
Nov 03 03:08:20.284001 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.283961    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerName="istio-proxy"
Nov 03 03:08:20.284087 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.284022    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e5dd3c-21ec-4786-ae03-49a41fb8e9ac" containerName="istio-proxy"
Nov 03 03:08:20.287564 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.287541    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"
Nov 03 03:08:20.314143 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.314101    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-4s7mt\" (UID: \"ad593da4-1691-4121-88c1-53cb536b4007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"
Nov 03 03:08:20.314321 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.314146    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpzpw\" (UniqueName: \"kubernetes.io/projected/ad593da4-1691-4121-88c1-53cb536b4007-kube-api-access-kpzpw\") pod \"maas-api-revoke-sub-d576e68b-4s7mt\" (UID: \"ad593da4-1691-4121-88c1-53cb536b4007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"
Nov 03 03:08:20.314321 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.314260    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-4s7mt\" (UID: \"ad593da4-1691-4121-88c1-53cb536b4007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"
Nov 03 03:08:20.336563 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.336530    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:08:20.382714 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.382675    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:08:20.382714 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.382673    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:08:20.382973 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.382767    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:08:20.386423 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.386393    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"]
Nov 03 03:08:20.415160 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.415115    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-4s7mt\" (UID: \"ad593da4-1691-4121-88c1-53cb536b4007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"
Nov 03 03:08:20.415410 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.415218    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-4s7mt\" (UID: \"ad593da4-1691-4121-88c1-53cb536b4007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"
Nov 03 03:08:20.415410 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.415262    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpzpw\" (UniqueName: \"kubernetes.io/projected/ad593da4-1691-4121-88c1-53cb536b4007-kube-api-access-kpzpw\") pod \"maas-api-revoke-sub-d576e68b-4s7mt\" (UID: \"ad593da4-1691-4121-88c1-53cb536b4007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"
Nov 03 03:08:20.415922 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.415899    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-4s7mt\" (UID: \"ad593da4-1691-4121-88c1-53cb536b4007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"
Nov 03 03:08:20.417557 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.417538    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-4s7mt\" (UID: \"ad593da4-1691-4121-88c1-53cb536b4007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"
Nov 03 03:08:20.473781 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.473746    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpzpw\" (UniqueName: \"kubernetes.io/projected/ad593da4-1691-4121-88c1-53cb536b4007-kube-api-access-kpzpw\") pod \"maas-api-revoke-sub-d576e68b-4s7mt\" (UID: \"ad593da4-1691-4121-88c1-53cb536b4007\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"
Nov 03 03:08:20.600563 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.600481    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"
Nov 03 03:08:20.973165 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:20.973122    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"]
Nov 03 03:08:20.980519 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:08:20.980490    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad593da4_1691_4121_88c1_53cb536b4007.slice/crio-92e7f7768526684fff51e6d523b6b793d846679c0dec9cc2812a2f2f89375c8e WatchSource:0}: Error finding container 92e7f7768526684fff51e6d523b6b793d846679c0dec9cc2812a2f2f89375c8e: Status 404 returned error can't find the container with id 92e7f7768526684fff51e6d523b6b793d846679c0dec9cc2812a2f2f89375c8e
Nov 03 03:08:21.598859 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:21.598758    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt" event={"ID":"ad593da4-1691-4121-88c1-53cb536b4007","Type":"ContainerStarted","Data":"5d9c6c8bd7057501204e3be3a99caa3f7a4f46dac7fe11b3075e504d1a35e1f6"}
Nov 03 03:08:21.598859 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:21.598824    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt" event={"ID":"ad593da4-1691-4121-88c1-53cb536b4007","Type":"ContainerStarted","Data":"92e7f7768526684fff51e6d523b6b793d846679c0dec9cc2812a2f2f89375c8e"}
Nov 03 03:08:21.708180 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:21.708124    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt" podStartSLOduration=1.708110006 podStartE2EDuration="1.708110006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:21.707244413 +0000 UTC m=+6581.132850876" watchObservedRunningTime="2026-11-03 03:08:21.708110006 +0000 UTC m=+6581.133716469"
Nov 03 03:08:22.605772 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:22.605735    5508 generic.go:356] "Generic (PLEG): container finished" podID="ad593da4-1691-4121-88c1-53cb536b4007" containerID="5d9c6c8bd7057501204e3be3a99caa3f7a4f46dac7fe11b3075e504d1a35e1f6" exitCode=0
Nov 03 03:08:22.606212 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:22.605819    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt" event={"ID":"ad593da4-1691-4121-88c1-53cb536b4007","Type":"ContainerDied","Data":"5d9c6c8bd7057501204e3be3a99caa3f7a4f46dac7fe11b3075e504d1a35e1f6"}
Nov 03 03:08:23.747475 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:23.747143    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"
Nov 03 03:08:23.848419 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:23.848379    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-xks-ca\") pod \"ad593da4-1691-4121-88c1-53cb536b4007\" (UID: \"ad593da4-1691-4121-88c1-53cb536b4007\") "
Nov 03 03:08:23.848609 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:23.848442    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad593da4-1691-4121-88c1-53cb536b4007-kube-api-access-kpzpw\" (UniqueName: \"kubernetes.io/projected/ad593da4-1691-4121-88c1-53cb536b4007-kube-api-access-kpzpw\") pod \"ad593da4-1691-4121-88c1-53cb536b4007\" (UID: \"ad593da4-1691-4121-88c1-53cb536b4007\") "
Nov 03 03:08:23.848609 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:23.848514    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-openshift-ca\") pod \"ad593da4-1691-4121-88c1-53cb536b4007\" (UID: \"ad593da4-1691-4121-88c1-53cb536b4007\") "
Nov 03 03:08:23.848917 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:23.848885    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-openshift-ca" pod "ad593da4-1691-4121-88c1-53cb536b4007" (UID: "ad593da4-1691-4121-88c1-53cb536b4007"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:08:23.850601 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:23.850568    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-xks-ca" pod "ad593da4-1691-4121-88c1-53cb536b4007" (UID: "ad593da4-1691-4121-88c1-53cb536b4007"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:08:23.850688 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:23.850632    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad593da4-1691-4121-88c1-53cb536b4007-kube-api-access-kpzpw" pod "ad593da4-1691-4121-88c1-53cb536b4007" (UID: "ad593da4-1691-4121-88c1-53cb536b4007"). InnerVolumeSpecName "kube-api-access-kpzpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:08:23.950184 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:23.950153    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpzpw\" (UniqueName: \"kubernetes.io/projected/ad593da4-1691-4121-88c1-53cb536b4007-kube-api-access-kpzpw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:23.950184 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:23.950182    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:23.950473 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:23.950194    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad593da4-1691-4121-88c1-53cb536b4007-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:24.618445 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:24.618411    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"
Nov 03 03:08:24.618651 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:24.618450    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt" event={"ID":"ad593da4-1691-4121-88c1-53cb536b4007","Type":"ContainerDied","Data":"92e7f7768526684fff51e6d523b6b793d846679c0dec9cc2812a2f2f89375c8e"}
Nov 03 03:08:24.618651 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:24.618497    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92e7f7768526684fff51e6d523b6b793d846679c0dec9cc2812a2f2f89375c8e"
Nov 03 03:08:29.046691 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:29.046649    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"]
Nov 03 03:08:29.092904 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:29.092868    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-b5nlx"]
Nov 03 03:08:30.690993 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:30.690961    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22707a93-24c4-490e-8d79-f73610306d08" path="/var/lib/kubelet/pods/22707a93-24c4-490e-8d79-f73610306d08/volumes"
Nov 03 03:08:32.999178 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:32.999142    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:08:40.060374 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:40.059631    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"]
Nov 03 03:08:40.104371 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:40.104091    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j24kn"]
Nov 03 03:08:40.724822 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:40.724785    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70559420-e3ed-4cc1-ac36-17ed98066b74" path="/var/lib/kubelet/pods/70559420-e3ed-4cc1-ac36-17ed98066b74/volumes"
Nov 03 03:08:41.866520 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:41.866491    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 03:08:41.993965 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:41.993935    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 03:08:42.084148 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.084115    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"]
Nov 03 03:08:42.084730 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.084712    5508 cpu_manager.go:414] "containerMap: removing container" podUID="ad593da4-1691-4121-88c1-53cb536b4007" containerName="revoke-keys"
Nov 03 03:08:42.084800 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.084733    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad593da4-1691-4121-88c1-53cb536b4007" containerName="revoke-keys"
Nov 03 03:08:42.084863 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.084852    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad593da4-1691-4121-88c1-53cb536b4007" containerName="revoke-keys"
Nov 03 03:08:42.089479 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.089455    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"
Nov 03 03:08:42.184453 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.184421    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:08:42.184670 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.184509    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 03:08:42.184670 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.184565    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:08:42.184670 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.184570    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bfxtx\""
Nov 03 03:08:42.185888 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.185866    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-g4ftz\" (UID: \"8b4816c8-2912-484e-8929-9f6e6ad4b234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"
Nov 03 03:08:42.185956 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.185901    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-g4ftz\" (UID: \"8b4816c8-2912-484e-8929-9f6e6ad4b234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"
Nov 03 03:08:42.185956 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.185948    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcwkh\" (UniqueName: \"kubernetes.io/projected/8b4816c8-2912-484e-8929-9f6e6ad4b234-kube-api-access-kcwkh\") pod \"maas-api-revoke-sub-d2bc93bc-g4ftz\" (UID: \"8b4816c8-2912-484e-8929-9f6e6ad4b234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"
Nov 03 03:08:42.186879 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.186860    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"]
Nov 03 03:08:42.287400 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.287365    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-g4ftz\" (UID: \"8b4816c8-2912-484e-8929-9f6e6ad4b234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"
Nov 03 03:08:42.287400 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.287407    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-g4ftz\" (UID: \"8b4816c8-2912-484e-8929-9f6e6ad4b234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"
Nov 03 03:08:42.287688 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.287443    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcwkh\" (UniqueName: \"kubernetes.io/projected/8b4816c8-2912-484e-8929-9f6e6ad4b234-kube-api-access-kcwkh\") pod \"maas-api-revoke-sub-d2bc93bc-g4ftz\" (UID: \"8b4816c8-2912-484e-8929-9f6e6ad4b234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"
Nov 03 03:08:42.288159 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.288133    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-g4ftz\" (UID: \"8b4816c8-2912-484e-8929-9f6e6ad4b234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"
Nov 03 03:08:42.289770 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.289747    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-g4ftz\" (UID: \"8b4816c8-2912-484e-8929-9f6e6ad4b234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"
Nov 03 03:08:42.344853 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.344816    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcwkh\" (UniqueName: \"kubernetes.io/projected/8b4816c8-2912-484e-8929-9f6e6ad4b234-kube-api-access-kcwkh\") pod \"maas-api-revoke-sub-d2bc93bc-g4ftz\" (UID: \"8b4816c8-2912-484e-8929-9f6e6ad4b234\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"
Nov 03 03:08:42.391038 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.391003    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:08:42.403580 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.403545    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"
Nov 03 03:08:42.774482 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:42.774402    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"]
Nov 03 03:08:42.782197 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:08:42.782167    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4816c8_2912_484e_8929_9f6e6ad4b234.slice/crio-4c22d2ab4c914e41357451c76b5b8921d85eb05dd44e25c570592467b0b82a21 WatchSource:0}: Error finding container 4c22d2ab4c914e41357451c76b5b8921d85eb05dd44e25c570592467b0b82a21: Status 404 returned error can't find the container with id 4c22d2ab4c914e41357451c76b5b8921d85eb05dd44e25c570592467b0b82a21
Nov 03 03:08:43.752939 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:43.752898    5508 generic.go:356] "Generic (PLEG): container finished" podID="8b4816c8-2912-484e-8929-9f6e6ad4b234" containerID="c9d443acf0ef83cff678fd11660fb1838def9b37948a8d65c8e4803bf1d51bbf" exitCode=0
Nov 03 03:08:43.753387 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:43.752938    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz" event={"ID":"8b4816c8-2912-484e-8929-9f6e6ad4b234","Type":"ContainerDied","Data":"c9d443acf0ef83cff678fd11660fb1838def9b37948a8d65c8e4803bf1d51bbf"}
Nov 03 03:08:43.753387 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:43.752999    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz" event={"ID":"8b4816c8-2912-484e-8929-9f6e6ad4b234","Type":"ContainerStarted","Data":"4c22d2ab4c914e41357451c76b5b8921d85eb05dd44e25c570592467b0b82a21"}
Nov 03 03:08:43.811932 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:08:43.811848    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:08:44.896267 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:44.896242    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"
Nov 03 03:08:45.014727 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:45.014643    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4816c8-2912-484e-8929-9f6e6ad4b234-kube-api-access-kcwkh\" (UniqueName: \"kubernetes.io/projected/8b4816c8-2912-484e-8929-9f6e6ad4b234-kube-api-access-kcwkh\") pod \"8b4816c8-2912-484e-8929-9f6e6ad4b234\" (UID: \"8b4816c8-2912-484e-8929-9f6e6ad4b234\") "
Nov 03 03:08:45.014882 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:45.014815    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-openshift-ca\") pod \"8b4816c8-2912-484e-8929-9f6e6ad4b234\" (UID: \"8b4816c8-2912-484e-8929-9f6e6ad4b234\") "
Nov 03 03:08:45.014882 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:45.014848    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-xks-ca\") pod \"8b4816c8-2912-484e-8929-9f6e6ad4b234\" (UID: \"8b4816c8-2912-484e-8929-9f6e6ad4b234\") "
Nov 03 03:08:45.015240 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:45.015211    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-openshift-ca" pod "8b4816c8-2912-484e-8929-9f6e6ad4b234" (UID: "8b4816c8-2912-484e-8929-9f6e6ad4b234"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:08:45.017078 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:45.017052    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-xks-ca" pod "8b4816c8-2912-484e-8929-9f6e6ad4b234" (UID: "8b4816c8-2912-484e-8929-9f6e6ad4b234"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:08:45.017263 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:45.017238    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4816c8-2912-484e-8929-9f6e6ad4b234-kube-api-access-kcwkh" pod "8b4816c8-2912-484e-8929-9f6e6ad4b234" (UID: "8b4816c8-2912-484e-8929-9f6e6ad4b234"). InnerVolumeSpecName "kube-api-access-kcwkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:08:45.116720 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:45.116683    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-openshift-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:45.116720 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:45.116716    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4816c8-2912-484e-8929-9f6e6ad4b234-maas-api-xks-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:45.116720 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:45.116728    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcwkh\" (UniqueName: \"kubernetes.io/projected/8b4816c8-2912-484e-8929-9f6e6ad4b234-kube-api-access-kcwkh\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:08:45.765898 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:45.765865    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz" event={"ID":"8b4816c8-2912-484e-8929-9f6e6ad4b234","Type":"ContainerDied","Data":"4c22d2ab4c914e41357451c76b5b8921d85eb05dd44e25c570592467b0b82a21"}
Nov 03 03:08:45.765898 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:45.765885    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"
Nov 03 03:08:45.766194 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:45.765914    5508 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c22d2ab4c914e41357451c76b5b8921d85eb05dd44e25c570592467b0b82a21"
Nov 03 03:08:47.589626 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:47.589591    5508 scope.go:122] "RemoveContainer" containerID="73354cbabb7da95b9aa1ab243f8b7cf14462c225ab524bee7da0da86d1606544"
Nov 03 03:08:47.599517 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:47.599494    5508 scope.go:122] "RemoveContainer" containerID="9c120378626f62938987e5886bdcb16fc25eac792d481e9dcad009e62b90caae"
Nov 03 03:08:47.609547 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:47.609525    5508 scope.go:122] "RemoveContainer" containerID="e941c9897830c5ccd417b924aee0ec738dcccee573f40063121205030c86b260"
Nov 03 03:08:50.560330 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:50.560290    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:08:55.372717 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.372682    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"]
Nov 03 03:08:55.373139 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.373074    5508 cpu_manager.go:414] "containerMap: removing container" podUID="8b4816c8-2912-484e-8929-9f6e6ad4b234" containerName="revoke-keys"
Nov 03 03:08:55.373139 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.373085    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4816c8-2912-484e-8929-9f6e6ad4b234" containerName="revoke-keys"
Nov 03 03:08:55.373227 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.373171    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4816c8-2912-484e-8929-9f6e6ad4b234" containerName="revoke-keys"
Nov 03 03:08:55.378084 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.378061    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:55.427174 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.427134    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 03:08:55.473436 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.473394    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 03:08:55.473624 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.473477    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sxqdt\""
Nov 03 03:08:55.473624 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.473480    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 03:08:55.473624 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.473591    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 03:08:55.477173 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.477144    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"]
Nov 03 03:08:55.523497 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.523458    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dafa3d14-bec4-46df-84ee-8f418156e7f9-openshift-service-ca\") pod \"maas-api-6b6d569d44-5kgfw\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:55.523685 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.523505    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqflw\" (UniqueName: \"kubernetes.io/projected/dafa3d14-bec4-46df-84ee-8f418156e7f9-kube-api-access-mqflw\") pod \"maas-api-6b6d569d44-5kgfw\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:55.523685 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.523612    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-maas-api-tls\") pod \"maas-api-6b6d569d44-5kgfw\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:55.523685 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.523665    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-metrics-certs\") pod \"maas-api-6b6d569d44-5kgfw\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:55.624862 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.624754    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dafa3d14-bec4-46df-84ee-8f418156e7f9-openshift-service-ca\") pod \"maas-api-6b6d569d44-5kgfw\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:55.624862 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.624811    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqflw\" (UniqueName: \"kubernetes.io/projected/dafa3d14-bec4-46df-84ee-8f418156e7f9-kube-api-access-mqflw\") pod \"maas-api-6b6d569d44-5kgfw\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:55.624862 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.624860    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-maas-api-tls\") pod \"maas-api-6b6d569d44-5kgfw\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:55.625190 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.624909    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-metrics-certs\") pod \"maas-api-6b6d569d44-5kgfw\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:55.625480 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.625455    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dafa3d14-bec4-46df-84ee-8f418156e7f9-openshift-service-ca\") pod \"maas-api-6b6d569d44-5kgfw\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:55.627325 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.627303    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-maas-api-tls\") pod \"maas-api-6b6d569d44-5kgfw\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:55.627441 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.627337    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-metrics-certs\") pod \"maas-api-6b6d569d44-5kgfw\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:55.682677 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.682645    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqflw\" (UniqueName: \"kubernetes.io/projected/dafa3d14-bec4-46df-84ee-8f418156e7f9-kube-api-access-mqflw\") pod \"maas-api-6b6d569d44-5kgfw\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") " pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:55.691663 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:55.691632    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:56.065191 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:56.065156    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"]
Nov 03 03:08:56.073611 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:08:56.073581    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafa3d14_bec4_46df_84ee_8f418156e7f9.slice/crio-7f043ed25dc29c912c78365fe1b82670f0d08a6b59b4cac77c3a3324e0c60683 WatchSource:0}: Error finding container 7f043ed25dc29c912c78365fe1b82670f0d08a6b59b4cac77c3a3324e0c60683: Status 404 returned error can't find the container with id 7f043ed25dc29c912c78365fe1b82670f0d08a6b59b4cac77c3a3324e0c60683
Nov 03 03:08:56.822635 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:56.822600    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw" event={"ID":"dafa3d14-bec4-46df-84ee-8f418156e7f9","Type":"ContainerStarted","Data":"7f043ed25dc29c912c78365fe1b82670f0d08a6b59b4cac77c3a3324e0c60683"}
Nov 03 03:08:57.829067 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:57.829029    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw" event={"ID":"dafa3d14-bec4-46df-84ee-8f418156e7f9","Type":"ContainerStarted","Data":"a5c872a82fb9314439b64f8475cda3f48afbb639f0324ee5f19b769922b44f42"}
Nov 03 03:08:57.829510 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:57.829105    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:08:57.941802 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:08:57.941745    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw" podStartSLOduration=1.997379661 podStartE2EDuration="2.941731831s" totalImagesPullingTime="944.35217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:08:57.939612441 +0000 UTC m=+6617.365218925" watchObservedRunningTime="2026-11-03 03:08:57.941731831 +0000 UTC m=+6617.367338296"
Nov 03 03:09:03.840375 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:03.840321    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:09:04.092249 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.092154    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"]
Nov 03 03:09:04.095009 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.094981    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.155703 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.155671    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"]
Nov 03 03:09:04.207987 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.207952    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f550c92c-848c-404c-946e-d5859646cdef-maas-api-tls\") pod \"maas-api-67759c8968-j5f8l\" (UID: \"f550c92c-848c-404c-946e-d5859646cdef\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.208147 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.208104    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brqgz\" (UniqueName: \"kubernetes.io/projected/f550c92c-848c-404c-946e-d5859646cdef-kube-api-access-brqgz\") pod \"maas-api-67759c8968-j5f8l\" (UID: \"f550c92c-848c-404c-946e-d5859646cdef\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.208212 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.208156    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f550c92c-848c-404c-946e-d5859646cdef-openshift-service-ca\") pod \"maas-api-67759c8968-j5f8l\" (UID: \"f550c92c-848c-404c-946e-d5859646cdef\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.208212 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.208200    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f550c92c-848c-404c-946e-d5859646cdef-metrics-certs\") pod \"maas-api-67759c8968-j5f8l\" (UID: \"f550c92c-848c-404c-946e-d5859646cdef\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.309712 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.309668    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brqgz\" (UniqueName: \"kubernetes.io/projected/f550c92c-848c-404c-946e-d5859646cdef-kube-api-access-brqgz\") pod \"maas-api-67759c8968-j5f8l\" (UID: \"f550c92c-848c-404c-946e-d5859646cdef\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.309712 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.309716    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f550c92c-848c-404c-946e-d5859646cdef-openshift-service-ca\") pod \"maas-api-67759c8968-j5f8l\" (UID: \"f550c92c-848c-404c-946e-d5859646cdef\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.309950 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.309749    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f550c92c-848c-404c-946e-d5859646cdef-metrics-certs\") pod \"maas-api-67759c8968-j5f8l\" (UID: \"f550c92c-848c-404c-946e-d5859646cdef\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.309950 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.309777    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f550c92c-848c-404c-946e-d5859646cdef-maas-api-tls\") pod \"maas-api-67759c8968-j5f8l\" (UID: \"f550c92c-848c-404c-946e-d5859646cdef\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.310469 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.310445    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f550c92c-848c-404c-946e-d5859646cdef-openshift-service-ca\") pod \"maas-api-67759c8968-j5f8l\" (UID: \"f550c92c-848c-404c-946e-d5859646cdef\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.312343 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.312318    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f550c92c-848c-404c-946e-d5859646cdef-maas-api-tls\") pod \"maas-api-67759c8968-j5f8l\" (UID: \"f550c92c-848c-404c-946e-d5859646cdef\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.312453 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.312373    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f550c92c-848c-404c-946e-d5859646cdef-metrics-certs\") pod \"maas-api-67759c8968-j5f8l\" (UID: \"f550c92c-848c-404c-946e-d5859646cdef\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.368204 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.368115    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brqgz\" (UniqueName: \"kubernetes.io/projected/f550c92c-848c-404c-946e-d5859646cdef-kube-api-access-brqgz\") pod \"maas-api-67759c8968-j5f8l\" (UID: \"f550c92c-848c-404c-946e-d5859646cdef\") " pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.407444 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.407409    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:04.779865 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.779826    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"]
Nov 03 03:09:04.787413 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:09:04.787381    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf550c92c_848c_404c_946e_d5859646cdef.slice/crio-1fa9f744de53ce7357df00a38f377410b132b779151000f32fbfc1c001dff911 WatchSource:0}: Error finding container 1fa9f744de53ce7357df00a38f377410b132b779151000f32fbfc1c001dff911: Status 404 returned error can't find the container with id 1fa9f744de53ce7357df00a38f377410b132b779151000f32fbfc1c001dff911
Nov 03 03:09:04.869202 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:04.869164    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l" event={"ID":"f550c92c-848c-404c-946e-d5859646cdef","Type":"ContainerStarted","Data":"1fa9f744de53ce7357df00a38f377410b132b779151000f32fbfc1c001dff911"}
Nov 03 03:09:05.877798 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:05.877763    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l" event={"ID":"f550c92c-848c-404c-946e-d5859646cdef","Type":"ContainerStarted","Data":"85f1cb0063053b37310484140e288cdda84f41ee288387ee2dbf3511fcb0b206"}
Nov 03 03:09:05.878229 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:05.877859    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:05.990296 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:05.990177    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l" podStartSLOduration=1.084961914 podStartE2EDuration="1.99016126s" totalImagesPullingTime="905.199346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 03:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:09:05.988145161 +0000 UTC m=+6625.413751628" watchObservedRunningTime="2026-11-03 03:09:05.99016126 +0000 UTC m=+6625.415767724"
Nov 03 03:09:08.203378 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:08.203329    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:09:11.890081 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:11.890051    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67759c8968-j5f8l"
Nov 03 03:09:12.051053 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.051020    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"]
Nov 03 03:09:12.051672 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.051314    5508 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw" podUID="dafa3d14-bec4-46df-84ee-8f418156e7f9" containerName="maas-api" containerID="cri-o://a5c872a82fb9314439b64f8475cda3f48afbb639f0324ee5f19b769922b44f42" gracePeriod=30
Nov 03 03:09:12.299065 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.299037    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:09:12.435456 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.435372    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafa3d14-bec4-46df-84ee-8f418156e7f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dafa3d14-bec4-46df-84ee-8f418156e7f9-openshift-service-ca\") pod \"dafa3d14-bec4-46df-84ee-8f418156e7f9\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") "
Nov 03 03:09:12.435638 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.435463    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-maas-api-tls\") pod \"dafa3d14-bec4-46df-84ee-8f418156e7f9\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") "
Nov 03 03:09:12.435638 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.435531    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-metrics-certs\") pod \"dafa3d14-bec4-46df-84ee-8f418156e7f9\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") "
Nov 03 03:09:12.435758 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.435636    5508 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafa3d14-bec4-46df-84ee-8f418156e7f9-kube-api-access-mqflw\" (UniqueName: \"kubernetes.io/projected/dafa3d14-bec4-46df-84ee-8f418156e7f9-kube-api-access-mqflw\") pod \"dafa3d14-bec4-46df-84ee-8f418156e7f9\" (UID: \"dafa3d14-bec4-46df-84ee-8f418156e7f9\") "
Nov 03 03:09:12.435877 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.435848    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafa3d14-bec4-46df-84ee-8f418156e7f9-openshift-service-ca" pod "dafa3d14-bec4-46df-84ee-8f418156e7f9" (UID: "dafa3d14-bec4-46df-84ee-8f418156e7f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 03:09:12.436062 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.436036    5508 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dafa3d14-bec4-46df-84ee-8f418156e7f9-openshift-service-ca\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:09:12.437991 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.437957    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafa3d14-bec4-46df-84ee-8f418156e7f9-kube-api-access-mqflw" pod "dafa3d14-bec4-46df-84ee-8f418156e7f9" (UID: "dafa3d14-bec4-46df-84ee-8f418156e7f9"). InnerVolumeSpecName "kube-api-access-mqflw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 03:09:12.437991 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.437977    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-metrics-certs" pod "dafa3d14-bec4-46df-84ee-8f418156e7f9" (UID: "dafa3d14-bec4-46df-84ee-8f418156e7f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:09:12.438143 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.438056    5508 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-maas-api-tls" pod "dafa3d14-bec4-46df-84ee-8f418156e7f9" (UID: "dafa3d14-bec4-46df-84ee-8f418156e7f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 03:09:12.536917 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.536876    5508 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-maas-api-tls\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:09:12.536917 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.536913    5508 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dafa3d14-bec4-46df-84ee-8f418156e7f9-metrics-certs\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:09:12.536917 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.536923    5508 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqflw\" (UniqueName: \"kubernetes.io/projected/dafa3d14-bec4-46df-84ee-8f418156e7f9-kube-api-access-mqflw\") on node \"ip-10-0-140-255.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 03:09:12.935384 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.935329    5508 generic.go:356] "Generic (PLEG): container finished" podID="dafa3d14-bec4-46df-84ee-8f418156e7f9" containerID="a5c872a82fb9314439b64f8475cda3f48afbb639f0324ee5f19b769922b44f42" exitCode=0
Nov 03 03:09:12.935384 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.935394    5508 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"
Nov 03 03:09:12.935983 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.935413    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw" event={"ID":"dafa3d14-bec4-46df-84ee-8f418156e7f9","Type":"ContainerDied","Data":"a5c872a82fb9314439b64f8475cda3f48afbb639f0324ee5f19b769922b44f42"}
Nov 03 03:09:12.935983 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.935480    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw" event={"ID":"dafa3d14-bec4-46df-84ee-8f418156e7f9","Type":"ContainerDied","Data":"7f043ed25dc29c912c78365fe1b82670f0d08a6b59b4cac77c3a3324e0c60683"}
Nov 03 03:09:12.935983 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.935518    5508 scope.go:122] "RemoveContainer" containerID="a5c872a82fb9314439b64f8475cda3f48afbb639f0324ee5f19b769922b44f42"
Nov 03 03:09:12.945001 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.944984    5508 scope.go:122] "RemoveContainer" containerID="a5c872a82fb9314439b64f8475cda3f48afbb639f0324ee5f19b769922b44f42"
Nov 03 03:09:12.945262 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:09:12.945240    5508 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c872a82fb9314439b64f8475cda3f48afbb639f0324ee5f19b769922b44f42\": container with ID starting with a5c872a82fb9314439b64f8475cda3f48afbb639f0324ee5f19b769922b44f42 not found: ID does not exist" containerID="a5c872a82fb9314439b64f8475cda3f48afbb639f0324ee5f19b769922b44f42"
Nov 03 03:09:12.945360 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:12.945270    5508 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c872a82fb9314439b64f8475cda3f48afbb639f0324ee5f19b769922b44f42"} err="failed to get container status \"a5c872a82fb9314439b64f8475cda3f48afbb639f0324ee5f19b769922b44f42\": rpc error: code = NotFound desc = could not find container \"a5c872a82fb9314439b64f8475cda3f48afbb639f0324ee5f19b769922b44f42\": container with ID starting with a5c872a82fb9314439b64f8475cda3f48afbb639f0324ee5f19b769922b44f42 not found: ID does not exist"
Nov 03 03:09:13.089386 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:13.089256    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"]
Nov 03 03:09:13.135880 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:13.135840    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6d569d44-5kgfw"]
Nov 03 03:09:13.811907 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:09:13.811872    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:09:14.690809 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:14.690775    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafa3d14-bec4-46df-84ee-8f418156e7f9" path="/var/lib/kubelet/pods/dafa3d14-bec4-46df-84ee-8f418156e7f9/volumes"
Nov 03 03:09:43.812941 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:09:43.812894    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:09:47.371378 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:47.371281    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:09:47.643572 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:47.643480    5508 scope.go:122] "RemoveContainer" containerID="50ca5788aaa9ca56a5fe880d8f7397c57dd45bb98e09b2dc8bfdc6604a64c42c"
Nov 03 03:09:47.655513 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:47.655487    5508 scope.go:122] "RemoveContainer" containerID="1f573fb6eb8d413a07b1ef2ae90861aef19a41b3de22182d7e05e61090e7e24b"
Nov 03 03:09:47.675153 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:47.675121    5508 scope.go:122] "RemoveContainer" containerID="d4b7c3644c28e306c2f213e87fc04312d926fd89201a3ab0e81a7c84976821b2"
Nov 03 03:09:59.787795 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:09:59.787754    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:10:13.811574 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:10:13.811538    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:10:21.580077 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:10:21.580041    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:10:33.964568 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:10:33.964530    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:10:43.811877 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:10:43.811839    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:10:47.741686 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:10:47.741656    5508 scope.go:122] "RemoveContainer" containerID="bead43e8ddf871b5e5657b4311ba034be3e4c141cbd3a05dcd59d6222614d993"
Nov 03 03:11:04.867037 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:11:04.866980    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:11:13.811087 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:11:13.811044    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:11:15.568678 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:11:15.568638    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:11:28.174545 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:11:28.174505    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:11:37.055763 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:11:37.055730    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"]
Nov 03 03:11:37.101943 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:11:37.101915    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zslk6"]
Nov 03 03:11:38.690805 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:11:38.690766    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67489fa9-855b-4b1b-a118-368708d5a5d1" path="/var/lib/kubelet/pods/67489fa9-855b-4b1b-a118-368708d5a5d1/volumes"
Nov 03 03:11:38.774752 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:11:38.774716    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:11:41.944563 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:11:41.944508    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-xzjj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:11:41.945116 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:11:41.944583    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-xzjj9" podUID="1185a711-3994-4129-a641-b93e62e8effe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:11:43.811444 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:11:43.811408    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:12:02.484321 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:12:02.484286    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:12:13.298387 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:12:13.298335    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:12:13.910950 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:12:13.910876    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:12:23.216600 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:12:23.216560    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:12:43.670987 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:12:43.670948    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:12:43.811552 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:12:43.811512    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:12:47.769723 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:12:47.769643    5508 scope.go:122] "RemoveContainer" containerID="454fd3bce7e48896b6f0c71e40ad107ec44c97973b2784cc82a65a8cee407a01"
Nov 03 03:13:04.369234 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:04.369193    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:13:13.811253 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:13:13.811212    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:13:24.045368 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:24.045288    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"]
Nov 03 03:13:24.091367 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:24.091328    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-4s7mt"]
Nov 03 03:13:24.690744 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:24.690712    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad593da4-1691-4121-88c1-53cb536b4007" path="/var/lib/kubelet/pods/ad593da4-1691-4121-88c1-53cb536b4007/volumes"
Nov 03 03:13:28.784374 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:28.783479    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-4jmtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:13:28.784374 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:28.783588    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-4jmtt" podUID="9d32d522-440d-42da-84ed-bb3b611a1a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:13:28.835880 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:28.835844    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cnxmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:13:28.836240 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:28.836211    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podUID="f93283bb-8765-407f-86eb-af53fa0e399e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:13:28.976373 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:28.975464    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-bh6ts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:13:28.976373 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:28.975553    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-bh6ts" podUID="9b416926-1f08-4197-80e5-5c3c82166abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:13:41.904824 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:41.904796    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 03:13:42.033741 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:42.033717    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 03:13:43.812596 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:13:43.812553    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:13:46.044885 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:46.044851    5508 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"]
Nov 03 03:13:46.092194 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:46.092156    5508 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g4ftz"]
Nov 03 03:13:46.692358 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:13:46.692318    5508 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4816c8-2912-484e-8929-9f6e6ad4b234" path="/var/lib/kubelet/pods/8b4816c8-2912-484e-8929-9f6e6ad4b234/volumes"
Nov 03 03:14:13.811672 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:14:13.811632    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:14:43.816088 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:14:43.815967    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:14:43.827523 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:14:43.827490    5508 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-cnxmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 03:14:43.827691 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:14:43.827573    5508 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-cnxmt" podUID="f93283bb-8765-407f-86eb-af53fa0e399e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 03:14:47.837997 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:14:47.837962    5508 scope.go:122] "RemoveContainer" containerID="c9d443acf0ef83cff678fd11660fb1838def9b37948a8d65c8e4803bf1d51bbf"
Nov 03 03:14:47.846914 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:14:47.846894    5508 scope.go:122] "RemoveContainer" containerID="5d9c6c8bd7057501204e3be3a99caa3f7a4f46dac7fe11b3075e504d1a35e1f6"
Nov 03 03:15:01.955852 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:01.955814    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-4pqmh"]
Nov 03 03:15:13.812692 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:15:13.812645    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:15:25.803742 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:25.803696    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b6b9b56d5-vpvgj_43b5424f-45e9-49e4-975c-bc9c0dcda261/authorino/0.log"
Nov 03 03:15:40.556373 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:40.556308    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vzvsm_2bfd7d91-c916-4aa1-882f-a068f42ccfc8/proxy/0.log"
Nov 03 03:15:42.104691 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:42.104657    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-67759c8968-j5f8l_f550c92c-848c-404c-946e-d5859646cdef/maas-api/0.log"
Nov 03 03:15:43.810910 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:15:43.810869    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:15:48.058074 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:48.057982    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b6b9b56d5-vpvgj_43b5424f-45e9-49e4-975c-bc9c0dcda261/authorino/0.log"
Nov 03 03:15:49.551905 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:49.551850    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-whgfw_ea2b4e9e-4a03-44e4-b089-c7a7e857562e/manager/0.log"
Nov 03 03:15:49.846246 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:49.846166    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4pqmh_6904580a-c37d-4058-9f86-dc69f52efdf6/limitador/0.log"
Nov 03 03:15:51.537104 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:51.537065    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-44x49_e65b09d4-68d0-4223-af89-6423bc81a63a/discovery/0.log"
Nov 03 03:15:51.780829 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:51.780802    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68dcf9949f-7zlkw_894c4881-96dc-4a44-864b-00d9965f35da/kube-auth-proxy/0.log"
Nov 03 03:15:52.219122 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:52.219090    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-4jmtt_9d32d522-440d-42da-84ed-bb3b611a1a2f/istio-proxy/0.log"
Nov 03 03:15:52.786335 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:52.786293    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-bh6ts_9b416926-1f08-4197-80e5-5c3c82166abd/istio-proxy/0.log"
Nov 03 03:15:53.057476 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:53.057389    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-cnxmt_f93283bb-8765-407f-86eb-af53fa0e399e/istio-proxy/0.log"
Nov 03 03:15:54.487684 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:54.487653    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-xzjj9_1185a711-3994-4129-a641-b93e62e8effe/istio-proxy/0.log"
Nov 03 03:15:55.226482 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:55.226441    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74cf4f96d-l2rms_1978ac08-1656-4999-96cc-7f84264728c1/payload-pre-processing/0.log"
Nov 03 03:15:58.195677 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:15:58.195644    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-x2n9z_1d843061-ce76-4dd2-ad3f-5109467435fb/main/0.log"
Nov 03 03:16:13.811968 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:16:13.811928    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:16:18.650882 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:18.650845    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b4htm_4a5d501e-fa40-467b-a4fa-9fc52f9c45fa/global-pull-secret-syncer/0.log"
Nov 03 03:16:22.338650 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:22.338608    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g4jmf_ee4a4694-e358-4491-b4e1-c05bd477f19a/konnectivity-agent/0.log"
Nov 03 03:16:24.114000 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:24.113972    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-255.us-west-2.compute.internal_0eb27a029b0b668ca0885e8f9e1efac2/haproxy/0.log"
Nov 03 03:16:25.136846 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:25.136810    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vzvsm_2bfd7d91-c916-4aa1-882f-a068f42ccfc8/proxy/0.log"
Nov 03 03:16:27.895997 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:27.895942    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b6b9b56d5-vpvgj_43b5424f-45e9-49e4-975c-bc9c0dcda261/authorino/0.log"
Nov 03 03:16:29.415971 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.415935    5508 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"]
Nov 03 03:16:29.416495 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.416381    5508 cpu_manager.go:414] "containerMap: removing container" podUID="dafa3d14-bec4-46df-84ee-8f418156e7f9" containerName="maas-api"
Nov 03 03:16:29.416495 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.416397    5508 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafa3d14-bec4-46df-84ee-8f418156e7f9" containerName="maas-api"
Nov 03 03:16:29.416495 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.416468    5508 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafa3d14-bec4-46df-84ee-8f418156e7f9" containerName="maas-api"
Nov 03 03:16:29.419900 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.419879    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.468821 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.468785    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kjgqj\"/\"kube-root-ca.crt\""
Nov 03 03:16:29.468821 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.468783    5508 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kjgqj\"/\"openshift-service-ca.crt\""
Nov 03 03:16:29.490513 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.490473    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzvr8\" (UniqueName: \"kubernetes.io/projected/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-kube-api-access-lzvr8\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.490692 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.490589    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-podres\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.490692 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.490620    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-proc\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.490843 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.490698    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-lib-modules\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.490843 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.490746    5508 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-sys\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.515465 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.515433    5508 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kjgqj\"/\"default-dockercfg-9859t\""
Nov 03 03:16:29.517664 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.517636    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"]
Nov 03 03:16:29.592193 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.592156    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-podres\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.592402 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.592203    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-proc\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.592402 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.592271    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-proc\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.592402 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.592289    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-podres\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.592402 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.592315    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-lib-modules\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.592402 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.592388    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-sys\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.592652 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.592444    5508 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzvr8\" (UniqueName: \"kubernetes.io/projected/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-kube-api-access-lzvr8\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.592652 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.592444    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-lib-modules\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.592652 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.592473    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-sys\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.649971 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.649940    5508 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzvr8\" (UniqueName: \"kubernetes.io/projected/51d1c3b2-1ab3-4f93-9165-e55e041dd8d5-kube-api-access-lzvr8\") pod \"perf-node-gather-daemonset-tzdpr\" (UID: \"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5\") " pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:29.732564 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:29.732473    5508 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:30.106226 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:30.106130    5508 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"]
Nov 03 03:16:30.114612 ip-10-0-140-255 kubenswrapper[5508]: W1103 03:16:30.114574    5508 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod51d1c3b2_1ab3_4f93_9165_e55e041dd8d5.slice/crio-54d49672f1a61bd648e89093cc7f57fbc408f44cbc3c0cde9c748cf02dce54a7 WatchSource:0}: Error finding container 54d49672f1a61bd648e89093cc7f57fbc408f44cbc3c0cde9c748cf02dce54a7: Status 404 returned error can't find the container with id 54d49672f1a61bd648e89093cc7f57fbc408f44cbc3c0cde9c748cf02dce54a7
Nov 03 03:16:30.711548 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:30.711508    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-whgfw_ea2b4e9e-4a03-44e4-b089-c7a7e857562e/manager/0.log"
Nov 03 03:16:31.086491 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:31.086382    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr" event={"ID":"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5","Type":"ContainerStarted","Data":"a1d59cbce1e09aa88a5f0c1c9a63b051b0d21d4a37c49042efd17fa5ad5ddd19"}
Nov 03 03:16:31.086655 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:31.086565    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:31.086655 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:31.086603    5508 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr" event={"ID":"51d1c3b2-1ab3-4f93-9165-e55e041dd8d5","Type":"ContainerStarted","Data":"54d49672f1a61bd648e89093cc7f57fbc408f44cbc3c0cde9c748cf02dce54a7"}
Nov 03 03:16:31.091107 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:31.091084    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4pqmh_6904580a-c37d-4058-9f86-dc69f52efdf6/limitador/0.log"
Nov 03 03:16:31.195624 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:31.195572    5508 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr" podStartSLOduration=2.195558319 podStartE2EDuration="2.195558319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 03:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 03:16:31.194584405 +0000 UTC m=+7070.620190868" watchObservedRunningTime="2026-11-03 03:16:31.195558319 +0000 UTC m=+7070.621164781"
Nov 03 03:16:31.289692 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:31.289660    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vzvsm_2bfd7d91-c916-4aa1-882f-a068f42ccfc8/proxy/0.log"
Nov 03 03:16:34.223444 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:34.223414    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84681513-1a85-478c-8ba7-b2e37090dbc0/alertmanager/0.log"
Nov 03 03:16:34.753968 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:34.753945    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84681513-1a85-478c-8ba7-b2e37090dbc0/config-reloader/0.log"
Nov 03 03:16:35.377017 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:35.376987    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84681513-1a85-478c-8ba7-b2e37090dbc0/kube-rbac-proxy-web/0.log"
Nov 03 03:16:35.806865 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:35.806838    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84681513-1a85-478c-8ba7-b2e37090dbc0/kube-rbac-proxy/0.log"
Nov 03 03:16:36.238650 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:36.238613    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84681513-1a85-478c-8ba7-b2e37090dbc0/kube-rbac-proxy-metric/0.log"
Nov 03 03:16:36.670018 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:36.669989    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84681513-1a85-478c-8ba7-b2e37090dbc0/prom-label-proxy/0.log"
Nov 03 03:16:37.101127 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:37.101051    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_84681513-1a85-478c-8ba7-b2e37090dbc0/init-config-reloader/0.log"
Nov 03 03:16:37.104121 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:37.104096    5508 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kjgqj/perf-node-gather-daemonset-tzdpr"
Nov 03 03:16:38.036477 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:38.036444    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-vzvsm_2bfd7d91-c916-4aa1-882f-a068f42ccfc8/proxy/0.log"
Nov 03 03:16:39.256790 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:39.256762    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-67f9494f54-92n2g_df19de7c-3e77-48ce-b5e4-b4b56ab26017/metrics-server/0.log"
Nov 03 03:16:41.429825 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:41.429794    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m24b6_b7d8a03d-d25a-448a-955f-9b7ea3f4e07b/node-exporter/0.log"
Nov 03 03:16:41.856110 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:41.856021    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m24b6_b7d8a03d-d25a-448a-955f-9b7ea3f4e07b/kube-rbac-proxy/0.log"
Nov 03 03:16:42.285650 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:42.285613    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m24b6_b7d8a03d-d25a-448a-955f-9b7ea3f4e07b/init-textfile/0.log"
Nov 03 03:16:43.316582 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:43.316546    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-44x49_e65b09d4-68d0-4223-af89-6423bc81a63a/discovery/0.log"
Nov 03 03:16:43.429493 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:43.429455    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68dcf9949f-7zlkw_894c4881-96dc-4a44-864b-00d9965f35da/kube-auth-proxy/0.log"
Nov 03 03:16:43.810978 ip-10-0-140-255 kubenswrapper[5508]: E1103 03:16:43.810941    5508 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5pxx9" containerName="extractor"
Nov 03 03:16:43.882796 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:43.882762    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-4jmtt_9d32d522-440d-42da-84ed-bb3b611a1a2f/istio-proxy/0.log"
Nov 03 03:16:44.325256 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:44.325220    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-bh6ts_9b416926-1f08-4197-80e5-5c3c82166abd/istio-proxy/0.log"
Nov 03 03:16:44.546040 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:44.546010    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-cnxmt_f93283bb-8765-407f-86eb-af53fa0e399e/istio-proxy/0.log"
Nov 03 03:16:45.309527 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:45.309496    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/0.log"
Nov 03 03:16:45.310657 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:45.310641    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/prometheus/1.log"
Nov 03 03:16:45.641315 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:45.641236    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/config-reloader/0.log"
Nov 03 03:16:45.668919 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:45.668889    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-xzjj9_1185a711-3994-4129-a641-b93e62e8effe/istio-proxy/0.log"
Nov 03 03:16:46.070507 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:46.070474    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/thanos-sidecar/0.log"
Nov 03 03:16:46.328708 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:46.328633    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74cf4f96d-l2rms_1978ac08-1656-4999-96cc-7f84264728c1/payload-pre-processing/0.log"
Nov 03 03:16:46.503505 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:46.503476    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/kube-rbac-proxy-web/0.log"
Nov 03 03:16:46.932071 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:46.932043    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/kube-rbac-proxy/0.log"
Nov 03 03:16:47.362826 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:47.362742    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/kube-rbac-proxy-thanos/0.log"
Nov 03 03:16:47.795251 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:47.795218    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89d794f0-b5b4-4c8b-ad25-165a99f69562/init-config-reloader/0.log"
Nov 03 03:16:49.511237 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:49.511207    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-dd6c498f-58kkc_502e7655-ead5-4ace-a420-cf4a546bcfc7/telemeter-client/0.log"
Nov 03 03:16:49.940514 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:49.940482    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-dd6c498f-58kkc_502e7655-ead5-4ace-a420-cf4a546bcfc7/reload/0.log"
Nov 03 03:16:50.370054 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:50.369985    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-dd6c498f-58kkc_502e7655-ead5-4ace-a420-cf4a546bcfc7/kube-rbac-proxy/0.log"
Nov 03 03:16:52.094113 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:52.094079    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b6b9b56d5-vpvgj_43b5424f-45e9-49e4-975c-bc9c0dcda261/authorino/0.log"
Nov 03 03:16:53.064286 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:53.064250    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-whgfw_ea2b4e9e-4a03-44e4-b089-c7a7e857562e/manager/0.log"
Nov 03 03:16:53.211983 ip-10-0-140-255 kubenswrapper[5508]: I1103 03:16:53.211930    5508 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-4pqmh_6904580a-c37d-4058-9f86-dc69f52efdf6/limitador/0.log"